Linus Groh

Kiesel Devlog #12: Write Once, Run Anywhere

2025-04-29T00:00:00Z

Your favorite JavaScript Engine turned two years old yesterday! I'll use that as an excuse to write another blog post — not just about the birthday though 🥳

Some Numbers

Kiesel is about 80,000 lines of Zig right now, give or take. A lot of it is spec comments so the amount of actual code is a bit lower. There are a few new contributors but I'm still responsible for 98% of commits, so that's definitely the largest codebase I've authored!

Let's start a spreadsheet:

Age	Lines Of Code	Commits	Contributors	test262
1	62768	1435	5	50.4%
2	83627	2106	9	75.1%

The Roadmap

Let's take a look at what's planned next!

Nothing. This is vibe coding before vibe coding was cool (sans the markov chain). If it's not fun I'm not doing it. If it is fun I might do it, maybe. No plans :^)

And Now For Something Completely Different

When I learned Python as a kid I kept reading about portability and that Python is a good fit for it because your applications will run on Windows, macOS, and Linux. These days I know that's bullshit, good luck finding a language that doesn't do that. Writing code that runs on the three major operating systems isn't difficult, it's the default.

Let's make it difficult!

Platform Abstractions

At some point I started experimenting with porting Kiesel to some rather obscure targets, like this one. The process was always the same: taking a fresh copy of the source code, adding hacks wherever needed until it builds and runs (two very different stages of porting!), taking a funny screenshot, maybe saving the diff somewhere, moving on.

Even if I tried to actually commit those changes it wouldn't have been maintainable. In C you end up with #ifdef soup and the Zig equivalent is not much nicer. After the third or fourth time I decided that having support for all of those odd devices out of the box would be nice and started working on a platform abstraction. I later discovered that V8 has something similar, although less flexible — v8::Platform.

The default implementation that Kiesel provides is largely built on top of the Zig standard library and avoids making any assumptions about the underlying platform. You don't have to think about anything:

const platform = Agent.Platform.default();
defer platform.deinit();

This gives you:

A garbage-collecting allocator that works on a large number of platforms
Writers for stdout/stderr
TTY configuration for pretty-printing that takes NO_COLOR and CLICOLOR_FORCE into account
Information about the native stack bounds for stack overflow detection, if supported
A neutral default locale for Intl (BCP 47 und)
A function to retrieve the current system time, if supported

Funnily enough, despite being the world's most portable language, the bits of Kiesel that rely on C are the first to fall apart when you leave the happy path of x86_64-linux. Even though the Zig toolchain has fantastic support for giving you access to libc when cross-compiling that doesn't work on all targets.

In particular we need to deal with the absence of any of these:

The allocator can easily be swapped out with malloc() or similar at the expense of leaks, regular expressions can operate in a stub mode where they pretend to work but don't, and support for Intl can be removed at build time.

$ zig build -Denable-libgc=false -Denable-libregexp=false -Denable-intl=false

So far, so good! But what happens when other parts of the default platform implementation stop working?

One such example is UEFI which is not supported by regular std.io APIs, at least until ziglang/zig#22226 is merged. Various other things don't work either so let's make our own platform:

const allocator = std.os.uefi.pool_allocator;
const stdout: std.io.AnyWriter = .{ ... };
const stderr: std.io.AnyWriter = .{ ... };

const platform: Agent.Platform = .{
    // UEFI pool allocator, memory is leaked instead of GC'd
    .gc_allocator = allocator,
    .gc_allocator_atomic = allocator,
    // Custom writers using the UEFI console
    .stdout = stdout,
    .stderr = stderr,
    // Coloring is only supported using ANSI escape codes and the
    // Windows console API, not UEFI console (yet!)
    .tty_config = .no_color,
    // Stack overflow detection is not supported
    .stack_info = null,
    // Build without Intl, default locale is void
    .default_locale = {},
    // This happens to have an implementation for UEFI :^)
    .currentTime = std.time.milliTimestamp,
};

Additionally Zig provides std.Options as a way to customize certain behavior of the standard library at compile time, which comes in handy too. In the case of UEFI we set a no-op logFn until std.io learns about the UEFI console, as well as a custom implementation of cryptoRandomSeed, which tries to use std.posix.getrandom() by default. And well, UEFI is very much not POSIX.

You can find the full code for this in src/uefi.zig. And just like that we can boot into JavaScript!

Showcases

Linux, macOS, Windows

To prove my earlier point: of course Kiesel runs without issues on the latest versions of Linux, macOS, and Windows. It's using statically linked musl by default so it'll work on pretty much any distro. The REPL — a nod to CPython — includes OS-specific instructions for exiting it (EOF works differently on Windows, as everything does).

Thanks to Sarah for this screenshot!

Linux, But Weird

Still Linux but we're leaving desktop user territory.

Here's Kiesel running on my projector at home, which came with Linux 4.9 (Android 9) out of the box. Zig officially supports Linux ≥ 5.10 at the time of writing and may use syscalls that fail miserably on older versions as a result, but for the most part it works. If you ever get an error.Unexpected back this is probably the reason.

Do you have a mainframe at home? If so, Kiesel for IBM Z may be what you need. I'm pretty sure I know people who this applies to, but for now I'm using a LinuxONE VM that IBM gave me for open source purposes :^)

To make sure Kiesel also works on less common big-endian systems (you're probably reading this on little-endian) famfo once tested it on ppc64.

Windows, But Ancient

Together Domi and I managed to run Kiesel on Windows NT 5.2, a.k.a. Windows Server 2003. I no longer have the patches but it involved finding DLLs online that backport APIs from newer Windows versions. Don't ask why it's running from the SeaMonkey installation directory.

Understandably Zig only wants to deal with Windows 10 and above nowadays so this is very much not supported.

Hey, we should do that again and document it <3

Thanks to Domi for this screenshot!

OpenBSD

Not too long ago one of the OpenBSD developers reached out:

Needed a newer Zig and a few tweaks, but I got it running. One more platform to add to the list :)

One of the tweaks turned out to be a bug in std.c that I fixed upstream. Unfortunately it's not possible to cross-compile for OpenBSD as Zig can't provide a libc:

error: error: unable to find or provide libc for target 'x86_64-openbsd.7.3...7.6-none'

I might look into setting up a VM so I can at least check if the build still works once in a while :^)

Thanks to Tobias for this screenshot!

UEFI

The Zig standard library already wraps large parts of the UEFI API, so I had to try. And sure enough, it worked!

Thanks to CxByte who kindly added UEFI support to his zigline library you can now boot into a JavaScript REPL. I doubt this is the first time ever this exists but I personally haven't seen any other projects doing it before :^)

And of course I had to install it on bare metal too:

Nintendo 3DS

I bought a used Nintendo 3DS, put homebrew on it, and then promptly buried it in a drawer.

Luckily someone already did the work of gluing devkitPro and the Zig toolchain together: let Zig output an object file compiled for arm-freestanding-eabihf, throw that into devkitPro's arm-none-eabi-gcc to get an ELF file, and finally feed that to 3dsxtool. It's not pretty but it works.

The first proof of concept was using a hardcoded snippet of JS, but over time I expanded it to use the software keyboard for an actual REPL, enabled pretty-printing, and ported more of the runtime. There are some crashes left to fix but nowadays it works almost as well as the regular CLI!

The 3DS even ships with JS out of the box (the built-in browser is based on WebKit), but I'm probably the first to use iterator helpers on it.

Your System Here?

🥺👉👈

Thanks!

…to everyone who keeps expanding the Kiesel Enterprise Cinematic Universe (mostly Eloy tbh), Carter for continuing to work on GC things I don't fully understand, Domi for providing CI infra so I can build for all the platforms no one uses, and in particular Alex from the Zig team for the huge effort to improve target support even more!

Let's see what platforms we can run JavaScript on in the next year :^)

Kiesel Devlog #11: Community Edition

2024-09-01T00:00:00Z

Since the last devlog I've barely done any work on Kiesel myself due to being on vacation, so this one is entirely dedicated to things other people contributed!

On that note: I created a Matrix space for the project where you can ask questions or just hang out and chat. I'll also try to post about in-progress things once in a while. Feel free to join!

#kiesel:matrix.org
- #kiesel-general:matrix.org
- #kiesel-development:matrix.org

NaN-Boxing

It was clear we'd need NaN-boxing in Kiesel eventually so I started investigating and doing some refactoring until Carter offered to implement it and came back with a PR two days later (kiesel-js/kiesel#37)!

It provided the performance gains we hoped for and I'm really happy with how it turned out :^)

There are good explanations about NaN-boxing here and here, I recommend you read those if you want to learn more.

kiesel(1)

To my pleasant surprise Clara wrote a traditional manual page for Kiesel: kiesel(1)! We'll extend it over time and I can't wait to run man kiesel once someone starts packaging it :^)

Weak References

Also implemented by Carter were the types for working with weak references:

WeakRef (kiesel-js/kiesel#42)
WeakSet (kiesel-js/kiesel#44)
WeakMap (kiesel-js/kiesel#45)

As well as some foundational work:

FinalizationRegistry (kiesel-js/kiesel#40)

We attempted this twice before and it's great to see how everything fell into place after he figured out the initial GC integration!

UEFI Interactive REPL

Yes, you read that right. I take great pride in Kiesel being a portable piece of software, and the main limitation of its UEFI port was having to hardcode the JS being evaluated.

Luckily I managed to bait CxByte into resolving this critical issue (kiesel-js/kiesel#50). All of the heavy lifting is done in zigline itself so you can use this in your own projects!

Wikidata

Thanks to sofia Kiesel recently got a Wikidata entry! It's probably not quite relevant enough for a Wikipedia article but maybe we'll get there too some day :^)

You're welcome to add more data points if that's your thing, and while I'll not edit this myself I'm always happy to fact-check or provide good sources!

It's Always DNS

Thanks to my friends at project SERVFAIL kiesel.dev now uses globally distributed highly available home grown name servers:

$ dig +noall +answer NS kiesel.dev
kiesel.dev.		4406	IN	NS	ns1.famfo.xyz.
kiesel.dev.		4406	IN	NS	ns2.famfo.xyz.
kiesel.dev.		4406	IN	NS	ns1.homecloud.lol.
kiesel.dev.		4406	IN	NS	miyuki.sakamoto.pl.

Unfortunately they don't use any JS so I put some in a TXT record.

Marketing Go Brrr

Speaking of SERVFAIL, I also managed to recruit their marketing manager!

Shebang has been hard at work producing misinformation about the project at an alarming rate. We gotta have some fun, too :^)

Expand to read the bangers (there's a lot of them)

As the new Kiesel marketing lead, I would like to announce some new Kiesel features coming your way (thread)
Kiesel will now be built on other Ecma technical committies, other than TC39. Meaning that yes, Kiesel will support being used as a CD, as per TC31: https://ecma-international.org/technical-committees/tc31/
A new subscription tier will arrive to Kiesel soon. (Linus needs to eat somehow!) Kiesel Pro will allow you to access features such as Promises, Array prototype methods and booleans!
here’s a sneak peek:
> a = [];
[]
> a.push("meow");
PaywallError: This feature requires a Kiesel™️ Pro subscription.
Kiesel will now be ported to those little phones that go "ayayaya im a little butterfly!"
Kiesel will support being PID 1, so you can go into userspace with Kiesel!

I'm sorry if you followed this account in hopes that I'd be serious about announcing things about #SERVFAIL and/or Kiesel.
Because I definitely am. Anyways who wants a free Kiesel Pro product key??
wow congratulations to this completely uninvolved party that definitely would not lie about winning a giveaway!!!! @iro_miya you just won the Kiesel Pro key tell us all about how you are using it right now and that I definitely DMed it to you
hiiii, thank you so much for the Kiesel Pro key! I am using the Kiesel Pro key to open the Kiesel Pro door. I am currently chilling in the Kiesel Pro room. :ablobcatbongo:

test262.fyi now has #Kiesel Pro added. would you look at that! 173.9%!
if you can't find it on test262.fyi, it's because only Kiesel Pro users can see the value

Reminder to Kiesel Pro customers that Kiesel Pro continues to pass 180% of the test262 suite
oh they're just throwing new ideas in there. IntermediateMap. Array.reverseReverse. Number.nullify. you name it, it's in Kiesel Pro

did you know? @linus is the creator of Kiesel, Linux and runs a very successful Youtube channel called "Linus Tech Tips". This is a true fact!
i can say this per my kiesel contract, which stipulates i can say anything about kiesel

That's all for now! It's really nice to see Kiesel slowly moving away from being just me (it was like that on purpose, read some of the earlier devlogs for the history). Both code and non-code contributions are much appreciated, if you want to get involved please reach out!

Thanks to Domi for proof-reading this post <3

Kiesel Devlog #10: Let's Make It Fast!

2024-08-19T00:00:00Z

Over the last year and a bit Kiesel has grown beyond being a small experiment. I've been churning out feature after feature, and while correctness has been of importance from day one, one thing hasn't been on my radar as much: performance.

You could go as far as saying Kiesel is blazingly slow 🔥. Let's change that :^)

Make It Work

This part is done, it works. Some things better than others, the object model is rock solid for instance (a lesson learned from LibJS), while others fall apart if you look at them the wrong way (looking at you, src/language/bytecode/Vm.zig). The secret here is to make it work most of the time, which gets you pretty far.

At the time of writing Kiesel passes 59.2% of test262, so things are clearly going into a good direction.

I've also been untying various bits of runtime code from the VM recently and am mentally prepared to throw that part away and start over. Making it work all the time is on the horizon.

Make It Right

As mentioned I made sure that everything's following the spec as much as reasonably possible, which is another lesson I learned from working on LibJS. Back then I spent many months fixing countless correctness issues after excitement died down and most early contributors moved on. Let's just say MDN has fantastic docs but it's not a very good reference for writing a JS engine.

Of course there are still correctness bugs in Kiesel, some of them yet to be found, some of them known. The former will be flushed out by test262 eventually and I try to clearly mark all of the latter with TODOs. Frequently they are results of cutting corners for the sake of quick progress, such as number parsing which defers to Zig's std.fmt.parseFloat() and std.fmt.parseInt() with some basic upfront validation. Not having to write a floating point parser was a welcome shortcut at the time :^)

Oh, and so many RegExp parsing bugs. Probably need to rewrite that too.

Make It Fast

I'm occasionally talking with Jason — original creator of the Boa JS engine — about all things engine development. A few months ago he mentioned they set up a benchmarks page which uses an old version of the V8 Benchmark Suite to compare Boa against various other engines. I was keen to add Kiesel but at the time it couldn't even parse the entire benchmark suite, so we abandoned that idea.

Recently I tried again and a few bug fixes later Kiesel was able to run the whole thing to completion! There's just one problem: it was slow. Like an hour while most engines complete within a few minutes slow. Armed with perf(1) and speedscope I got to work :^)

Improving Expression and Statement Parsing

[source]

The very first thing I found was a shocking amount of time spent in the parser! Kiesel has a hand-written parser using the parser-toolkit library, which allows for patterns that lead to very readable code at the expense of becoming quite inefficient for large grammars.

We were effectively brute forcing our way through all possibilities using Zig's error-unwrapping if/else:

pub fn acceptPrimaryExpression(self: *Self) AcceptError!ast.PrimaryExpression {
    const state = self.core.saveState();
    errdefer self.core.restoreState(state);

    if (self.core.accept(RuleSet.is(.this))) |_|
        return .this
    else |_| if (self.acceptIdentifierReference()) |identifier_reference|
        return .{ .identifier_reference = identifier_reference }
    else |_| if (self.acceptLiteral()) |literal|
        return .{ .literal = literal }
    else |_| if (self.acceptArrayLiteral()) |array_literal|
        return .{ .array_literal = array_literal }
    else |_| if (self.acceptObjectLiteral()) |object_literal|
        return .{ .object_literal = object_literal }
    // ... snip ...
    else |_|
        return error.UnexpectedToken;
}

This statement had 15 branches so in the worst case you'd have 14 failed attempts of progressing the parser including state restores until arriving at the right accept function. We can do much better!

The code now looks at the next token and switches on it, which is less readable but means we can immediately pick the right code path:

pub fn acceptPrimaryExpression(self: *Self) AcceptError!ast.PrimaryExpression {
    const state = self.core.saveState();
    errdefer self.core.restoreState(state);

    const next_token = try self.core.peek() orelse return error.UnexpectedToken;
    return switch (next_token.type) {
        .this => blk: {
            _ = self.core.accept(RuleSet.is(.this)) catch unreachable;
            break :blk .this;
        },
        .identifier, .yield, .@"await" => if (self.acceptIdentifierReference()) |identifier_reference|
            .{ .identifier_reference = identifier_reference }
        // ... snip ...
        else |_|
            error.UnexpectedToken,
        .@"[" => .{ .array_literal = try self.acceptArrayLiteral() },
        .@"{" => .{ .object_literal = try self.acceptObjectLiteral() },
        .numeric, .string, .null, .true, .false => .{ .literal = try self.acceptLiteral() },
        // ... snip ...
        else => error.UnexpectedToken,
    };
}

Lazy Global Object Properties

[source]

I implemented intrinsics (the underlying objects behind things like Array and Array.prototype) with lazy initialization in mind from the beginning, allocating everything upfront is wasteful. This works by having a function like this returning Allocator.Error!Object:

try realm.intrinsics.@"%Array.prototype%"();

If the intrinsic has never been accessed before, it will be allocated on the heap and initialized. After that it's a guaranteed allocation-free lookup. The only problem is that the global object would initialize every single constructor and its prototype on startup, making this extra step pointless.

I had to do a bit of surgery to still create those properties upfront but only initialize their value (an intrinsic object) when it is first being looked up from the property storage. This makes creation of the global object and thus startup times much faster as demonstrated here by running an empty script:

Making the AST Less Allocation-Heavy

[source]

A common pattern in the AST is to recursively collect a list of something from a node and all of its children, e.g. a function's var declared names for scoping purposes. This used to be done with each node creating a std.ArrayList for its own var declared names and returning the final result as an immutable slice, i.e. []const ast.Identifier. Parent nodes would collect these and join them together.

All of this incurred a significant allocation overhead, especially with std.ArrayList allocating more space upfront every time it needs to grow. With that in mind I figured a much better solution would be to only ever create one such list and pass it around to not waste any memory from merging and throwing away intermediate results. This means the caller now owns the std.ArrayList and child nodes recursively pass it to collectVarDeclaredNames():

-const var_names = try code.varDeclaredNames(agent.gc_allocator);
-defer agent.gc_allocator.free(var_names);
+var var_names = std.ArrayList(ast.Identifier).init(agent.gc_allocator);
+defer var_names.deinit();
+try code.collectVarDeclaredNames(&var_names);

Results for this microbenchmark look as follows:

function foo() {
  var a;
  {
    var b;
    {
      var c;
      {
        var d;
        {
          var e;
        }
      }
    }
  }
}

for (var i = 0; i < 50_000; ++i) {
  foo();
}

Later on I also tried to cache these results across function runs, but the results didn't convince me.

Optimizing Array Truncation via `.length`

[source]

Another interesting bottleneck I found was array truncation by setting the length property to a lower value than before, e.g.:

const a = [1, 2, 3, 4, 5];
a.length = 3; // a -> [1, 2, 3]

In that case we would collect all numeric property keys, sort them in descending order, and delete them one by one up to the new length. If that sounds a lot like a backwards for loop to you that's because it really should be one :^)

I ended up with a hybrid solution that uses a threshold for switching between either mechanism to not turn something like this into a performance cliff:

const a = [];
a[1_000_000_000] = 123;
a.length = 0; // One billion iterations to check for numeric properties to delete

Avoiding UTF-8 Conversion Churn in Bindings

[source]

Strings in Kiesel have a complicated history: they used to be UTF-8 as that's all modern software should deal with, but are specified to be UTF-16 in ECMAScript and may contain unpaired surrogates which cannot be present in valid UTF-8. These shortcomings as well as the additional complexity from transcoding between UTF-8 and UTF-16 on the fly led to a big refactor to store strings either as a sequence of u8 for ASCII or as a sequence of u16 for UTF-16. This makes indexing (accessing a single code unit) O(1) and generally simplifies a few things.

While doing that refactor I left the code dealing with bindings (the foo in let foo = 42;) untouched, these were still assumed to be UTF-8 (as is the source code that Kiesel parses). Once more this led to a number of on the fly conversions between UTF-8 and UTF-16, and even for the simple ASCII to UTF-8 case we'd end up making a full copy of the string.

Long story short, I updated all of that to use the same underlying String type as regular JS values and the problem went away :^)

Shrinking Objects for Memory Savings Fun and Profit

[source], [source]

Not all performance work is about achieving better Big-O, there's another classic: "allocations are slow". And well, that's not entirely wrong, especially when allocating means generating garbage that needs to be collected later on. You know, like in JS. I've seen profiles where over time the heap would grow to 10GB+ and virtually all time was spent in garbage collection!

The number one reason for heap allocations in JS ought to be objects, so I looked into those. Right now the implementation is a really straightforward property key (string, symbol, or number) to property descriptor (value or getter/setter pair for accessors, three attributes stored as boolean flags) mapping. The only issue is that partial property descriptors can exist in a few situations so all the fields have to be nullable (including a funky double-nullable Object):

const PropertyDescriptor = struct {
    value: ?Value = null,       // 24 bytes
    writable: ?bool = null,     // 2 bytes
    get: ??Object = null,       // 24 bytes
    set: ??Object = null,       // 24 bytes
    enumerable: ?bool = null,   // 2 bytes
    configurable: ?bool = null, // 2 bytes
};                              // 2 bytes (padding)
// -> 80 bytes!

Additionally a populated property descriptor will have either of these fields set, never an intersection of them:

value, writable, enumerable, configurable ("data descriptor")
get, set, enumerable, configurable ("accessor descriptor")

With that in mind we can design this as a tagged enum instead:

// PropertyStorage.zig
const Entry = union(enum) {     // 8 bytes (enum tag + padding)
    accessor: struct {
        get: ?*Object.Data,     // 8 bytes
        set: ?*Object.Data,     // 8 bytes
        attributes: Attributes, // 3 bytes
    },                          // 5 bytes (padding)
                                // -> 24 bytes
    data: struct {
        value: Value,           // 16 bytes
        attributes: Attributes, // 3 bytes
    },                          // 5 bytes (padding)
                                // -> 24 bytes

    const Attributes = packed struct(u3) {
        writable: bool,
        enumerable: bool,
        configurable: bool,
    };
};
// -> 32 bytes :^)

Much better!

Avoiding Number-To-String Conversions for Property Keys

[source]

It's a common misconception that JS has "numeric" property keys; it does not. foo[0] is the exact same thing as foo["0"] — yes, those wacky type coercions :^)

However, engines can use the opportunity to store property keys that also happen to be positive integers as such internally, and Kiesel is no exception. For instance we can then store those property values in an actual array with linear indexing instead of a hashmap.

When it comes to these little unobservable diversions from the spec it's easy to miss additional work that no longer needs to be done. In this case I noticed something in the ToPropertyKey abstract operation:

The abstract operation ToPropertyKey takes argument argument (an ECMAScript language value) and returns either a normal completion containing a property key or a throw completion. It converts argument to a value that can be used as a property key. It performs the following steps when called:
1. Let key be ? ToPrimitive(argument, string).
2. If key is a Symbol, then
   a. Return key.
3. Return ! ToString(key).

Since the spec only deals with strings and symbols as property keys it unconditionally does that ToString at the end, but we can insert an extra step to check coercing key to a primitive value resulted in a number and then use that!

And a Few Others…

…that I won't explain in detail here because they're a bit boring. Feel free to take a look yourself :^)

language: Determine if an arguments object will be needed during parsing
gc: Enable parallel marking
types: Compute string hashes once upfront
A number of commits related to making Value fit into 8 bytes, which will get its own blog post!

Results

Was it worth it? Absolutely!

Comparing the last commit before the first optimization mentioned above (33e5444) with main (e972e9c at the time of writing) we get massive improvements across the board — the V8 benchmark suite (here combined.js) now completes in five minutes instead of 50! We also peak at less than 2GB of memory instead of 9GB.

This is using -Doptimize=ReleaseFast builds and running on my AMD Ryzen 7 7840U Framework Laptop.

That was definitely good enough to participate in the nightly Boa benchmark run, so we finally added Kiesel last week. It started with a score of 33, which quickly climbed up to 56:

For comparison: Boa scores 85, LibJS 265, duktape 380, and QuickJS 900. V8 with JIT disabled scores ~2400. There is room for improvement :^)

The Future

This is just the beginning, and we're barely scratching the surface here. There's a long list of optimizations to look into in the future, just to name a few:

NaN-boxing to shrink the fundamental Value type from 16 to 8 bytes
Linear property storage for numeric properties (i.e. arrays) instead of throwing everything into a hashmap
"Object shapes"/"Hidden classes"
Fast local variables instead of implementing declarative environments with a simple hashmap and parent pointer like described (but not prescribed!) in the spec
Storing small strings in the value itself rather than heap-allocating them, we have some 7 bytes left after the tag
~~A JIT compiler~~ Please stop suggesting this

I'll have to get back to feature development eventually and we're not aiming for a V8 competitor here, but squeezing more and more performance out of the engine over time will be fun!

Kiesel Devlog #9: JavaScript on a Printer

2024-07-02T00:00:00Z

Firstly: go and read Pwning a Brother labelmaker, for fun and interop! by Domi until the end!

Super cool, right? Getting SSH credentials in response to my shitpost was not exactly what I expected but here we are. Let's briefly dive into how I ported my toy JS engine to this Linux-based labelmaker :^)

Initial Attempt

I knew the thing is powered by a 32-bit ARM CPU, so I quickly looked for a suitable Zig target:

$ zig targets | jq .libc | grep arm
  "armeb-linux-gnueabi",
  "armeb-linux-gnueabihf",
  "armeb-linux-musleabi",
  "armeb-linux-musleabihf",
  "arm-linux-gnueabi",
  "arm-linux-gnueabihf",
  "arm-linux-musleabi",
  "arm-linux-musleabihf",
  "arm-windows-gnu",

arm-linux-musleabi seemed the most promising (statically linked libc, softfloat) so I tried that, which seemingly worked fine:

$ zig build -Dtarget=arm-linux-musleabi -Doptimize=ReleaseSafe
$ file ./zig-out/bin/kiesel
./zig-out/bin/kiesel: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, stripped

Alas, on the other side:

# wget https://f.sakamoto.pl/linus/kiesel && chmod +x kiesel
# ./kiesel
Illegal instruction
#

Luckily the crash is obvious: we need to refine the compilation target instead of letting Zig pick a baseline CPU.

What's Inside, Exactly?

Definitely not Intel:

# cat /proc/cpuinfo
Processor	: ARM926EJ-S rev 5 (v5l)
BogoMIPS	: 177.15
Features	: swp half thumb fastmult edsp java
CPU implementer	: 0x41
CPU architecture: 5TEJ
CPU variant	: 0x0
CPU part	: 0x926
CPU revision	: 5

Hardware	: Conexant DigiColor
Revision	: 0000
Serial		: 0000000000000000

Aha, it's the processor that first introduced running Java bytecode in hardware, from 2001. Off to a good start!

the first processor with Jazelle technology was the ARM926EJ-S.

Luckily Zig has a target for that:

// std/Target/arm.zig
pub const cpu = struct {
    // ...
    pub const arm926ej_s = CpuModel{
        .name = "arm926ej_s",
        .llvm_name = "arm926ej-s",
        .features = featureSet(&[_]Feature{
            .v5te,
        }),
    };
};

Which we can use like this:

// build.zig
pub fn build(b: *std.Build) void {
    const target = b.resolveTargetQuery(.{
        .cpu_arch = .arm,
        .cpu_model = .{ .explicit = &std.Target.arm.cpu.arm926ej_s },
        .os_tag = .linux,
        .abi = .musleabi,
    });
    // ...
}

The baseline ARM CPU model specifies v7a as its feature set, which explains the illegal instruction.

Atomics, Atomics Everywhere

Next I was greeted with a sea of __atomic_* linker errors. It appears this CPU doesn't have the features needed by Zig's compiler_rt atomics implementation so it doesn't end up exporting any of the symbols.

I was eagerly starting to write stubs for a bunch of functions (what could go wrong!) but I soon realised we can improve the situation somewhat:

We can disable Atomics and SharedArrayBuffer, the two JS features needing support for atomics by the compiler
There is an option that tells the compiler to assume single-threaded code, which turns atomic operations into non-atomic ones in a bunch of places (e.g. std.Thread.Mutex)
Building in release mode reduced the number of errors even further

Now I was left with this, similar to what's been reported in ziglang/zig#4959:

install
└─ install kiesel
   └─ zig build-exe kiesel ReleaseSafe arm-linux-musleabi 5 errors
error: ld.lld: undefined symbol: __sync_val_compare_and_swap_4
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(heap.PageAllocator.alloc)
error: ld.lld: undefined symbol: __sync_fetch_and_add_1
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(debug.handleSegfaultPosix)
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(debug.panicImpl)
error: ld.lld: undefined symbol: __sync_fetch_and_sub_1
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(debug.handleSegfaultPosix)
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(debug.panicImpl)
error: ld.lld: undefined symbol: __sync_val_compare_and_swap_1
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(crypto.tlcsprng.tlsCsprngFill)
error: ld.lld: undefined symbol: __sync_lock_test_and_set_1
    note: referenced by kiesel
    note:               /home/linus/Dev/kiesel/.zig-cache/o/b4dc1cb84146a4bec8e0e93957b1843c/kiesel.o:(once.Once((function 'do')).callSlow)

No problem, we can stub those out. One can get very far with function stubs alone :^)

export fn __sync_fetch_and_add_1(_: *u8, _: u8) callconv(.C) u8 {
    return 0;
}
export fn __sync_fetch_and_sub_1(_: *u8, _: u8) callconv(.C) u8 {
    return 0;
}
export fn __sync_val_compare_and_swap_1(_: *u8, _: u8, _: u8) callconv(.C) u8 {
    return 0;
}
export fn __sync_val_compare_and_swap_4(_: *u32, _: u32, _: u32) callconv(.C) u32 {
    return 0;
}
export fn __sync_lock_test_and_set_1(_: *u8, _: u8) callconv(.C) u8 {
    return 0;
}

`Panicked during a panic. Aborting.`

Now we get further but still crash, with a somewhat amusing message:

# ./kiesel
Illegal instruction at address 0x5bf4d4
Unable to dump stack trace: debug info stripped
Panicked during a panic. Aborting.
Aborted

Turns out this goes away when building with both -Denable-intl=false and -Denable-libgc=false. Intl is not needed for what we're trying to achieve here, and thanks to Zig's allocator pattern the GC is easily replaced with a std.heap.c_allocator, at the expense of leaking memory.

A non-stripped release build would probably offer some insights, but I didn't look further.

It Works!

As you saw in the blog post linked at the very beginning, this adventure has a good ending:

This article is part of a series called "Linus runs JS on things that absolutely shouldn't, but it's fun and Zig quite portable so he does it anyway". Previous entries:

To quote what I said to CanadaHonk afterwards:

It's a printer. I ran JavaScript on a printer.

Kiesel Devlog #8: SSR, but it's CGI

2024-05-19T00:00:00Z

So, I had an idea the other day. It's enough of a shitpost for April 1st but given we're barely through May and I don't want to wait that long you get to see it now :^)

TL;DR:

$ curl -i http://localhost:1337/cgi-bin/hello.js
HTTP/1.0 200 OK
Content-Type: text/html
Server: Kiesel/0.1.0-dev+c498d0465

Hello world!

Why?

Yes.

On a more serious note, I have no idea how I even got to this point. The last time I touched CGI was about a decade ago, using Python at the time (which has its cgi module scheduled for removal in 3.13 o7).

SSR (Server Side Rendering) with JS has been pretty popular in recent years, but usually involves a lot of setup and bespoke configuration for each framework. I'm sure this CGI-based version of SSR already exists for runtimes like Node but I haven't checked — naturally I'll use my own.

This is also the first time anyone has used Kiesel in production, probably!

How It Works

If you've never used CGI, all you need to know is that it stands for Common Gateway Interface and basically means "run this script to process the request and return its output as the response". Traditionally that's something like Perl or Python, but it can also be a compiled binary or shell script. Or JavaScript!

RFC 3875 goes into more detail if you're interested.

There are age-old CGI solutions for most popular web servers (e.g. FastCGI), but I'm not interested in making this a proper thing. Instead I chose HTTP.sh, a web server and framework written in Bash.

It's mostly made by Domi and much more capable than it sounds! Definitely check it out, you don't always need to throw an entire Django at web-shaped problems :^)

To pull this off we more or less need to do the following:

Turn request data into something JS can read (e.g. JSON)
Run the cgi-bin script with Kiesel while passing it said data somehow and capture the output
Split the output on the first double newline (\n\n) to separate headers from the response body
Return a response

CGI has a number of well-known Request Meta-Variables which can be passed to the script in any way, usually via environment variables or a runtime API. To keep things simple I decided to modify the script source before passing it to Kiesel and prepend an object. Additionally it provides parsed GET and POST params, as well as the full URL and user-agent string. HTTP.sh does most of the heavy lifting (i.e. HTTP request parsing) here.

This makes interacting with the request data as simple as accessing a JS property, e.g. CGI.vars.REQUEST_METHOD or CGI.params.GET["name"]. The full object shape looks like this:

declare const CGI: {
  vars: {
    AUTH_TYPE: string;
    CONTENT_LENGTH?: string;
    CONTENT_TYPE?: string;
    GATEWAY_INTERFACE: "CGI/1.1";
    PATH_INFO: "";
    REMOTE_ADDR: string;
    REMOTE_HOST: null;
    REQUEST_METHOD: "GET" | "POST";
    SCRIPT_NAME: string;
    SERVER_NAME: string;
    SERVER_PORT: number | null;
    SERVER_PROTOCOL: "HTTP/1.0";
    SERVER_SOFTWARE: string;
  };
  params: {
    GET: Record<string, string>;
    POST: Record<string, string>;
  };
  url: string;
  userAgent: string;
};

Examples

Hello World

A basic hello world looks like this:

console.log("Content-Type: text/html\n\nHello world!");

Try it!

Form with `POST` data

console.log("Content-Type: application/json");
console.log("");

const data = CGI.params.POST;
console.log(JSON.stringify(data, null, 4));

Try it!

Visitor Counter

Using the existing Kiesel.readFile() and Kiesel.writeFile() APIs we can even implement a database!

This may not scale, but it works:

console.log("Content-Type: text/html");
console.log("");

const path = "/tmp/db.json";
let db;
try {
  db = JSON.parse(Kiesel.readFile(path));
} catch {
  db = { visits: 0 };
}
db.visits += 1;
Kiesel.writeFile(path, JSON.stringify(db));

console.log(`Visits: ${db.visits}`);

Try it!

Evaluate Code From The Request

console.log("Content-Type: text/plain");
console.log("");

const code = CGI.params.GET["code"] ?? "";
console.log(eval(code));

~~Try it!~~ I've chosen to not deploy publicly available RCE at this time. Here's what it would look like:

$ curl -i 'http://localhost:1337/cgi-bin/eval.js?code=Array(16).join(%22wat%22%20-%201)%20%2B%20%22%20Batman!%22'
HTTP/1.0 200 OK
Content-Type: text/plain
Server: Kiesel/0.1.0-dev+c498d0465

NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN Batman!

Use At Your Own Risk

The code is on Codeberg at kiesel-js/cgi-bin. You're free to use it, but I will reiterate an important part of the MIT license here:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Kiesel Devlog #7: Happy Birthday!

2024-04-28T00:00:00Z

As of today Kiesel, the JS engine I'm building in my spare time, is one year old! This post is a short overview of where we are, how we got there, and where we're going.

commit 929bb044ffcd1776c850a5b924f9992915277e58
Author: Linus Groh 
Date:   Fri Apr 28 19:48:13 2023 +0100

    Initial commit

    Writing another JS engine seems like a fun project for learning Zig :^)

Why Are You Doing This, Again?

What it says in that commit message is true, I did need a project to learn Zig. It's a simple yet powerful language so that didn't take very long. Nowadays I'm doing this purely for fun :^)

I'm happy if Kiesel will be useful to someone some day, but in the end that's not really my motivation. In fact, from my past involvement in open source projects I know that something having users means more work for me. That's also the reason why I initially prevented anyone from participating by disabling both issues and PRs for the repo on Codeberg. That has changed since then but it allowed me to not worry about maintainership and just enjoy the process.

And that's still what this is: recreational programming at its best!

Conformance

Last week Kiesel passed 50% of test262, and it's still around that number — 50.4% at the time of writing. Progress has slowed a bit over time as more and more of the low hanging fruit were implemented.

Something that really helped here is learning from the mistakes made in LibJS and avoid them entirely, including getting the object model and cross-realm interactions right from the beginning.

Another lesson learned from LibJS was that having spec comments on everything is incredibly helpful. You will get things wrong otherwise, unless you run test262 from the beginning, which you can't because it needs a somewhat working runtime.

I enforced this in LibJS at some point, introduced it in Kiesel right from the beginning, and have been seeing this style in various other engines since then. It's great!

Making Progress

If you've never worked on a project of this scale it can seem intimidating. I'm somewhat surprised by the amount of progress myself, but the gist of it is: commit to making a little bit of progress frequently instead of a lot of progress infrequently — at least for me the former works much better.

This extends to a few other patterns, like not blocking yourself on implementing something fully. Both async functions and generators are not fully implemented yet and useless for practical purposes. They don't support await and yield respectively, so why would I commit that code?

Well, already having implemented 80% of what's needed means that those last missing pieces will be much easier to do than if I were to do everything at once :^)

You can see the different phases of development in the LOC graph below:

Side note: Those who know me will not be surprised that I have a Grafana dashboard for this :^)

Roughly it went like this:

Getting the basics up and running (3 months)
Break during August 2023 (travelling, cccamp23)
I got back to work and implemented a huge number of builtins (another 3 months)
At the end of 2023 things slowed down as I started a new job, but I continue to regularly work on various things ever since!

Speaking of lines of code, scc currently reports:

$ scc
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Zig                        113     62768     9273     14038    39457       5732
YAML                         4       130        8         3      119          0
Markdown                     2       136       36         0      100          0
License                      1        21        4         0       17          0
gitignore                    1         2        0         0        2          0
───────────────────────────────────────────────────────────────────────────────
Total                      121     63057     9321     14041    39695       5732
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop (organic) $1,289,044
Estimated Schedule Effort (organic) 15.15 months
Estimated People Required (organic) 7.56
───────────────────────────────────────────────────────────────────────────────
Processed 2533110 bytes, 2.533 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────

Ignore the part that says I'm a time traveller, 7.5 people, and someone owes me 1.2M USD.

Side Quests

A couple of other projects started as a result of this one:

linus/icu4zig: I started writing Zig bindings for the ICU4X library, in order to implement Intl in Kiesel. Progress has stalled but will continue once I work on Intl more.
kiesel-js/runtime: This is a web-compatible runtime (read: WinterCG) for Kiesel, still WIP. It's integrated into the Kiesel CLI by default so you won't even notice it exists separately, but this separation is important to me. Kiesel is an implementation of ECMA-262 and ECMA-402, nothing more.
I'm working on it on and off but it's not as much of a priority to me as the rest of the project. It already has an official runtime key though!
kiesel-js/website: I bought kiesel.dev and made a small website for it. It doesn't have much content at the moment but most importantly powers the Wasm-based playground which I'm still very happy about!
A web browser: this idea has been floating around in my head for a while, it's the natural extension of building a JS engine. I've started working on something related yesterday but progress will only be shared with a few friends until I have more to show. Stay tuned :^)

Thanks!

While this is still largely a me ~~problem~~ project there were a handful of other contributors:

$ git shortlog -sn
  1425  Linus Groh
     7  Carter Snook
     1  Ali Mohammad Pur
     1  Andrew Kaster
     1  Dominique Liberda

Especially seeing CxByte and Andrew show up with some small contributions made me quite happy, I haven't really kept in touch since leaving SerenityOS. Related fun fact: 100% of Kiesel contributors are also SerenityOS contributors. :^)

I've also had countless conversations with CanadaHonk and became friends with them over the last year while they were hacking on their JS engine, porffor. It's been really fun to have that in the absence of a community around Kiesel! They also made test262.fyi which I still look at almost daily.

Very recently Domi provided me with a much more powerful CI runner, shortening build times from 35 minutes to less than 10. This should now allow me to look into running test262 in CI instead of relying on the nightly external test262 run to catch regressions. She's also the author of this banger of a post <3

Lastly the folks from the Zig project have been very responsive to my bug and regression reports, and I even contributed a few patches myself. In a year Zig has already replaced Python as my favourite general purpose language, and I used that for close to a decade!

Overall this has been heaps of fun and I'll at least continue for another year. If this prediction is right I'll be done by then, and in the meantime you're welcome to join the fun :^)

Kiesel Devlog #6: Catching up :^)

2024-01-29T00:00:00Z

Happy new year! Well, new is a stretch, but it's been a while since I last did one of these. This post is gonna be all over the place, but I want to get you up to speed before focusing on more specific things again :^)

TL;DR

The last (exactly!) three months can be summed up as follows:

Kiesel has three new contributors! Considering the total number is four this is a lot 🎉

Ali Mohammad Pur   Fri Oct 27 02:53:42 2023 +0330  cli+language: Use zigline to provide a nice syntax-highlighted REPL
Andrew Kaster      Fri Dec 22 12:00:35 2023 -0500  builtins: Bump comptime limit in reg_exp to 20,000
Dominique Liberda  Fri Dec 29 01:44:25 2023 +0100  builtins: Implement Promise.withResolvers()

273 commits to main, with tons of new features and bug fixes
test262 went from 35.3% to 45.8%, almost half way there 📈

Let's take a closer look :^)

A Better REPL

I already hinted towards this in the last devlog:

CxByte managed to yakbait himself into writing a line editor in Zig and kindly integrated it into Kiesel's REPL!

It still had a bug or two early on but things seem stable now — if you happen to notice any issues, please report them upstream.

Additionally, I added a preamble to the REPL to make it more informative instead of dropping you into a prompt right away:

$ kiesel
Kiesel 0.1.0 [Zig 0.12.0-dev.2341+92211135f] on linux
Use Ctrl+D to exit.
>

macOS CI Builds

Zig is excellent at cross compiling for other targets, but for the longest time I didn't add macOS CI builds because libgc includes mach-o/getsect.h from the macOS SDK, which needs to be installed on the build host. Vendoring the file didn't seem desirable, so I simply ignored the problem — macOS users could still build from source.

After finding and integrating this Zig package, the problem solved itself and there is now a kiesel-macos-aarch64 build on files.kiesel.dev!

Thanks to Andrew for fixing the build after I unknowingly broke it, which lead to me thinking about this again :^)

`Promise.withResolvers()`

This is a new function (stage 4 in November 2023) that implements the commonly used deferred promise pattern. The proposal was championed by a colleague of mine, and I ended up contributing the upstream TypeScript definitions to help getting it ready for production use.

The spec text is seven lines so I resisted writing an implementation myself, knowing it could be a good first task for someone wanting to contribute to Kiesel — and it was! At 37c3 Domi and I sat down and did some Zig hacking together :^)

`Atomics`, `DataView`, `TypedArray`

I finally got around to implementing these three:

TypedArray: still completely unoptimized (using the regular object get/set code paths), but at least memory-wise a raw blob of bytes is already an improvement over using a regular array (which, in Kiesel, is still backed by a hashmap of property keys to JS values)
DataView: Really happy about this, ArrayBuffer is largely useless if you can't read and write bytes
Atomics: only vaguely useful as there are no shared array buffers yet — but hey, free test262 points

`Int`ernationa`l`ization

ECMA-402 (aka Intl) was never that high on my list of priorities for Kiesel (and still isn't), but it's nice to have those APIs I suppose!

Unlike LibJS I opted to not implement my own ICU replacement and chose ICU4X instead (using their C API as ICU4X is written in Rust 🦀).

Generally speaking you can get away with using C APIs in Zig directly, but in this case the code looked horrendous:

I decided to write a simple wrapper as a standalone library — and that's how icu4zig was born!

So far I've used it to implement locale handling, adding other objects will require more work on icu4zig first.

Expand to see the full list of newly implemented Intl builtins 📝

Intl.getCanonicalLocales()
Intl.Locale.prototype.baseName
Intl.Locale.prototype.calendar
Intl.Locale.prototype.caseFirst
Intl.Locale.prototype.collation
Intl.Locale.prototype.hourCycle
Intl.Locale.prototype.language
Intl.Locale.prototype.maximize()
Intl.Locale.prototype.minimize()
Intl.Locale.prototype.numberingSystem
Intl.Locale.prototype.numeric
Intl.Locale.prototype.region
Intl.Locale.prototype.script
Intl.Locale.prototype.toString()
Intl.Locale.prototype[@@toStringTag]
Intl[@@toStringTag]

Annex B Has Joined the Chat

B Additional ECMAScript Features for Web Browsers
The ECMAScript language syntax and semantics defined in this annex are required when the ECMAScript host is a web browser. The content of this annex is normative but optional if the ECMAScript host is not a web browser.

Implementing it means:

I get more test262 points
You can now use Kiesel to implement a web browser. You're welcome.

Jokes aside, while these language features are hopefully not being relied upon in new code, there's existing JS out there that needs them. I've added a compile time flag to make them opt out (-Denable-annex-b=false), and given the implementation is not too invasive I'm happy to have it around :^)

So far this includes builtins and and the [[IsHTMLDDA]] internal slot (if you don't know what that means: good for you!), but none of the syntax changes.

Expand to see the full list of newly implemented Annex B builtins 📝

Date.prototype.getYear()
Date.prototype.setYear()
Date.prototype.toGMTString()
RegExp.prototype.compile()
String.prototype.substr()
String.prototype.anchor()
String.prototype.big()
String.prototype.blink()
String.prototype.bold()
String.prototype.fixed()
String.prototype.fontcolor()
String.prototype.fontsize()
String.prototype.italics()
String.prototype.link()
String.prototype.small()
String.prototype.strike()
String.prototype.sub()
String.prototype.sup()
String.prototype.trimLeft()
String.prototype.trimRight()

Performance 🚀🚀🚀

While Kiesel remains largely unoptimized — make it work, make it right (we are here), make it fast — I've finally started profiling and implemented some quick wins:

Bytecode of functions is now cached after they are first called, no need to regenerate the same one on every call.
Determining the UTF-16 string length from the internal UTF-8 representation was incredibly heavy and is now done only once per string.
Asking a string for one of its code units is now implemented using an iterator that doesn't require allocating a list.
I copied an optimization for property lookups from LibJS, which in turn copied it from JavaScriptCore — when doing property lookups on primitives we don't need to create an object wrapper as per the spec, we can directly start looking on the respective prototype object, no own properties can exist on the wrapper object anyway!
The result can be seen in these flame graphs (before and after):

In general, optimizing is both necessary and heaps of fun, so I'll definitely do more of that! Special shout-out to CanadaHonk who was the source of yakbait for most of these while benchmarking against Porffor :^)

Modules

Not much happened here and you still can't fully import modules in Kiesel, but I got some basics up and running:

Parsing of various (not all!) import/export declarations
Parsing and evaluation of dynamic import() calls
Module environments and namespace objects
Hooking up module resolution in the CLI, i.e. turning module specifiers into paths and reading files from disk

Summary

A few other bits that I didn't mention yet:

Added support for function parameter initializers (default values), \x and \u string escape sequences, and for-in/of statements (although the patch for that one is rather ugly)
I've done a decently large refactor to rip printing and bytecode generation out of the AST itself, those are separate now. Not only did it become unwieldy, but this also makes me more confident doing a rewrite or the bytecode VM (it's kinda bad lol), or adding other backends (e.g. outputting 3rd party IR, AOC, JIT, …)

Expand to see the full list of newly implemented builtins 📝

Atomics.add()
Atomics.and()
Atomics.exchange()
Atomics.isLockFree()
Atomics.load()
Atomics.or()
Atomics.store()
Atomics.sub()
Atomics.xor()
Atomics[@@toStringTag]
DataView.prototype.buffer
DataView.prototype.byteLength
DataView.prototype.byteOffset
DataView.prototype.getFoo()
DataView.prototype.setFoo()
DataView.prototype[@@toStringTag]
decodeURI()
decodeURIComponent()
encodeURI()
encodeURIComponent()
Function.prototype[@@hasInstance]()
Map.groupBy()
Object.fromEntries()
Object.groupBy()
Promise.all()
Promise.allSettled()
Promise.any()
Promise.race()
Promise.withResolvers()
RegExp.prototype[@@match]()
RegExp.prototype[@@split]()
String.fromCharCode()
String.fromCodePoint()
String.prototype.codePointAt()
String.prototype.includes()
String.prototype.lastIndexOf()
String.prototype.match()
String.prototype.replace(), sans substitutions
String.prototype.replaceAll()
String.prototype.split()
String.prototype.substring()
String.prototype.toLowerCase()
String.prototype.toUpperCase()
String.prototype.trim()
String.prototype.trimEnd()
String.prototype.trimStart()
Symbol.prototype.description
TypedArray.BYTES_PER_ELEMENT
TypedArray.prototype.BYTES_PER_ELEMENT
%TypedArray%.from()
%TypedArray%.of()
%TypedArray%.prototype.at()
%TypedArray%.prototype.buffer
%TypedArray%.prototype.byteLength
%TypedArray%.prototype.byteOffset
%TypedArray%.prototype.copyWithin()
%TypedArray%.prototype.entries()
%TypedArray%.prototype.every()
%TypedArray%.prototype.fill()
%TypedArray%.prototype.filter()
%TypedArray%.prototype.find()
%TypedArray%.prototype.findIndex()
%TypedArray%.prototype.findLast()
%TypedArray%.prototype.findLastIndex()
%TypedArray%.prototype.forEach()
%TypedArray%.prototype.includes()
%TypedArray%.prototype.indexOf()
%TypedArray%.prototype.join()
%TypedArray%.prototype.keys()
%TypedArray%.prototype.lastIndexOf()
%TypedArray%.prototype.length
%TypedArray%.prototype.map()
%TypedArray%.prototype.reduce()
%TypedArray%.prototype.reduceRight()
%TypedArray%.prototype.reverse()
%TypedArray%.prototype.slice()
%TypedArray%.prototype.some()
%TypedArray%.prototype.sort()
%TypedArray%.prototype.subarray()
%TypedArray%.prototype.toLocaleString()
%TypedArray%.prototype.toReversed()
%TypedArray%.prototype.toSorted()
%TypedArray%.prototype.toString()
%TypedArray%.prototype.values()
%TypedArray%.prototype.with()
%TypedArray%.prototype[@@iterator]()
%TypedArray%.prototype[@@toStringTag]
%TypedArray%[@@species]

Thanks for reading, until next time! :^)

Kiesel Devlog #5: Progress powered by the Shadow web engine

2023-10-29T00:00:00Z

Shadow Web Engine

This is a project my friend CanadaHonk started working on earlier this week — it's a simple web engine written in JS! More about that here.

For JS execution within Shadow they had the idea to use existing Wasm builds of JS engines instead of relying on the host browser, and to my surprise they picked SpiderMonkey and Kiesel for the initial proof of concept, switchable at runtime!

This resulted in me spending an evening fixing bugs and looking into missing features they asked about, including a few of the below. Of course this isn't anywhere near being a serious web engine just yet, but it's really cool to have Kiesel featured in one regardless :^)

Classes

I decided to implement classes even before Shadow happened just to make some progress on language features again — they definitely win in the category weirdest codegen across the entire engine, but other than that I'm happy with how that turned out :^)

Supported features include class expressions/declarations, methods (of course), constructor(), static and instance class fields, static blocks, as well as super calls and properties (no private properties yet). You can see some of those in action here.

I also added support for parsing async, generator, and async generator method definitions in classes or object literals, but they're just as non-functional at runtime as their function declaration counterparts.

Sharing a screenshot of an entire class being written on a single line with no syntax highlighting prompted CxByte (whom you might know from the SerenityOS project! 👋) to start working on a LibLine Zig port to improve the REPL, more on that soon!

The Beginning of Modules

Don't get too excited just yet, there's not much working here! For now I implemented the distinction between scripts and modules, a flag for the CLI, some basic parsing of import declaration, and the import.meta object.

I had very little to do with the module support in LibJS, so this is new territory for me :^)

Other Language Features

I implemented a couple more small language features:

Function rest parameters
Spreading in array and object literals
Unlabelled break and continue statements (yes, I went 6+ months without those 😅)
Optional expressions (foo?.bar, foo?.["bar"], foo?.()) - I remember struggling with this when attempting an implementation in LibJS once, but here it's a very straightforward 140 line diff. Huh!

Resizable Array Buffers

These existed as a proposal for a while and finally got merged into the main spec a couple of weeks ago, so I promptly implemented the new functionality :^)

Reporting of Unhandled Promise Rejections

A small DX improvement; the CLI will now report these two cases at the end of script evaluation:

A promise was rejected without any handlers
A handler was added to an already rejected promise

Under the hood this uses the same mechanism as the unhandledrejection and rejectionhandled events on the web.

Improved CI Builds

I continue to publish pre-built binaries on files.kiesel.dev, with a few recent changes:

Use musl for Linux builds
Strip executable in release modes
Addition of a Windows aarch64 build (because we can)
Addition of a Linux riscv64 build (also because we can)

Summary

Expand to see the full list of newly implemented builtins 📝

ArrayBuffer.prototype.maxByteLength
ArrayBuffer.prototype.resizable
ArrayBuffer.prototype.resize()
JSON.parse()
JSON.stringify()
Number.parseFloat()
Number.parseInt()
RegExp.prototype[@@matchAll]()
RegExp.prototype[@@search]()
String.prototype.endsWith()
String.prototype.indexOf()
String.prototype.matchAll()
String.prototype.search()
String.prototype.startsWith()

Non-standard:

Kiesel.readFile()
Kiesel.readLine()
Kiesel.readStdin()
Kiesel.sleep()
Kiesel.writeFile()

test262 is still steadily going up, now at 35.3%. I check the graph every morning, seeing the small bump from last night's run is very motivating 📈

Collaborating with CanadaHonk so they can use Kiesel in the Shadow web engine has been really fun, and I made lots of progress thanks to that! I'm not sure how quickly modules will take shape from here on, but they create some exciting new abilities such as importable native modules:

Kiesel Devlog #4: The biggest update yet!

2023-10-11T00:00:00Z

It's been two weeks and 108 commits since the last devlog, and I'm really happy with the progress I made! Let's take a look at all those shiny new features :^)

A quick check of test262.fyi reveals that Kiesel now scores exactly 33%.

ArrayBuffer

First up, ArrayBuffer. They're not very capable and mostly serve as memory storage for other kinds of objects (DataView, typed arrays), and are thus fairly easy to implement. The resizable ArrayBuffer proposal will make things slightly more complicated, but I chose a std.ArrayList(u8) for the underlying buffer, which already is resizable.

I also added a non-standard detachArrayBuffer() function to the Kiesel namespace object, which exposes the abstract operation of the same name — primarily for test262. Here is it in action, along with the REPL's pretty-printing:

I haven't started working on the aforementioned typed arrays yet, but I think Zig has a nice way of writing a generic implementation that handles all the different integer types.

Promise

I previously implemented promises in LibJS, and it basically took me a year to finish — luckily doing it again in Kiesel was much quicker :^)

I recorded most of the process in a two-part video series, first adding Promise objects and then implementing promise reaction jobs:

RegExp

To keep scope creep under control I decided to not write a JS-flavored regular expression library at this point, the same way I opted to use an existing library for GC. Both of these are fairly decoupled from everything else and easy to replace later on if desired.

I ended up using quickjs's libregexp:

libregexp: small and fast regexp library fully compliant with the Javascript ES 2019 specification.

I also briefly considered using SerenityOS's LibRegex, which is what LibJS uses, but adding C wrappers for a C++ library is a can of worms I wasn't keen to open.

However, choosing to not write this myself comes at a cost: C code is compiled with much less safety than Zig code, and I already found a bug that I had to patch (and at least one other crash I still need to investigate). These flaws aside, using C libraries from Zig is one of its major selling points, so the actual integration took me less than a day!

And a quick demo:

Other Builtins

Lastly I implemented a few remaining Math functions, parseInt(), a half-assed version of parseFloat(), and bound function objects via Function.prototype.bind().

Expand to see the full list of newly implemented builtins 📝

ArrayBuffer()
ArrayBuffer.isView()
ArrayBuffer.prototype.byteLength
ArrayBuffer.prototype.slice()
ArrayBuffer.prototype[@@toStringTag]
Function.prototype.bind()
Math.atan2()
Math.fround()
Math.imul()
parseFloat(), somewhat
parseInt()
Promise()
Promise.prototype.catch()
Promise.prototype.finally()
Promise.prototype.then()
Promise.prototype[@@toStringTag]
Promise.reject()
Promise.resolve()
RegExp()
RegExp.prototype.dotAll
RegExp.prototype.exec()
RegExp.prototype.flags
RegExp.prototype.global
RegExp.prototype.hasIndices
RegExp.prototype.ignoreCase
RegExp.prototype.multiline
RegExp.prototype.source
RegExp.prototype.sticky
RegExp.prototype.test()
RegExp.prototype.toString()
RegExp.prototype.unicode
RegExp.prototype.unicodeSets

New Language Features

I continue to favor working on runtime, but I've made some progress with language features as well:

Regular expression literals (effectively syntactic sugar for the RegExp() constructor)
Hashbang comments [MDN]
The new.target meta property [MDN]
Mapped arguments objects [Spec]
An arguments exotic object is an exotic object whose array index properties map to the formal parameters bindings of an invocation of its associated ECMAScript function.
Parsing and basic bytecode generation for more function types:
- Async arrow functions
- Async function expressions/declarations
- Async generator expressions/declarations
- Generator expressions/declarations
However, they cannot run their function bodies in a meaningful way yet.

Windows Build

I personally haven't used Windows for anything in a long time, but I aim for Kiesel to work on all major operating systems. Thanks to Zig's amazing cross-compilation capabilities testing that doesn't even involve VMs or dual-booting!

Someone on the Zig Discord server gave me a hint how to fix one of the issues I encountered when trying to build a Windows executable, so I went ahead and fixed a few more. And now we have a working Windows build!

SyntaxError Source Hints

Syntax errors are edge cases and thus I haven't spent much effort on making them nice so far. Many of the error messages are not very useful, especially with the parser backtracking on unknown tokens and then telling you this nonsense:

$ cat file.js
function foo() { # }
$ kiesel file.js
SyntaxError: Unexpected token 'function' (file.js:1:1)

For the cases where the location is accurate this kind of line/column output is still not super helpful though, so I added source hints:

$ cat file.js
/[a-z]/x
$ kiesel file.js
/[a-z]/x
       ^
Uncaught exception: SyntaxError: Invalid RegExp flags (file.js:1:8)

Much better!

Recordings Are Back!

As mentioned above: I started recording some of the work I do on Kiesel, and there will definitely be more of that!

Videos are uploaded to both PeerTube and YouTube, and will be as irregular as ever :^)

Kiesel Devlog #3: Accessors, Date/Map/Set, test262 history graph

2023-09-26T00:00:00Z

Accessors in Object Expressions

I first implemented the fairly similar object method definition syntax ({ foo() {} }), but cheated with the AST representation — this would output the same structure as { foo: function() {} }, which obviously falls apart when the definition has a get or set prefix.

One small refactor later I had an object_define_method bytecode instruction tailored to this use case, which then allowed me to implement getters and setters easily by passing a method, get, or set enum argument to the instruction. The underlying accessor property mechanism was already working and used for some builtin objects.

The generated bytecode looks like this (the indices refer to the constant and function expression tables embedded in the executable):

> { foo() {}, get bar() {}, set baz(arg) {} }
Generated Bytecode
------------------
 0: object_create
 1: load
 2: store_constant "foo" [0]
 5: load
 6: object_define_method [0] (type: method)
11: load
12: store_constant "bar" [1]
15: load
16: object_define_method [1] (type: get)
21: load
22: store_constant "baz" [2]
25: load
26: object_define_method [2] (type: set)
31: <end>

{ "foo": fn foo, "bar": <accessor>, "baz": <accessor> }
>

Or, in the case of a computed property (that store_constant + load sequence can be optimized to a single load_constant instruction eventually):

> { get [foo + "bar"]() {} }
Generated Bytecode
------------------
 0: object_create
 1: load
 2: resolve_binding foo [0] (strict: false)
 7: get_value
 8: load
 9: store_constant "bar" [0]
12: load
13: apply_string_or_numeric_binary_operator (operator: +)
16: load
17: object_define_method [0] (type: get)
22: <end>

New Builtins!

TL;DR:

Finished implementing Array.prototype
Added most of Date including basic parsing, no non-UTC time zones yet
Added all of Map and Set, including their iterators

This amounts to a almost 100 new functions!

A lot of this was fairly repetitive (Date having a getter and setter for both local time and UTC, as well as Map and Set being nearly identical), but it was still fun to implement these :^)

Expand to see the full list of newly implemented builtins 📝

Array.prototype.concat()
Array.prototype.copyWithin()
Array.prototype.fill()
Array.prototype.flat()
Array.prototype.flatMap()
Array.prototype.reduce()
Array.prototype.reduceRight()
Array.prototype.reverse()
Array.prototype.slice()
Array.prototype.sort()
Array.prototype.splice()
Array.prototype.toReversed()
Array.prototype.toSorted()
Array.prototype.toSpliced()
Date()
Date.now()
Date.parse()
Date.prototype.getDate()
Date.prototype.getDay()
Date.prototype.getFullYear()
Date.prototype.getHours()
Date.prototype.getMilliseconds()
Date.prototype.getMinutes()
Date.prototype.getMonth()
Date.prototype.getSeconds()
Date.prototype.getTime()
Date.prototype.getTimezoneOffset()
Date.prototype.getUTCDate()
Date.prototype.getUTCDay()
Date.prototype.getUTCFullYear()
Date.prototype.getUTCHours()
Date.prototype.getUTCMilliseconds()
Date.prototype.getUTCMinutes()
Date.prototype.getUTCMonth()
Date.prototype.getUTCSeconds()
Date.prototype.setDate()
Date.prototype.setFullYear()
Date.prototype.setHours()
Date.prototype.setMilliseconds()
Date.prototype.setMinutes()
Date.prototype.setMonth()
Date.prototype.setSeconds()
Date.prototype.setTime()
Date.prototype.setUTCDate()
Date.prototype.setUTCFullYear()
Date.prototype.setUTCHours()
Date.prototype.setUTCMilliseconds()
Date.prototype.setUTCMinutes()
Date.prototype.setUTCMonth()
Date.prototype.setUTCSeconds()
Date.prototype.toDateString()
Date.prototype.toISOString()
Date.prototype.toJSON()
Date.prototype.toLocaleDateString()
Date.prototype.toLocaleString()
Date.prototype.toLocaleTimeString()
Date.prototype.toString()
Date.prototype.toTimeString()
Date.prototype.toUTCString()
Date.prototype.valueOf()
Date.prototype[@@toPrimitive]()
Date.UTC()
Map()
Map.prototype.clear()
Map.prototype.delete()
Map.prototype.entries()
Map.prototype.forEach()
Map.prototype.get()
Map.prototype.has()
Map.prototype.keys()
Map.prototype.set()
Map.prototype.size
Map.prototype.values()
Map.prototype[@@iterator]()
Map.prototype[@@toStringTag]
Math.max()
Math.min()
Set()
Set.prototype.add()
Set.prototype.clear()
Set.prototype.delete()
Set.prototype.entries()
Set.prototype.forEach()
Set.prototype.has()
Set.prototype.keys()
Set.prototype.size
Set.prototype.values()
Set.prototype[@@iterator]()
Set.prototype[@@toStringTag]
String.prototype.concat()
String.prototype.repeat()
String.prototype.slice()

A History Graph for test262.fyi

As mentioned in the previous devlog, test262.fyi was updated to start collecting historic data instead of only keeping results for the last run. The obvious use case is a graph comparing all engines against the total number of tests, which I've implemented using Chart.js:

This isn't on the live site yet, but I hope to upstream it soon. You can also see that Kiesel passed the 30% mark two days ago! :^)

What's Next?

Promises or typed arrays probably, both of which are relatively large and will probably have an entire devlog dedicated to them, which is why I'm writing this one now to have a clean slate.

Ignoring async/await promises aren't used that much within ECMAScript itself, but typed arrays bring a whole lot of related functionality with them (ArrayBuffer/SharedArrayBuffer, DataView, Atomics).

Kiesel Devlog #2: Iterators, more math, and a bug in the Zig stdlib

2023-09-15T00:00:00Z

Welcome to the second devlog for the Kiesel JS engine! It hasn't been long since the last one, but I worked on a couple of fun things since then :^)

Iterators

Quoting myself from last week:

[…] two major features needed to unlock a bunch of runtime functionality are promises and iterators, so I'll likely work on those soon.

And that's what I did the next day. Turns out iterators are much simpler than I remembered — the entire implementation is less than 300 lines!

As expected this unlocked a few more builtins which I did next:

Array iterators (both @@iterator and the keys()/values()/entries() prototype functions)
String iterators (no key/value concept here, only @@iterator giving back code points)
Array.from(), which can collect items either from an iterable object or a plain array-like object
The AggregateError constructor, which is commonly called with an array of errors but accepts an arbitrary iterable object

More `Math` Functions

I only implemented about half of the Math object, so a bunch of functions were still missing. I picked mostly trigonometric and logarithmic functions next, all of which have an equivalent in Zig's std.math library (or even compiler intrinsics) and thus were easy to implement.

Conveniently they are directly mirroring their equivalents from the standard C library in terms of behavior for "special" inputs (NaN, infinity, negative zero), as do the ones in ECMAScript.

This means that most steps can be folded into a single function call, e.g. for Math.sin():

/// 21.3.2.30 Math.sin ( x )
/// https://tc39.es/ecma262/#sec-math.sin
fn sin(agent: *Agent, _: Value, arguments: ArgumentsList) !Value {
    const x = arguments.get(0);

    // 1. Let n be ? ToNumber(x).
    const n = try x.toNumber(agent);

    // 2. If n is one of NaN, +0𝔽, or -0𝔽, return n.
    // 3. If n is either +∞𝔽 or -∞𝔽, return NaN.
    // 4. Return an implementation-approximated Number value representing the result of the
    //    sine of ℝ(n).
    return Value.from(@sin(n.asFloat()));
}

While doing this I noticed two functions misbehaving in a subtle way: both std.math.asinh() and std.math.cbrt() would return 0 for -0 even though they are supposed to preserve the sign. I filed issues #17111 and #17112, which promptly got fixed in PRs #17120 and #17121 🎉

This is of course much less likely to happen in mature language and library implementations, but I'm more than happy to be an "early adopter" and improve the language for everyone by reporting/fixing such issues :^)

Pretty-Printing in Scripts

The Kiesel REPL already implements basic pretty-printing, generally trying to print relevant internal object state instead of just serializing (enumerable) properties.

I decided to expose this functionality to scripts — similar to pprint.pprint() in Python — by adding an optional second argument for options to the non-standard Kiesel.print() builtin:

Kiesel.print(value, { pretty: true });

Summary

test262 went up exactly 1%, now at 26%. CanadaHonk also prepared data collection for making a graph of progress over time for each engine, which I'm really looking forward to!

Expand to see the full list of newly implemented builtins 📝

AggregateError()
AggregateError.prototype.message
AggregateError.prototype.name
Array.from()
Array.prototype.entries()
Array.prototype.filter()
Array.prototype.keys()
Array.prototype.shift()
Array.prototype.unshift()
Array.prototype.values()
Array.prototype[@@iterator]()
Math.acos()
Math.acosh()
Math.asin()
Math.asinh()
Math.atan()
Math.atanh()
Math.cbrt()
Math.cos()
Math.cosh()
Math.exp()
Math.expm1()
Math.log()
Math.log10()
Math.log1p()
Math.log2()
Math.sin()
Math.sinh()
Math.sqrt()
Math.tan()
Math.tanh()
String.prototype.at()
String.prototype[@@iterator]()

There also were a couple of people reaching out about how to file issues or contribute after the previous devlog ended up on Lobsters. The short version is that I'm treating this as a read-only open source project for the moment — the source code is freely available, and everyone is welcome to use and modify it. However I want to avoid falling back into a mode where I spend more time reviewing PRs and handling issues than making actual progress myself.

There are plenty of issues that I'm aware of right now, and more that I'm not currently aware of, but this isn't the time for a backlog. Feel free to reach out directly if you have any questions/suggestions/fixes for obvious mistakes (example). :^)

Kiesel Devlog #1: Now passing 25% of test262!

2023-09-10T00:00:00Z

Except for the initial announcement and a couple of fedi posts I haven't talked much about the latest project I'm working on - writing a JS engine from scratch in Zig!

A little over four months and exactly 600 commits later it now passes 25% of test262 (20% only a week ago!), so this seems like a good time to write the first devlog :^)

Humble Beginnings

commit 929bb044ffcd1776c850a5b924f9992915277e58
Author: Linus Groh 
Date:   Fri Apr 28 19:48:13 2023 +0100

    Initial commit

    Writing another JS engine seems like a fun project for learning Zig :^)

There was nothing useful in this commit yet, it contains a hello world generated by zig init-exe. Over the next few weeks I stubbed out some widely used language building blocks (Agent, Realm, PropertyDescriptor, …), the fundamental Value type, and most importantly the object model.

I spent the last three years or so working on LibJS, SerenityOS's JS engine used in the Ladybird browser — hence another. Already having a solid mental model of the ECMAScript spec meant that I could fully focus on learning Zig as a new language instead having to wrap my head around JS language concepts.

Or, as domi puts it:

the "an average person writes two javascript engines in their life" factoid is false. A statistical person writes 0 javascript engines; Linus Groh, who writes them for fun is an outlier and should not have been counted

And indeed, it is lots of fun :^)

Until I made a basic tokenizer, parser, and bytecode VM, everything was glued together by hand which looked something like this (adapted for a couple of API changes):

pub fn main() !void {
    var agent = try Agent.init(gc.allocator(), .{});
    defer agent.deinit();

    try Realm.initializeHostDefinedRealm(&agent, .{});

    const object1 = try builtins.Object.create(&agent, .{
        .prototype = null,
    });
    _ = try object1.internalMethods().defineOwnProperty(
        object1,
        PropertyKey.from("foo"),
        PropertyDescriptor{ .value = Value.from(123) },
    );
    const object2 = try builtins.Object.create(&agent, .{
        .prototype = object1,
    });
    const value = try object2.internalMethods().get(object2, PropertyKey.from("foo"), Value.from(object2));

    std.debug.print("object2.foo = {any}\n", .{value});
}

What's Implemented So Far?

After getting the basics up and running I mostly focused on implementing enough syntax and builtins to run test262, the official ECMAScript conformance test suite. This was planned from the very beginning so I could avoid having to write my own test runner and suite.

Not focusing on running test262 early on is also one of my big regrets from LibJS (as well as not getting objects and realms right from the beginning), so that was to be avoided.

Turns out: you don't need a huge amount of features for test262, which is great! So little in fact that I only implemented for loops yesterday to get another test harness file working.

Builtins

JS has a lot of built-in functions and keeps getting more, so this will take a while. Sometimes I target missing functions used in the test262 harness causing tests for implemented functionality to fail, but for the most part I randomly pick something that seems fun to work on :^)

Yes, there's a partial implementation of eval() (before pretty-printing and when the code was still on GitHub). Why do you ask?

The famous demo from the Wat talk also works, and more recently I added proxies:

I also added a couple of non-standard functions (mostly for test262), but they're implemented in the kiesel utility, not the engine itself:

Kiesel.gc.collect()
Kiesel.createRealm()
Kiesel.evalScript()
Kiesel.print() (no console object yet)

Expand to see the full list of currently implemented builtins 📝

globalThis
Infinity
NaN
undefined
eval()
isFinite()
isNaN()
Array()
Array.isArray()
Array.of()
Array.prototype.length
Array.prototype.at()
Array.prototype.every()
Array.prototype.find()
Array.prototype.findIndex()
Array.prototype.findLast()
Array.prototype.findLastIndex()
Array.prototype.forEach()
Array.prototype.includes()
Array.prototype.indexOf()
Array.prototype.join()
Array.prototype.lastIndexOf()
Array.prototype.map()
Array.prototype.pop()
Array.prototype.push()
Array.prototype.some()
Array.prototype.toLocaleString()
Array.prototype.toString()
Array.prototype.with()
BigInt()
BigInt.prototype.toLocaleString()
BigInt.prototype.toString()
BigInt.prototype.valueOf()
Boolean()
Boolean.prototype.toString()
Boolean.prototype.valueOf()
Error()
Error.prototype.message
Error.prototype.name
Error.prototype.toString()
NativeError() (EvalError, RangeError, ReferenceError, SyntaxError, TypeError, URIError)
NativeError.prototype.message
NativeError.prototype.name
Function()
Function.prototype.apply()
Function.prototype.call()
Function.prototype.toString()
Math.E
Math.LN10
Math.LN2
Math.LOG10E
Math.LOG2E
Math.PI
Math.SQRT1_2
Math.SQRT2
Math.abs()
Math.ceil()
Math.clz32()
Math.floor()
Math.pow()
Math.random()
Math.round()
Math.sign()
Math.trunc()
Number()
Number.EPSILON
Number.MAX_SAFE_INTEGER
Number.MAX_VALUE
Number.MIN_SAFE_INTEGER
Number.MIN_VALUE
Number.NaN
Number.NEGATIVE_INFINITY
Number.POSITIVE_INFINITY
Number.isFinite()
Number.isInteger()
Number.isNaN()
Number.isSafeInteger()
Number.prototype.toLocaleString()
Number.prototype.toString()
Number.prototype.valueOf()
Object()
Object.assign()
Object.create()
Object.defineProperties()
Object.defineProperty()
Object.entries()
Object.freeze()
Object.getOwnPropertyDescriptor()
Object.getOwnPropertyDescriptors()
Object.getOwnPropertyNames()
Object.getOwnPropertySymbols()
Object.getPrototypeOf()
Object.hasOwn()
Object.is()
Object.isExtensible()
Object.isFrozen()
Object.isSealed()
Object.keys()
Object.preventExtensions()
Object.seal()
Object.setPrototypeOf()
Object.values()
Object.prototype.hasOwnProperty()
Object.prototype.isPrototypeOf()
Object.prototype.propertyIsEnumerable()
Object.prototype.toLocaleString()
Object.prototype.toString()
Object.prototype.valueOf()
Proxy()
Proxy.revocable()
Reflect.apply()
Reflect.construct()
Reflect.defineProperty()
Reflect.deleteProperty()
Reflect.get()
Reflect.getOwnPropertyDescriptor()
Reflect.getPrototypeOf()
Reflect.has()
Reflect.isExtensible()
Reflect.ownKeys()
Reflect.preventExtensions()
Reflect.set()
Reflect.setPrototypeOf()
String()
String.prototype.charAt()
String.prototype.charCodeAt()
String.prototype.toString()
String.prototype.valueOf()
Symbol()
Symbol.asyncIterator
Symbol.for
Symbol.hasInstance
Symbol.isConcatSpreadable
Symbol.iterator
Symbol.keyFor
Symbol.match
Symbol.matchAll
Symbol.prototype
Symbol.replace
Symbol.search
Symbol.species
Symbol.split
Symbol.toPrimitive
Symbol.toStringTag
Symbol.unscopables
Symbol.prototype.toString()
Symbol.prototype.valueOf()

Syntax

I'm using xq's fantastic parser-toolkit library. Translating the context-free grammar from the ECMAScript spec into that isn't always straightforward, but so far I'm not feeling awful about the parser. It's currently 1.4k lines and somehow hasn't turned into a pile of spaghetti code yet. :^)

Expand to see the full list of currently implemented syntax features 📝

Literals:
- true/false/null (undefined is not a literal :^))
- numbers
- bigints
- strings (no template literals or escapes)
- arrays (including array holes)
- objects (no function property shorthands)
Statements:
- blocks
- var, for
- while
- do/while
- if/else
- try/catch/finally
- throw
- return
- debugger
- empty
Functions:
- function declarations
- function expressions
- arrow functions
Other expressions:
- identifiers references
- member expressions
- call expressions
- new expressions
- update expressions (prefix/suffix ++/--)
- unary expressions (prefix delete, void, typeof, +, -, ~, !)
- binary expressions (**, *, /, %, +, -, <<, >>, >>>, &, ^, |)
- relational expressions (<, >, <=, >=, instanceof, in)
- equality expressions (==, !=, ===, !==)
- logical expressions (&&, ||, ??)
- conditional expressions (a ? b : c)
- assignment expressions (=, *=, /=, %=, +=, -=, <<=, >>=, >>>=, &=, ^=, |=, **=, &&=, ||=, ??=)
- sequence expressions (, operator)

Lexical declarations are parsed but use the same bytecode and scoping rules as var declarations; this was to unbreak tests that relied on simple variable assignment using let or const for unrelated functionality.

I prefer working on runtime over syntax/language, mostly because the abstractions in Kiesel for that are much nicer compared to parsing and bytecode generation. So if you're wondering why a certain language feature is missing: that's why.

What's Missing?

Everything else, of course :^)

I consider the entire latest ECMAScript draft to be in scope, only a handful of language features marked Normative Optional, Legacy will be skipped.

There's no roadmap, but two major features needed to unlock a bunch of runtime functionality are promises and iterators, so I'll likely work on those soon. Lesser-used builtins like Weak{Map,Ref,Set} or Atomics are low on my priority list and will take longer to appear.

A huge deficiency is the current error reporting, which manifests in two ways:

There are no tracebacks, so finding the exact error source can involve some guesswork and/or printing out the current bytecode offset:
```
$ cat -p foo.js
throw new Error("oh no!")
$ kiesel foo.js
Uncaught exception: Error: oh no!
$
```
The majority of early errors (syntax errors for certain invalid constructs) are not implemented, and the parser will generally backtrack to the construct it's currently parsing when encountering syntax it doesn't understand (with statements in this case):
```
$ cat foo.js
if (foo) {
    // a comment
}

function bar() {
    with (baz) {}
}
$ kiesel foo.js
Uncaught exception: SyntaxError: Unexpected token 'function' (foo.js:3:2)
$
```
(there's also something going wrong with the source location but that's another story…)

Thoughts On Zig

As I mentioned at the beginning, the whole point of this was to learn Zig, which I did — I'm by no means an expert but would probably call myself proficient already. It's a joy to use, despite being a 0.x project including compiler bugs and breaking changes (which I was fully aware of from the beginning). They're a relatively small team with large ambitions, and I like that :^)

You can see what it's all about on the website. I did find the standard library lacking on a few occasions, e.g. there seems to be no function for checking if a "list of strings" (slices, []const []const u8) contains a string ([]const u8) — and for (haystack) |value| { if (std.mem.eql(u8, value, needle)) break true; } else false ain't it. I have this in a helper function for now :^)

This also happens to be the first time I learned a language fully out of interest (I need a project for this language I want to learn), not out of necessity (I need to learn a language for this project I want to work on)!

Oh, btw…

…I also want to mention porffor:

a basic wip js aot optimizing wasm compiler in js

It's another new JS engine project, (compiling JS to Wasm, written in JS), and I often talk to CanadaHonk while we're both hacking away on different things or comparing functionality or benchmarks for fun (porffor-compiled code runs on the V8 Wasm engine and thus always wins).

They also built the incredibly useful test262.fyi site, which involved forking and modernizing some existing tooling (esvu, eshost, test262-harness). And of course it features results from Kiesel :^)

Setting up a Self-Hosted Forgejo Actions Runner with Docker Compose

2023-07-07T00:00:00Z

I created an account on Codeberg the other day with the intention of hosting some of my code there in the future. So far, 100% of my open source activity happens on GitHub, and while they do a lot of things well, this seems like a good time to try alternatives considering the ongoing enshittification of major parts of the web.

I've been aware of Codeberg for a while, and after a friend (hi n0toose!) confirmed that the underlying software (Forgejo, a fork of Gitea) supports disabling pull request for a repository — a feature famously missing on GitHub — I was fully on board.

They optionally provide a hosted instance of Woodpecker CI for projects hosted on Codeberg, but I quickly ran into issues with workflows being slow and more importantly logs not loading, making the conversion from GitHub Actions a rather painful task. The other option is using Forgejo Actions, a relatively new feature building on top of act for compatibility with existing GitHub Actions workflows. However, Codeberg currently doesn't provide hosted runners for this type of CI, so I looked into hosting my own :^)

Setting up a Forgejo Runner

Installation of the Forgejo Runner is covered in their documentation and doesn't need me writing this post. They only describe running directly on the host and in LXC containers; but since my entire self-hosted software stack runs on Docker Compose this wasn't an option.

Adding the Docker Compose service

Luckily a Dockerfile is available, so I only had to figure out the configuration. Summarized we need the following:

I haven't found a pre-built variant of the docker image anywhere, so the first step is cloning the repo (to src/forgejo-runner/ in this case) and adding a service with a build entry
The ENTRYPOINT is a plain /bin/forgejo-runner, so we add the daemon argument
A volume mounting /var/run/docker.sock from the host into the container so containers can be spawned for running workflows

In YAML it looks like this:

services:
  forgejo-runner:
    build: ./src/forgejo-runner
    command: "daemon"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

At this point, we can register the runner with a Forgejo instance, in my case Codeberg. Follow the steps outlined here to obtain a registration token, then run:

docker compose run --rm --entrypoint sh forgejo-runner

…which will drop you into a shell within the container, from where we can invoke the registration:

forgejo-runner register

After completing all prompts, a file .runner should be created, the contents of which need to be copied to the host so we can mount it as a volume later.

Configuring the Forgejo Runner

Additionally, create a config file for the runner (config.yml) which will also be mounted as a volume and specified via a command line argument.

All available config options are explained in the docs, and if you're happy with the defaults you can skip this step. However, I'd definitely recommend changing runner.capacity to something greater than one, and if you want to keep caching enabled we'll need to tweak a few other things. Updated Docker Compose config:

services:
  forgejo-runner:
    build: ./src/forgejo-runner
    command: "daemon --config config.yml"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./volume/.runner:/.runner
      - ./volume/config.yml:/config.yml

While I'm using bind mounts here, you can of course use any other available method of putting files into the container!

Making Caching Work

One of the hardest problems in computer science, and no exception here. Caching (e.g. for various setup actions and actions/cache) is enabled by default, but the cache server will not be reachable from within the created workflow containers without intervention. This is because a random port is used by default (cache.port set to 0), which is not exposed on the runner container. I also had to tweak the container.network option to use Docker Compose's docker_default network which the runner container is reachable on.

Change this accordingly if your Docker networking setup is more involved :^)

This is what the config.yaml file looks like for the above:

cache:
  enabled: true
  host: forgejo-runner
  port: 8080
container:
  network: docker_default

Additionally, expose port 8080 on the container:

services:
  forgejo-runner:
    ...
    expose:
      - "8080"

Optionally, you can change the cache directory (/root/.cache) and/or create a separate volume for it.

Putting It All together

Starting the container should now allow the daemon to run and show up as active on the Forgejo interface:

For me, importing existing workflow files written for GitHub Actions worked seamlessly after making the following changes:

Moving them to .forgejo/workflows/ (optional but less confusing than keeping .github/workflows/ IMO)
Setting jobs.<job_id>.runs-on to docker (Labels in the screenshot above, not Name!)
Setting jobs.<job_id>.container.image to ghcr.io/catthehacker/ubuntu:act-22.04. Other images will work as well but might require additional setup steps, this one matches GitHub Actions closely.
Updating jobs.<job_id>.steps[*].uses URLs, e.g. uses: goto-bus-stop/setup-zig@v2 becomes uses: https://github.com/goto-bus-stop/setup-zig@v2. This is not needed for actions mirrored on Codeberg, e.g. actions/checkout (https://codeberg.org/actions/checkout).

Lastly, enable Actions in the repository settings, push some changes, and hope for the best :^)

This Week in Ladybird #4

2023-04-16T00:00:00Z

Overview of changes across the Ladybird Browser project in the week from April 10 to April 16 (plus everything merged Sunday evening) :^)

Summary

In chronological order:

Browser: Have `BookmarksBarWidget` signal bookmark changes for `Tab`

#18096 by @kemzeb

This ensures that the bookmark button is updated correctly when a bookmark for the currently active tab is changed:

#18172 by @krkk

This adds support for resolving the following CSS properties so that Window.getComputedStyle() can return their actual values:

background-attachment
background-clip
background-image
background-origin
background-position
background-repeat
background-size

LibWeb: Return from "the end" during HTML fragment parsing

#18224 by @Lubrsi

This adds an early return to the HTML parser's the end step when parsing HTML fragments, fixing some issues caused by the spec not matching web reality (Luke filed whatwg/html#8646 for that back in December).

It also features the longest commit message you will ever see for a two line change!

LibVideo+LibWeb: Generalize PlaybackManager for use within LibWeb...and use it

#18264 by @trflynn89

This updates the video element implementation in LibWeb to use the PlaybackManager class from LibVideo, which abstracts away the exact demuxer being used, provides buffering, and a few other things.

This is already being used in SerenityOS's VideoPlayer application, so future improvements once again benefit multiple parts of the system!

LibWeb: Implement a few methods on WritableStream and WritableStreamDefaultWriter

#18265 by @mattco98

More work on the Streams API, at this rate it might be feature-complete by the end of the month!

This implements the following APIs:

WritableStream:
- WritableStream.getWriter()
- WritableStream.abort()
WritableStreamDefaultWriter:

LibWeb: Implement WritableStreamDefaultWriter.write() and an AO involved in WritableStream's constructor

#18266 by @mattco98

This implements the WritableStreamDefaultWriter.write() API and various related AOs.

LibWeb: Convert video control dimensions from CSSPixels to DevicePixels

#18270 by @trflynn89

The video controls were initially developed on a HiDPI machine, and would thus look too large at 1x scaling. They now are sized in display-independent CSS pixels and converted to device pixels during painting.

LibWeb+AK: Get Google Street View working, along with SVG viewboxes, transforms, scaling, and hit testing!

#18271 by @MacDue

This PR implements all sorts of things with the goal of getting Google street View actually working, including looking and moving around!

And of course these features work in isolation too:

LibWeb: Track playback position and display a timeline on the layout node

#18281 by @trflynn89

We now keep track of the video's playback position and render a timeline accordingly. Seeking is not yet implemented, but this is of course a necessary prerequisite :^)

LibWeb: Begin adding ReadableByteStream interfaces

#18283 by @mattco98

This implements the following APIs:

ReadableByteStreamController:
- ReadableByteStreamController.byobRequest
- ReadableByteStreamController.desiredSize
ReadableStreamBYOBRequest:
- ReadableStreamBYOBRequest.view

As it turns out, BYOB stands for bring your own buffer and I'm somewhat surprised they named a Web API like that :^)

Nonetheless, very exciting to see some progress on the more efficient byte streams, too!

LibJS: Update spec numbers for the Intl NumberFormat v3 proposal

#18286 by @trflynn89

Much like this similar change from last week, the Intl NumberFormat v3 proposal has been merged into the main ECMA-402 spec, and we had to incorporate some final spec changes.

LibWeb: Implement URL.canParse(url, base)

#18287 by @networkException

This implements the brand new (last month) URL.canParse() API:

LibWeb: Convert `calc()` to spec's internal representation, fixing nested `calc()` in the process

#18289 by @AtkinsSJ

This re-implements calculated CSS values according to the CSS-VALUES-4 spec, fixing nested calc() in the process and allowing us to implement other functionality from that spec more easily later on.

For example, this is the internal representation of calc(100px + 30% - (120px / (2*4 + 3 ))):

LibWeb: Handle null values when making args for attributeChangedCallback

#18293 by @Lubrsi

This fixes incorrect handling of the DeprecatedString null state, which also returns true for is_empty() checks and would therefore create an empty JS string where a null value was expected.

This makes a lot of custom elements on the YouTube video page appear!

#18295 by @Lubrsi

We already create a basic DOM for images directly loaded via the URL bar so they can be shown in the web view, and now do the same for videos:

(video playback performance has improved a bit since then!)

LibWeb: Integrate ReadableByteStreamController into ReadableStreamController type

#18297 by @mattco98

More work on the Streams API. As a reviewer I'm very happy that Matthew is doing this incrementally :^)

LibJS: Comment all the things

#18336 by @linusg

I dusted off an old branch with the aim of adding spec comments to everything in LibJS/Runtime/ — we're not quite there yet, but it's a gradual process that often leads to small cleanups along the way :^)

LibJS: Make intrinsics getters and well-known symbol getters return NonnullGCPtr

#18343 by @linusg

Another ongoing effort in LibJS is updating raw pointers to either GCPtr or NonnullGCPtr — two lightweight wrappers around pointers to memory allocated on the JS heap and thus controlled by the garbage collector (GC).

In this case I updated all the well-known intrinsic objects (built-in constructors and prototypes mostly, written as %name% in the spec) and well-known symbols (@@name) to return NonnullGCPtrs instead of raw pointers:

- ArrayConstructor* Intrinsics::array_constructor();
+ NonnullGCPtr<ArrayConstructor> Intrinsics::array_constructor();

Even though many of the intrinsics are lazily allocated, calling their accessors is what causes that so a null pointer is never returned.

LibWeb: Whine instead of dying on unexpected box during line layout

#18344 by @awesomekling

This works around a VERIFY() (assertion) in the Inline Formatting Context that was slightly too optimistic and would cause crashes on various websites, including GitHub.

It's really annoying to have the entire web page crash because of some random unimplemented code path, so we largely avoid crashing macros like TODO() and VERIFY() in favour of no-op code paths that log a FIXME.

LibJS: Port more AOs to {Nonnull,}GCPtr

#18346 by @linusg

Doing the same as above to the return types of more AOs.

LibWeb: Don't match the root node of HTMLCollection

#18349 by @Lubrsi

When a HTMLCollection is created, e.g. via Element.getElementsByTagName(), it traverses the subtree from that element, which we have two slightly different methods for:

for_each_in_inclusive_subtree_of_type()
for_each_in_subtree_of_type()

It now correctly uses the latter to not include the element the method was called on.

LibWeb+Ladybird: Add support for more CSS cursors in Ladybird

#18359 by @srikavin

This adds support for cursor: not-allowed; in LibWeb, and maps more cursors from the LibGfx StandardCursor enum to their Qt equivalents in Ladybird (previously, only the hand, i-beam, and arrow cursors were mapped).

LibWeb: Set Comment's prototype

#18362 by @Lubrsi

Quite possibly the best bug silliness to fix effectiveness ratio we've had so far. Simply by setting the correct prototype of any instantiated Comment object, thumbnails started appearing on YouTube!

Luke wrote a summary of their investigation of this bug, check out the PR if you're curious :^)

LibWeb: Honor column-gap and row-gap CSS properties in flex layout

#18363 by @awesomekling

Another case of web reality not matching the specs; the PR description explains this change nicely:

This isn't actually part of CSS-FLEXBOX-1, but all major engines honor these properties in flex layout, and it's widely used on the web.
There's a bug open against the flexbox spec where fantasai says the algorithm will be updated in CSS-FLEXBOX-2: w3c/csswg-drafts#2336
I've added comments to all the places where we adjust calculations for gaps with "CSS-FLEXBOX-2" so we can find them easily. When that spec becomes available, we can add proper spec links.

LibWeb: Do not dereference empty Optional in ReadableStream::visit_edges

#18367 by @trflynn89

A bug fix for ReadableStream, which would dereference an empty Optional<T> (which causes a crash) during garbage collection if its controller hasn't been set yet.

LibWeb/Streams: Couple of small cleanups

#18370 by @linusg

Have you noticed I'm in a spring cleaning mood yet? :^)

LibJS: Comment all the things (continued)

#18374 by @linusg

More of that. I think all the built-in objects are covered now…

LibWeb: Introduce DocumentState and update SessionHistoryEntry

#18378 by @kalenikaliaksandr

Alexander has adopted and continued Andreas' refactor from browsing contexts to navigables so we can start merging those changes, this is the beginning of that :^)

Ladybird+LibWebView: Add -P/--enable-callgrind-profiling option

#18383 by @MacDue

This adds an option to Ladybird to run WebContent processes in valgrind with callgrind enabled, which is useful for profiling on Linux! 🐧

LibWeb: Scale SVG stroke-width based on viewbox

#18388 by @MacDue

This fixes the clipping of strokes in quite a few cases and now fixes the Gartic Phone logo :^)

LibWeb: Honor gap between flex lines when using align-content: stretch

#18400 by @awesomekling

As usual, layout changes are best explained by the author themselves :^)

We have to take the cross gap into account when calculating the "sum of flex line cross sizes" in "Handle 'align-content: stretch'".

LibWeb: Fix multi-line flex column layouts with auto height on container

#18403 by @awesomekling

When sizing a flex container with flex-direction:column under a max-content height constraint, we were incorrectly truncating the infinite available height to 0 when collecting flex items into lines.
This caused us to put every flex item in its own flex line, which is the complete opposite of what we want during max-content intrinsic sizing, as the layout would grow wide but not tall.

Screenshots

During his CSS calc() refactor, Sam had a happy little accident. Luke's been trying all sorts of things on YouTube that work now (after briefly breaking the thumbnails layout), which is really impressive! And of course, there's always that one website complaining about your ad blocker… we happen to have one. And something's wrong with our rendering of the mozilla logo :^)

And that's all for this week! Lots of nice progress, video playback is improving every day, Street View basically works (pending some perf improvements), and I'd not be surprised if Luke makes even more of YouTube work next week.

Looks like it's possible to build a browser after all :^)

This Week in Ladybird #3

2023-04-09T00:00:00Z

Overview of changes across the Ladybird Browser project in the week from April 3 to April 9 (plus everything merged Sunday evening) :^)

Summary

In chronological order:

AK+LibWeb: Port {HTML,UIEvents,XHR}::EventNames to new FlyString

#17825 by @kennethmyhra

This ports various event name enumerations (think mouseup, pageshow, …) from DeprecatedFlyString to FlyString.

LibWeb: Introduce CustomElementRegistry and creating custom elements

#18092 by @Lubrsi

This implements the CustomElementRegistry interface along with the underlying infrastructure required to create custom elements, the [CEReactions] IDL extended attribute, and the :defined CSS pseudo class!

LibWeb: Add borders functionality to CSS Grid

#18148 by @martinfalisse

Borders affect layout, and were previously not considered in a Grid Formatting Context. Now they are:

LibWeb: Implement `background-position-x` and `background-position-y` longhand longhands

#18157 by @MacDue

On a more serious note, the background-position-x and background-position-y longhands of the background-position property, which itself is a longhand of background have been widely supported in other engines for well over a decade (at least the single-value syntax), and are used on many websites, despite being part of a CSS spec that claims to be Not Ready For Implementation yet.

One such website is Steam (notice the left carousel button), before and after:

LibWeb: Fix background-position for multi-layer backgrounds

#18167 by @MacDue

This fixes a regression of multiple backgrounds rendering caused by the previous feature:

LibWeb: Implement multipart/form-data encoding algorithm

#18169 by @kennethmyhra

This implements the multipart/form-data encoding algorithm from the HTML spec, allowing us to use XMLHttpRequest.send() with FormData objects!

LibWeb: Use intrinsic aspect ratio when calculating max content height

#18173 by @matcool

This fixes images in a flex container with row direction being stretched when given a specific width, but auto height.

Before and after:

LibWeb: Add WritableStream and WritableStreamDefaultWriter interfaces

#18174 by @mattco98

Matthew is continuing his work on bringing the Streams API to LibWeb! This implements the following APIs:

WritableStream:
- The WritableStream() constructor
- WritableStream.locked
WritableStreamDefaultWriter:

LibWeb: A couple of error handling improvements

#18181 by @trflynn89

This is preparation for…

LibWeb: Begin support for videos with LibVideo

#18183 by @trflynn89

We finally have an initial implementation of video playback in LibWeb, thanks to Tim and everyone who previously worked on LibVideo for general video support in SerenityOS!

There's still a fair number of limitations (decodes within the page's WebContent process, no audio, hardcoded to use a Matroska demuxer and VP9 decoder, i.e. WebM) but at the same time it's of course 100% more functional than what we had before :^)

See (and I highly suggest trying, too!) for yourself:

LibJS: Update spec numbers for the Intl Enumeration proposal

#18194 by @trflynn89

The Intl Enumeration API (i.e Intl.supportedValuesOf() function) reached stage 4 and was merged into the main ECMA-402 spec, so we had to update some references.

LibWeb: Use max-width when present in TableFormattingContext

#18195 by @martinfalisse

Elements with display: table now respect max-width if given. Previously the created Table Formatting Context would take up the full available width.

LibWeb: Port a few IDL implementations to new String

#18197 by @kennethmyhra

This ports the following interfaces from DeprecatedString to String:

…as well as an internal class (AbstractBrowsingContext).

LibWeb: Ignore preferred width when calculating intrinsic width of block

#18204 by @awesomekling

Another week went by without me having any clue about layout stuff, so I'll let Andreas explain

When calculating the intrinsic width of a block-level box, we now ignore the preferred width entirely, and not just when the preferred width should be treated as auto.
The condition for this was both confused and wrong, as it looked at the available width around the box, but didn't check for a width constraint on the box itself.
Just because the available width has an intrinsic sizing constraint doesn't mean that the box is undergoing intrinsic sizing. It could also be the box's containing block!

LibWeb: Transform the default path in CRC2D#fill(CanvasFillRule)

#18206 by @Lubrsi

This fixes a bug where we didn't apply the canvas's current transform in CanvasRenderingContext2D.fill() when called with a fill rule, resulting in incorrect placement of the filled path.

LibWeb: Don't put abspos grid/flex items in anonymous wrapper

#18208 by @awesomekling

Grid and flex containers have their own rules for abspos items, so we shouldn't try to be clever and put them in the "current" anonymous wrapper block. That behavior is primarily for the benefit of block & inline layout.

LibWeb: Add an initial implementation of CRC2D.clip()

#18212 by @MacDue

This implements the CanvasRenderingContext2D.clip() API, resulting in major rendering improvements on Google Street View!

And even those little missing sections got fixed in the end:

LibWeb: Port {Custom,Mouse,UI,Wheel,}Event to new String

#18215 by @kennethmyhra

This ports the following classes from DeprecatedString to String:

LibWeb: Add the WritableStreamDefaultController

#18223 by @mattco98

This implements WritableStreamDefaultController with its two APIs:

LibWeb: Implement HTMLVideoElement `load`, `play`, `pause` and draw some basic controls

#18233 by @trflynn89

After video playback was initially implemented to start automatically and with no way of manually or programmatically controlling it, our implementation of the <video> element now gains play/pause controls, as well as the following APIs:

webp lossless: Implement pixel bundling for color indexing transform

#18235 by @nico

Nico has been (incrementally! ) writing a WebP decoder in LibGfx, which means it not only adds support for that format to virtually anything using images in SerenityOS, but also LibWeb!

With this, the webp lossless decoder is feature complete :^)

LibWeb/Fetch: Use a basic filtered response for redirect navigations

#18248 by @kalenikaliaksandr

This implements an upstream fix in the Fetch spec, which we need for Alexander's ongoing work on implementing the massive navigables refactor!

#18255 by @kennethmyhra

This ports the EventTarget interface from Deprecated{Fly,}String to {Fly,}String, as well as various internal classes and functions:

DOMEventListener
fire_progress_event()
fire_a_page_transition_event()
fire_a_synthetic_pointer_event()
fire_keyboard_event()
fire_webgl_context_event()

LibWeb: Implement WritableStream.close()

#18263 by @mattco98

This implements the WritableStream.close() API.

Screenshots

MacDue and Lubrsi spent some time visually debugging Google Maps, so here are some fun screenshots from that :^)

Even the smallest bug in an image decoder can yield images that are way off from the intended results, and Nico shared some wrongly decoded WebP images, too!

And that's all for week 3! I hope you're just as excited about the new video support as I am, more work for abstracting the specific codec away, as well as buffering are already in progress :^)

This Week in Ladybird #2

2023-04-02T00:00:00Z

Overview of changes across the Ladybird Browser project in the week from March 27 to April 2 (plus everything merged Sunday evening) :^)

Summary

In chronological order:

Ladybird: Open target _blank links in new tab

#18001 by @Coderdreams

This makes Ladybird (as in the Qt application) behavior match Browser (as in the LibGUI application) for the following actions:

target="_blank" links open in a new activated tab
Links clicked while Ctrl is pressed open in a new background tab
Links middle-clicked open in a new background tab

In the future we might make these configurable, but for now these are better defaults most people will be familiar with :^)

LibWeb: Implement custom element name validation and fix bug in element name validation

#18005 by @srikavin

We now validate the name of custom elements according to the spec so that valid names create an HTMLElement while invalid names create an HTMLUnknownElement.

LibWeb: Don't force relayout whole page on programmatic scroll

#18051 by @awesomekling

This implements two optimizations for page scrolling:

Reuse the layout tree if it is already up-to-date instead of always forcing a layout
Skip this completely when scrolling to (0, 0), which is always known to be a valid position and doesn't require an upfront layout update

Note that JS-invoked page scroll currently uses a different code path, so we're possibly missing out on this optimization in some places :^)

Ladybird: Enable and document build and debug steps with Xcode

#18052 by @ADKaster

I'm not an Xcode user myself, but it's always nice to have proper documentation for more development platforms! Now if someone could document the steps to debug a WebContent process with gdb… 😅

#18054 by @MacDue

We now tell you the current zoom level in both Browser and Ladybird, which you previously had to estimate yourself! The zoom actions have moved into a submenu for that to happen:

LibWeb: Stop returning the left padding for resolved padding-bottom

#18060 by @AtkinsSJ

I accidentally broke this 8 months ago and nobody noticed. 😅

I'm not sure how we managed to do that but thanks for the fix Sam!

LibWeb: Fix condition to early return in intrinsic sizing for TFC

#18064 by @kalenikaliaksandr

This fixes an incorrect auto height of a Table Formatting Context (TFC).

LibWeb: Handle flex column items with auto height that depends on auto width

#18066 by @awesomekling

Layout is like magic to me, so I'll trust you to understand this from the title alone :^)

LibWeb: Clamp fit-content widths in flex layout to min/max-width

#18072 by @awesomekling

More layout magic:

In situations where we need a width to calculate the intrinsic height of a flex item, we use the fit-content width as a stand-in. However, we also need to clamp it to any min-width and max-width properties present.

Tests/LibWeb: Add layout test for layout fix in PR #15780

#18074 by @Lubrsi

More tests = more good, even when the fix was from five months ago!

LibWeb+LibWasm: Implement and use the "reset the Memory buffer" steps

#18079 by @alimpfard

This implements reset the Memory buffer from the WebAssembly JS API spec, which is part of our checklist of features required for the Ruffle Flash Player emulator.

#18080 by @MacDue

Similar to Firefox we also show a button to quickly reset the current zoom level next to the URL bar when it is not 100%.

LibWeb: Parse Element.style url functions relative to the document

#18081 by @Lubrsi

url()s in CSS applied via HTMLElement.style are parsed now relative to that element's document URL.

This fixes the images on Steam's store carousel, for example:

LibWeb: Resolve percentage vertical-align values against line-height

#18084 by @awesomekling

Even more layout fixes. Andreas really did a lot of that!

LibWeb: Implement HTMLAnchorElement.text and .referrerPolicy properties

#18086 by @AtkinsSJ

This implements the following APIs:

HTMLAnchorElement.referrerPolicy
HTMLAnchorElement.text, an alias for Node.textContent

However we don't use the value of referrerPolicy yet when following the link.

LibWeb: Implement CRC2D.imageSmoothingEnabled

#18088 by @awesomekling

This implements the following APIs:

This means that pages can now choose whether we use the bilinear interpolation or nearest-neighbor interpolation (i.e. pixelated) when painting scaled images on a canvas. Previously bilinear interpolation was always used, matching the default of imageSmoothingEnabled = true.

Note that imageSmoothingQuality is not actually honored as LibGfx doesn't support this concept yet.

LibWeb: Remove redundant invocation of children changed in HTMLParser

#18098 by @awesomekling

A simple fix to avoid doing duplicate work, one of the easiest ways to make code faster :^)

Setting the data of a text node already triggers children changed per spec, so there's no need for an explicit call.

LibWeb: Don't re-sort StyleSheetList on every new sheet insertion

#18099 by @awesomekling

Another performance optimization:

This was causing a huge slowdown when loading some pages with weirdly huge number of style sheets. For example, amazon.com has over 200 style elements, which meant we had to resort the StyleSheetList 200 times. Instead of sorting, we now look for the correct insertion point when adding new style sheets, and we start the search from the end, which is where style sheets are typically added in the vast majority of cases.

And the result:

This removes a 600ms time sink when loading Amazon on my machine! :^)

Editor's note: Amazon really needs to get its websites under control. Andreas also shared the following screenshot earlier:

LibWeb: Actually visit rules and media queries in imported style sheets

#18101 by @awesomekling

This ensures that we also consider @media queries in style sheets loaded via CSS @import, which previously wouldn't happen due to a bogus getter:

bool has_import_result() const { return !m_style_sheet; }

LibWeb: Split CalculatedStyleValue out of StyleValue.{h,cpp}

#18104 by @AtkinsSJ

Another refactor of some CSS machinery, aimed at reducing dependency cycles and improving build times 🚀🚀🚀

LibJS: Add fast path to Value::to_u32() if Value is a positive i32

#18105 by @awesomekling

This change nicely demonstrates the kind of low hanging performance fruit we still have in LibJS!

6.6% speed-up on Kraken's stanford-crypto-aes subtest. :^)

This is the entire diff:

--- a/Userland/Libraries/LibJS/Runtime/Value.cpp
+++ b/Userland/Libraries/LibJS/Runtime/Value.cpp
@@ -947,6 +947,10 @@ ThrowCompletionOr<i32> Value::to_i32(VM& vm) const
 // 7.1.7 ToUint32 ( argument ), https://tc39.es/ecma262/#sec-touint32
 ThrowCompletionOr<u32> Value::to_u32(VM& vm) const
 {
+    // OPTIMIZATION: If this value is encoded as a positive i32, return it directly.
+    if (is_int32() && as_i32() >= 0)
+        return as_i32();
+
     // 1. Let number be ? ToNumber(argument).
     double number = TRY(to_number(vm)).as_double();

LibWeb: Begin implementing the Streams API

#18109 by @mattco98

With the exception of an empty ReadableStream interface, no one had done any work on implementing the Streams API yet — until now!

This implements the following APIs:

ReadableStream:
- The ReadableStream() constructor
- ReadableStream.locked
- ReadableStream.cancel()
- ReadableStream.getReader()`
ReadableStreamDefaultController:
ReadableStreamDefaultReader:

With those we can already run the Simple random stream example from MDN!

#18119 by @MacDue

A follow-up to the previous UI changes around zooming to make the new button's purpose more clear.

LibWeb: Don't churn HTML::EventLoop while in microtask checkpoint

#18125 by @awesomekling

This was the reason my Serenity VM's CPU was spinning at 100% for the entirety of the March update video (caused by having two GitHub tabs open) 😅

Luckily Andreas looked into it and found the cause: the event loop would constantly re-schedule itself even while we're performing a microtask checkpoint, except queued tasks wouldn't get processed in that state!

LibWeb: Fix application of intrinsic aspect ratio to flex column items

#18130 by @awesomekling

Only when using flex-direction: row{,-reverse} the intrinsic aspect ratio was calculated correctly, not when using a column{,-reverse} layout.

As many layout-related things this is somewhat vaguely defined in the spec, so it's easy to miss:

…then the flex base size is calculated from its inner cross size and the flex item’s intrinsic aspect ratio.

LibWeb+Tests: Allow subdirectories in layout_test.sh and add grid tests

#18134 by @martinfalisse

An initial set of 12 layout tests for CSS grid, which we use to ensure that changes don't regress layout — a rather complex part of the engine — by accident.

LibJS: Fix parsing of mixed `CoalesceExpression` and `Logical{AND,OR}Expression`

#18136 by @linusg

Bugs in the JS parser have mostly been flushed out over the years, but still occur at times; especially when we throw megabytes of JS from the web at it :^)

This specific bug prevented us from parsing an expression like this, which occurred on vscode.dev:

0 ?? 0 ? 0 : 0 || 0;

To be clear, I'm neither an expert on our JS parser nor do I really enjoy working on it, but I promised Andreas to fix the bug if he'd reduce it, which he did, so here we are. No test262 tests fixed, sadly 😄

LibWeb: Fix a few font scaling issues

#18137 by @MacDue

Even now that fonts scale properly with the CSS pixel mechanism we still occasionally find cases where device pixels are used. This fix corrected the scaling of text shadows and the markers of ordered lists (<ol>).

LibWeb: Don't apply element inline style & presentational hints to associated pseudo elements

#18144 by @awesomekling

This fixes a bug where inline styles (<foo style="...">) and presentational hints (e.g. <foo width="...">) would leak into the element's pseudo elements (::before, ::after) by accident — they should only apply to the element itself.

Screenshots

I didn't do this last week, but the browser engineer in me always gets excited when people share progress in form of a screenshot showing a mildly broken website so I think a section of random screenshots people shared is worth having 🥹

And that concludes the second post of the This Week in Ladybird series!

Also check out the SerenityOS update (March 2023) video if you haven't done that yet:

This Week in Ladybird #1

2023-03-26T00:00:00Z

Preface

Well hello friends! I've decided to start a This Week in Ladybird blog post series, a format you may already know from various other open source projects.

This will be in addition to my (almost-)monthly LibWeb/LibJS/Browser segment in our update videos, where I often have to exclude changes that are a little bit more technical or cannot be demo'd on camera very well.

The plan is to include all of these here once a week, both as a summarized list and briefly going into detail where I think it's warranted.

I'll of course be posting a link to each new post on Mastodon so follow me there if you want to get notified, or subscribe to this site's atom feed if that's your thing!

Overview of changes across the Ladybird Browser project in the week from March 20 to March 26 :^)

Summary

In chronological order:

LibWeb: Support more length types in SVG width/height attributes

#17922 by @MichielVrins

You can now specify the width and height of an <svg> element in any of the CSS <length> units, not just pixels.

LibWeb+LibGfx: Add support for scalable and theme-able checkboxes using SDFs

#17933 by @MacDue

After I recently changed checkbox painting from a bespoke implementation to a few lines of CSS in the user agent stylesheet, it's now changed back to a C++ implementation using SDFs to support seamless scaling and color theming, including via the accent-color CSS property!

LibWeb: Implement indeterminate checkboxes

#17940 by @srikavin

Previously we only supported the two obvious checkbox states, now the indeterminate checkbox state can be rendered as well! This was first implemented on top of the old checkbox style and later adopted by MacDue to use the new style (see above).

LibJS: Update spec numbers / text for the Change Array by Copy proposal

#17943 by @trflynn89

The Change Array by Copy proposal has been included in the main ECMA-262 spec, so we had to update various spec numbers/links and adopt a few adjustments from the final PR.

LibWeb: Support the :scope pseudo class

#17947 by @skyrising

This PR adds support for the :scope CSS pseudo class selector, which matches the element on which matches(), closest(), querySelector() or querySelectorAll() is called. MDN has some nice examples.

LibWeb: Make the Element.style setter work

#17949 by @skyrising

Element.style was previously implemented as a readonly property, we now support using it as a setter as well (which is a shorthand for setting Element.style.cssText):

element.style = "color: rebeccapurple;";
console.log(element.style.cssText);

LibWeb: Convert all tokenized features to a named enum

#17956 by @trflynn89

Tokenized features refers to the third parameter of window.open(), a comma-separated string of key=value pairs.

C++ lacks named arguments, so we prefer enums over plain booleans in many cases — I briefly noted this while reviewing a PR and Tim promptly fixed it :^)

-choose_a_browsing_context("_blank"sv, true, Web::HTML::ActivateTab::No);
+choose_a_browsing_context("_blank"sv, Web::HTML::TokenizedFeature::NoOpener::Yes, Web::HTML::ActivateTab::No);

LibWeb: Parse CSS identifiers case-insensitively

#17958 by @implicitfield

We now parse CSS tag name selectors so that HTML {}, Html {} and html {} are considered identical and match any casing of <html>.

LibWeb: Correct bugs in parsing of CSS unicode-ranges

#17989 by @AtkinsSJ

CSS numbers in scientific notation, as well as unicode-range values, are now parsed even more correctly than before!

LibWeb: Support loading file:// URLs via fetch (through ResourceLoader)

#17996 by @awesomekling

ResourceLoader is a (non-standard) layer for making individual HTTP requests, and sits at the very end of our Fetch API implementation. While Fetch itself does not specify how to load file:// URLs, ResourceLoader already supported it so only a little bit of plumbing was needed to make it work for Fetch as well.

This is important as stylesheet loading was recently implemented on top of Fetch, thus temporarily breaking local (file://) stylesheets.

LibWeb: Add workaround to restore cross-origin stylesheet functionality

#17997 by @Lubrsi

Another unexpected breakage by the stylesheet loading refactor, this time caused by a spec bug! Big thanks to Luke for filing an issue upstream and providing a workaround :^)

LibJS: Support the yy{/,-}mm{/,-}dd hh:mm format for Date

#18003 by @Lubrsi with a test fix by @Hendiadyoin1

It is not recommended to use any format other than subsets of the officially supported YYYY-MM-DDTHH:mm:ss.sssZ, but of course the web is gonna web and browsers have to follow suit. In a nutshell, every time we find a website feeding another previously unknown format to Date.parse(), we have to support that one as well.

LibWeb: Support parsing [HTMLConstructor] IDL attribute on constructors

#18004 by @srikavin

We now can parse the [HTMLConstructor] IDL extended attribute, but still throw an error when calling the generated function.

LibJS: Couple of small Intl editorials

#18007 by @trflynn89

Preparation for a few bigger upstream changes.

LibWeb: Implement performance.{mark,clearMarks,getEntries,getEntriesByType,getEntriesByName}

#18013 by @Lubrsi

Does exactly what it says on the tin! We now support the following functions:

This is used on YouTube, for example:

LibWeb: Create the correct error objects in XHR::handle_errors

#18016 by @Lubrsi

This fixes two typos that caused XMLHttpRequest to return the wrong types of error objects.

LibWeb: Run XML parser input through encoding decoder

#18021 by @kalenikaliaksandr

This fixes parsing of XML sources that start with a BOM, and generally allows us to handle non-UTF-8 encoded XML as well.

LibWeb: Add scalable radio buttons (with theme/accent-color support)

#18022 by @MacDue

In typical MacDue fashion, one painting PR is followed up by a few more! Radio buttons got the same visual treatment as checkboxes:

LibWeb: Split StyleValue.{h,cpp} into one set of files per StyleValue class

#18040 by @AtkinsSJ

Splitting one massive file into 34 smaller files? What's not to like!

LibWeb: Check all conditions of radio button groups

#18046 by @MacDue

This implements radio button grouping according to the HTML spec, fixing some issues we previously had: radio buttons unchecking checkboxes with the same name attribute, unchecking inputs across different forms, and treating no name attribute as a group.

And that's everything from this past week! Let me know if you enjoy this format or have any suggestions, and if you have a relevant PR open that hasn't been merged yet keep an eye out for next week's changelog 👀

Office Hours with Guest: Linus Groh

2023-01-27T00:00:00Z

Road to Working on SerenityOS & Ladybird Full-Time

2023-01-13T00:00:00Z

When I submitted my first pull request to the SerenityOS repo I had no idea where this journey would take me. Since then, almost three years and 3,200+ commits later, I have:

Learned C++, and a ton about systems programming in general
Helped building LibJS, our from-scratch JavaScript engine, from the ground up
Become the first appointed project maintainer
Joined TC39 as an Invited Expert to represent LibJS
Made a bunch of friends in our community :^)

You can read more about my history in the SerenityOS project in a previous blog post.

All of this I have done on the side, as a hobby project. I truly enjoy leading LibJS development, working on LibWeb, and everything else around it so I didn't mind spending a bunch of my spare time on it. The only issue here is that spare time is, well, limited. You might see where this is going… :^)

A New Chapter

It was clear this needed to happen eventually so I could keep up with the expectations I set for myself, but today it becomes official: I have now made it my goal to transition to working on SerenityOS, Ladybird, and open source in general full time!

As a first step I have reduced the hours at my regular job — beginning with one day a week for open source work, every week :^)

I will still do contributions in my spare time of course, but having dedicated full days allows me to tackle bigger projects and have more uninterrupted time.

It's not just doing more work for the sake of it, primarily I want to get Ladybird to a point where you & I can use it as an everyday web browser. This will take a lot of time and effort, but I think it's totally feasible!

Sponsorship

Of course I will need to find other sources of income before I can even think about fully committing, and GitHub sponsors is a big part of that! I joined back in June 2021 so people could support me financially if they wanted to, and while it definitely helped with covering domain costs and purchasing better hardware for making videos, I didn't want to advertise it much without a larger goal in mind.

Now I have such a goal, and the best way to help me achieving it is sponsorship on GitHub or Liberapay! A huge thanks to everyone already supporting me this way, or considering it now — I couldn't do the next part of this journey without you ❤️

Ladybird Browser Year in Review: What We Did in 2022!

2022-12-31T00:00:00Z

2022 is rapidly coming to an end, and of course we've been busy all year round in the SerenityOS project! Especially LibJS and LibWeb — also known as the Ladybird Browser Project — received a huge amount of work from dozens of contributors; and since they are what I work on and am most familiar with as well, let's look at some Ladybird highlights from the past year!

If you want to get a summary of the last 12 months of development for the whole SerenityOS project, take a look at the monthly update videos (30-40 minutes each).

ES Modules support in LibJS

This is from January, but easily one of my favourite LibJS changes of the entire year: davidot single-handedly added support for parsing, loading, and executing ECMAScript modules to LibJS! Actually, not just that, but some additional goodies like top-level await and dynamic import() as well — check out #11957 for a full list :^)

Initially limited to our standalone js interpreter, this was later followed up with a PR from networkException, who implemented support for <script type="module"> in LibWeb.

100/100 on Acid3

Acid3 is a popular, albeit a bit dated, browser test checking the compliance with features from a variety of web standards. I don't remember how we started chasing the full score, but a lot of people chipped in and we quickly went from 50/100 in February to 100/100 at the end of March, with Ali going as far as making an entire XML parser for XHTML support to get us that last point :^)

Shadows, Gradients, Filters

I think I still haven't fully grasped the amount of fancy CSS features that we started supporting in the last year, it's truly impressive! There's everything from antialiased rendering, to improved box shadows, support for all kinds of gradients, various backdrop-filters, and more. Most of this was done by MacDue!

The beginnings of WebGL

Usually we get to watch people when they work on cool projects in SerenityOS, but this one came out of nowhere:

<Lubrsi> my secret project worked out
<Lubrsi> everyone, I present to you, WebGL!
<awesomekling> bruh :poggie:

Yeah, Luke just did that. 1.5k line PR, approval from the folks working on LibGL, and done :^)

Not much has happened since then, but I hear the recent work on adding support for shaders to LibGL might be beneficial 👀

GC all the things

This effort was led by Andreas who wanted to do this back when he was working on WebKit, and now managed to do it in LibWeb instead :^)

It's all architectural and there's no screenshot for this one, but it's one of the nicest fundamental LibWeb changes we've had in years so it deserves a mention: basically we got to really embrace LibJS's garbage collector and replaced a whole bunch of ref-counted implementation classes, each having a GC'd wrapper object, with a single GC'd object where the implementation is the JS object itself and doesn't need a wrapper.

This also works for things that aren't directly exposed to user code, so we can fully avoid the ref-counting cycles there as well.

CSS Grid

Flexbox should be in every web developer's toolbox by now and we've had partial support for a while. Next up on the list is CSS Grid, which Martin started implementing in August and has continued to work on in the months since, making steady progress!

Ladybird: A new cross-platform browser project

…was the title of Andreas' blog post officially announcing the Ladybird browser project in September, but the journey started earlier! Initially in mid-2019 with LibHTML, the earliest iteration of what is now LibWeb; but also the cross-browser aspect of it, which is all thanks to Dexesttp who started porting LibWeb to Linux (and, by extension, other Unix-like platforms — still no Windows port) in March, resulting in a cross-platform, LibWeb-powered headless-browser utility that would later serve as the baseline for the Ladybird Qt GUI.

Since then, development of Ladybird as a cross-platform browser has of course led to better abstractions of platform-specific code in LibWeb, and even though it was a standalone project for a while, Andrew went ahead and imported it back into the mono repo after we broke the build with upstream API changes a few dozen times.

WebDriver and steps towards running WPT

WebDriver is an HTTP-driven interface for automated control of web browsers, and integral to running and processing results of test suites such as the Web Platform Tests in an automated fashion.

Implementing WebDriver support in LibWeb and both the LibGUI and Qt browser GUIs involved a whole bunch of people: we had an initial PR from Orphis back in May, which Sam then revived and finished in October, followed by implementing most of the endpoints with lots of help from TobyAsE, and finally integrating it with Ladybird by Tim, as well as lots of tweaks in the initial architecture and code structure to make things fully work and allow sharing most code between different embedders of LibWeb.

There's still a few things missing, but overall this progressed nicely and allows any WebDriver client to leverage this new functionality!

With lots and lots of hacks I already made the Browser run a single test via WPT, but running the entire suite needs more work. Next year!

LibJS representation at TC39

Back in April littledan reached out and asked if we're interested in participating in TC39, and of course we were! Fast forward to November and I became an Invited Expert to represent the LibJS engine, and even managed to travel to A Coruña for their most recent in-person meeting.

Attending for the first time definitely had something surreal to it as I didn't know anything about JS engines before starting to work on ours less than three years ago, but I'm super proud that we got to this point!

Bring-up work on complex web apps

Developing unrelated features in isolation only gets you so far, trying to get some real websites to load is when you truly see how far along the browser has (or hasn't) come. We've seen support for many major websites mature over the year (source: Andreas' Twitter feed 🐦), but what's even more impressive is when web apps start to load as those are usually vastly more complex in every category imaginable!

One of those was improving support for Discord, our main communication hub, initially done by Andreas as a video series earlier in the year. More recently, Luke has done a few as well, and I think we could call it a series at this point!

Obviously all of these are far from fully working, but it's exciting to see that bad performance is slowly () replacing missing APIs as one of our biggest issues!

Fetch API

I implemented support for Promise in LibJS back in 2020-2021, which is the foundation for many modern Web APIs, such as fetch(). Funnily enough I also predicted that I would eventually implement that, but for a while nothing happened beyond a few failed attempts.

My website was (and still is) using fetch for dynamically getting information from the GitHub API though, and the only way of getting that to work in Ladybird was to implement fetch() so I started doing that (yes, this is how a surprising number of things get implemented). It took a few months working on infrastructure and the JS API on and off, but eventually I arrived at a working fetch() method :^)

It's not completely to spec as we don't have the Streams standard implemented, but the foundations are solid and already used in other places (navigation, XHR) as well, as intended by those specs. I'm super happy that I managed to implement such a fundamental part of the modern web in LibWeb at last, but I also have to give some credit to Luke for their thorough testing and bug fixes!

CSS Pixels

This one is still in progress, but I'm eagerly looking forward to it as I'm writing this on a HiDPI laptop which makes websites look tiny compared to the window frame with our old assumption of 1px in CSS is one pixel on the display:

Sam, our local CSS expert, has gone and converted the entire LibWeb CSS layer to a new CSS Pixel unit, which then converts to physical pixels only when rendering. This leads to much nicer results than just 2x scaling a website rendered with the old approach, and has probably been supported by the browser you're reading this in for years. Soon also coming to a Ladybird Browser near you :^)

2023

Of course we'll continue working on new Web APIs, performance, layout, rendering, and everything else there is to making a browser! As usual there's no roadmap so I can't tell you what exactly will and won't be implemented, but I'm excited to see what everyone decides to hack on next year :^)

If you want to help out with development definitely join the Discord server, and for semi-regular updates follow me on Mastodon. You can also sponsor me on GitHub so that I can do more of this!

Happy new year! 🎉

SerenityOS Office Hours with Guest: Linus Groh

2022-09-02T00:00:00Z

Trying Real™ Websites in the SerenityOS Browser

2022-07-07T00:00:00Z

Well hello friends! After the recent release of Ladybird, a prototype browser GUI written in C++/Qt using the LibWeb engine, SerenityOS and specifically its browser engine subproject got quite a bit of attention on the internet again — everything from the usual "nooo you can't build a web browser from scratch" and "y tho", to excitement, to heise online suggesting we're "competing against Google" now, and while this is not a goal we set for ourselves, it's a noble one! :^)

I know better than to engage with the negative comments, but it did spark my interest in seeing where we're actually at right now in terms of usability. Now, being one of the developers, I obviously test websites all the time; but on the other hand browser devs like artifical test cases — colorful boxes, Lorem Ipsum, or the infamous border-radius-shadow-egg.

Getting these individual building blocks to work nicely on their own is a lot easier than the result of combining them into a modern website. To make things even more difficult, I'm only going to try landing/login pages, which are usually much more complex than your average "text in a box" pages. Custom fonts, plenty of images, complex layouts, you name it.

Setup

I did not prepare for this in any way, I'm simply using a regular build of Serenity, at commit b3deec0.

To compare each result to the expected rendering in a mature browser, I'm running Firefox 101 on Fedora, i.e. the Gecko browser engine 🦎. I did change the user agent string to Serenity's (Mozilla/5.0 (SerenityOS; x86_64) LibWeb+LibJS/1.0 Browser/1.0) to make sure I'm being served the same content. Google for example falls back to their old/legacy/simple version of search (not sure what they call it) when using an unrecognized UA string.

All pages were visited via a VPN connection into the US.

Results

Below are the screenshots of websites I tried: some of them are the most visited websites worldwide, and some are random ones of things I like & use. Enough talking, let's have a look!

Summary

As you might have noticed, I'm rating quite favorably here, always keeping in mind that this is a from-scratch engine that just turned three years old, mostly developed by hobbyists in their spare time; and Andreas :^)

The top two poorly performing features are clearly images not appearing consistently (which could be anything from using methods to load/apply them that we don't support to using formats we can't decode) and custom fonts (same reasons). And then of course there's a plethora of layout issues everywhere.

Of course most of these are far from perfect, and (hopefully) no one is claiming otherwise. However, just a year ago or so many of these would have crashed the browser or rendered a blank page (especially when JS language support was much weaker), so personally I find the progress we have made so far convincing enough to believe that we'll have results in a couple of years that are worthy of saying: "they're the same picture".

And even before then, I'm looking forward to doing this again in a few months to see what's changed!

SerenityOS Office Hours / Q&A with Linus Groh

2022-06-03T00:00:00Z

How To Build a Standalone GUI Application for SerenityOS

2022-05-11T00:00:00Z

Building a GUI application as part of SerenityOS is easy, and has been done plenty of times. Just take a look at any of the existing ones, or read the (slightly outdated!) Introduction to SerenityOS GUI programming on Andreas' blog.

But what if you wanted to build one that exists as a standalone project and is not part of the SerenityOS repository, and instead installed as a port, or not open source in the first place? That's also possible, but not entirely straightforward — especially when using GML, Serenity's very own GUI Markup Language.

Creating an Example Application

The file structure for our example application looks like this:

example
├── CMakeLists.txt
└── src
    ├── ExampleWindow.gml
    └── main.cpp

With src/main.cpp being a very simple Serenity C++ program using LibGUI and LibMain:

#include <LibGUI/Application.h>
#include <LibGUI/Window.h>
#include <LibMain/Main.h>
#include <ExampleWindowGML.h>

ErrorOr<int> serenity_main(Main::Arguments arguments)
{
    auto app = TRY(GUI::Application::try_create(arguments));
    auto window = TRY(GUI::Window::try_create());
    auto widget = TRY(window->try_set_main_widget<GUI::Widget>());

    widget->load_from_gml(example_window_gml);
    window->set_title("Example");
    window->resize(200, 100);
    window->show();

    return app->exec();
}

And the GML in src/ExampleWindow.gml used to define layouts and widgets:

@GUI::Widget {
    fill_with_background_color: true
    layout: @GUI::VerticalBoxLayout {
        margins: [4]
    }

    @GUI::Button {
        text: "Well Hello Friends!"
    }
}

The result looks like this:

Now, how do we turn this into an executable compiled for SerenityOS?

CMake Shenanigans

You might have noticed that we include a file in our program which I didn't mention yet, ExampleWindowGML.h, which is where the example_window_gml variable is defined. Generally speaking, the load_from_gml() function just takes a StringView, so we could define our GML source in the C++ file and pass it to that function directly.

That's ugly though and kind of defeats the point of trying to separate the GUI declaration from the program logic. So instead, we let CMake generate a header file from our GML source file!

This is all handled by the SerenityOS build system, which defines the compile_gml() CMake function, which in turn invokes a simple shell script outputting code. The result isn't very exciting, and yet this is a useful and commonly used piece of glue code:

extern const char example_window_gml[];
const char example_window_gml[] = R"~~~(@GUI::Widget {
    ...
}
)~~~";

A regular CMakeLists.txt for a SerenityOS GUI application including this GML-to-C++ transform looks like this:

cmake_minimum_required(VERSION 3.16)

project(Example)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

compile_gml(src/ExampleWindow.gml ExampleWindowGML.h example_window_gml)

set(SOURCES
    src/main.cpp
    ExampleWindowGML.h
)

add_executable(Example ${SOURCES})
target_link_libraries(Example gui main)

install(TARGETS Example RUNTIME DESTINATION bin)

In an actual application you'll also want to add some compiler flags, such as these in the Lagom CMakeLists.txt.

We set up the project using C++20 as required by the SerenityOS libraries, "compile" the GML into a generated header, specify that in SOURCES along with main.cpp, define an executable built from those files, link it against LibGUI and LibMain, and finally install it.

All of this is run by a handful of standard CMake/Ninja build commands, pointing it at Serenity's generated CMakeToolchain.txt:

mkdir -p Build
cd Build
cmake .. -GNinja -DCMAKE_TOOLCHAIN_FILE="${SERENITY_SOURCE_DIR}/Build/${SERENITY_ARCH}/CMakeToolchain.txt"
ninja
ninja install

So far, so good — right? Unfortunately not:

CMake Error at CMakeLists.txt:9 (compile_gml):
  Unknown CMake command "compile_gml".

More CMake Shenanigans

Within SerenityOS everything is set up so that you don't have to deal with where and how to make these extra functions available in any regular CMakeLists.txt, but here we're on our own.

compile_gml() is from Meta/CMake/code_generators.cmake, so let's include() that (using the SERENITY_SOURCE_DIR environment variable found in various other build scripts):

set(SerenityOS_SOURCE_DIR $ENV{SERENITY_SOURCE_DIR})
include(${SerenityOS_SOURCE_DIR}/Meta/CMake/code_generators.cmake)

SerenityOS_SOURCE_DIR is a variable referenced by the function itself and is usually set by CMake based on the project() name.

Trying again:

CMake Error at /path/to/serenity/Meta/CMake/code_generators.cmake:18 (add_dependencies):
  Cannot add target-level dependencies to non-existent target
  "all_generated".

  The add_dependencies works for top-level logical targets created by the
  add_executable, add_library, or add_custom_target commands.  If you want to
  add file-level dependencies see the DEPENDS option of the add_custom_target
  and add_custom_command commands.
Call Stack (most recent call first):
  CMakeLists.txt:12 (compile_gml)

This is an implementation detail of the build system: all generated files are added to a custom all_generated target. Let's simply define that as well, even though we're not going to use it:

add_custom_target(all_generated)

Now we get to a point where the compiler gets involved, but fails with:

FAILED: CMakeFiles/Example.dir/src/main.cpp.o
/path/to/serenity/Toolchain/Local/x86_64/bin/x86_64-pc-serenity-g++ --sysroot=/path/to/serenity/Build/superbuild-x86_64/../x86_64/Root   -std=c++20 -MD -MT CMakeFiles/Example.dir/src/main.cpp.o -MF CMakeFiles/Example.dir/src/main.cpp.o.d -o CMakeFiles/Example.dir/src/main.cpp.o -c /path/to/example/src/main.cpp
/path/to/example/src/main.cpp:4:10: fatal error: ExampleWindowGML.h: No such file or directory
    4 | #include <ExampleWindowGML.h>
      |          ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

It's our generated GML header! Let's check if it actually exists:

$ file Build/ExampleWindowGML.h
Build/ExampleWindowGML.h: C source, ASCII text

So we got that working, but the compiler doesn't know where to find it. Since the file simply ends up in the build directory, we can add CMAKE_CURRENT_BINARY_DIR to the include directories (which is also what Serenity's root CMakeLists.txt does):

include_directories(${CMAKE_CURRENT_BINARY_DIR})

And with that it finally builds and installs the executable into Serenity's Root/usr/local/bin directory! Let's confirm:

$ cd "${SERENITY_SOURCE_DIR}/Build/${SERENITY_ARCH}/Root/usr/local/bin"
$ file Example
Example: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/lib/Loader.so, with debug_info, not stripped
$ objdump -p Example | grep NEEDED
  NEEDED               libgui.so.serenity
  NEEDED               libm.so
  NEEDED               libgcc_s.so
  NEEDED               libc.so

NOTE: LibMain is statically linked.

Once you rebuild the QEMU _disk_image, the externally built example application will be available within the VM.

For completeness, here is the full working CMakeLists.txt:

cmake_minimum_required(VERSION 3.16)

project(Example)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

set(SerenityOS_SOURCE_DIR $ENV{SERENITY_SOURCE_DIR})
include(${SerenityOS_SOURCE_DIR}/Meta/CMake/code_generators.cmake)
add_custom_target(all_generated)
include_directories(${CMAKE_CURRENT_BINARY_DIR})

compile_gml(src/ExampleWindow.gml ExampleWindowGML.h example_window_gml)

set(SOURCES
    src/main.cpp
    ExampleWindowGML.h
)

add_executable(Example ${SOURCES})
target_link_libraries(Example gui main)

install(TARGETS Example RUNTIME DESTINATION bin)

Conclusion

By replicating a few things from Serenity's build system in our own CMakeLists.txt, we can get a code generator working that was never explicitly intended to be used outside of the project :^)

I haven't tried, but this should also be applicable to other kinds of code generators in case you need that. And, of course, if you saw anything that could be streamlined here, please let me know!

Running SerenityOS on Bare Metal

2022-02-19T00:00:00Z

In a previous post, I mentioned:

I haven't managed to do a full successful boot on bare metal yet though, so that will definitely have to join this list one day.

Well, five months later I finally did it! :^)

View tweet: https://twitter.com/linusgroh/status/1486111320181149697

This is a post explaining what it took to get there, both as documentation for myself and hopefully to help someone else to get Serenity up and running on their hardware!

Hardware

While I also tried running SerenityOS on my laptop in the past (ThinkPad T450S), all serious attempts were on my desktop machine, as the availability of a serial port is highly important for debugging — especially when the kernel panics before even setting up a framebuffer and console for graphical output, and you only end up with a blank screen after booting. The alternative to this is beeping the PC speaker…

This is also the machine I do recording and some development on, so I didn't want to mess with the Linux partitions and ended up just giving Serenity its own SSD.

Motherboard: MSI B350 PC MATE
CPU: AMD Ryzen 5 1600x
Graphics card: NVIDIA GeForce RTX 2060
RAM: 4 x 8 GB DDR4 (Crucial)
Disk: Serenity/Linux each on their own SSD (also Crucial)
Monitor: 27 inch QHD (Dell), connected to the GPU's HDMI output (CPU has no integrated graphics!)

Setup

Note that I won't explain the general bare metal installation process here, that's all outlined in Documentation/BareMetalInstallation.md.

I usually keep my local checkout up to date, to get the latest kernel and userspace changes. This can mean that things that used to work suddenly stop working, and another investigation is required. That's ok though, as things improve for the most part. :^)

A recent example of this is an update to the i8042 initialisation process that would fail on my machine (I8042: Trying to probe for existence of controller failed), causing the mouse and keyboard to not work. Fixed in #12641.

For the record though: I'm using commit eca8208 as of writing this. I also tested i686 and x86_64, and they both boot.

Building & Creating a Bootable Disk

Basically: apply patches (discussed later), do a regular build, create GRUB disk image, dd to disk, reboot.

Serial Console

I use an adapter cable that's permanently connected to the 9-pin JCOM1 connector on the motherboard and has an RS-232 connector on the other end. Then I plug in an RS-232 to USB cable whenever I need to and connect from my laptop by running cu -s 57600 -l /dev/ttyUSB0 in a terminal, where I will then see exactly what QEMU would show on the debug console.

Input Devices

While trying to get Serenity to boot I already knew I wouldn't be able to actually use it once successful — USB HID devices aren't working in the kernel yet. The recommended first step is checking if the BIOS has an option to emulate PS/2 input devices; mine doesn't, but luckily the computer has a PS/2 port, so I ordered a cheap PS/2 mouse and keyboard. In an ironic twist, both the USB mouse and keyboard start working on Serenity once their PS/2 counterparts are plugged in, so the BIOS does have some sort of emulation, just doesn't seem to expose an option for it. I haven't investigated further.

Required Tweaks

As mentioned before, I still have to do some small modifications for Serenity to actually boot.

Additional Kernel Options

While all of these could be provided in GRUB on demand, I think it makes sense to hardcode them if the boot is known to fail without them.

serial_debug: This should be self explanatory. I boot with serial debug enabled by default for now, but disabling it greatly decreases boot time, so I won't keep it forever. Can still be enabled manually in GRUB of course.

enable_ioapic=off: Without this option, I used to get the following panic:

0.000 [Kernel]: APICTimer: Using HPET as calibration source
[Kernel]: KERNEL PANIC! :^(
[Kernel]: Unhandled IRQ
[Kernel]: at ./Kernel/Arch/x86/common/Interrupts.cpp:532 in void Kernel::unimp_trap()

This was fixed in #12182, but still causes the boot to hang indefinitely when on.

ahci_reset_mode=aggressive: I wish I tried this option much earlier — many of my previous attempts failed at the stage where it was trying to locate the root file system, with the following panic:

[init_stage2(1:1)]: ASSERTION FAILED: !m_storage_devices.is_empty()
[init_stage2(1:1)]: ./Kernel/Storage/StorageManagement.cpp:153 in void Kernel::StorageManagement::enumerate_disk_partitions()
[init_stage2(1:1)]: KERNEL PANIC! :^(
[init_stage2(1:1)]: Aborted
[init_stage2(1:1)]: at ./Kernel/Arch/x86/common/CPU.cpp:35 in void abort()
[init_stage2(1:1)]: Kernel + 0x00c39df1  Kernel::__panic(char const*, unsigned int, char const*) +0xf1
[init_stage2(1:1)]: Kernel + 0x00fbb2d9  abort.localalias +0x245
[init_stage2(1:1)]: Kernel + 0x00fbb094  abort.localalias +0x0
[init_stage2(1:1)]: Kernel + 0x011c37f9  Kernel::StorageManagement::enumerate_disk_partitions() [clone .localalias] +0xdf9
[init_stage2(1:1)]: Kernel + 0x011c92c5  Kernel::StorageManagement::initialize(AK::StringView, bool, bool) +0xb5
[init_stage2(1:1)]: Kernel + 0x00f9d4ff  Kernel::init_stage2(void*) +0x11af
[init_stage2(1:1)]: Kernel + 0x00fc3a50  exit_kernel_thread +0x0

The different reset mode fixes that!

So the patch to add all of them to the first GRUB menu entry looks as follows:

--- a/Meta/grub-mbr.cfg
+++ b/Meta/grub-mbr.cfg
@@ -2,7 +2,7 @@ timeout=1

 menuentry 'SerenityOS (normal)' {
   root=hd0,1
-  multiboot /boot/Prekernel root=/dev/hda1
+  multiboot /boot/Prekernel root=/dev/hda1 serial_debug enable_ioapic=off ahci_reset_mode=aggressive
   module /boot/Kernel
 }

Note that in many circumstances you will also need to specify a different value for the root option, especially in a dual boot environment or when having multiple disks. I simply reordered the connected disks so that Serenity is connected to SATA1 and the default option (root=/dev/hda1) just works. :^)

Increasing `INITIAL_KMALLOC_MEMORY_SIZE`

With the default value of 2 MiB, the kernel panics:

[Kernel]: ASSERTION FAILED: m_has_value
[Kernel]: ././AK/Optional.h:157 in T& AK::Optional::value() & [with T = KmallocGlobalData::ExpansionData]
[Kernel]: KERNEL PANIC! :^(
[Kernel]: Aborted
[Kernel]: at ./Kernel/Arch/x86/common/CPU.cpp:35 in void abort()

Various kernel changes have been made over time to reduce the amount of initial kmalloc memory that's needed, but it's still not enough in my case.

Increasing it to 8 MiB fixes the panic:

--- a/Kernel/Heap/kmalloc.cpp
+++ b/Kernel/Heap/kmalloc.cpp
@@ -23,7 +23,7 @@ static constexpr size_t CHUNK_SIZE = 32;
 static constexpr size_t CHUNK_SIZE = 64;
 #endif

-static constexpr size_t INITIAL_KMALLOC_MEMORY_SIZE = 2 * MiB;
+static constexpr size_t INITIAL_KMALLOC_MEMORY_SIZE = 8 * MiB;

 // Treat the heap as logically separate from .bss
 __attribute__((section(".heap"))) static u8 initial_kmalloc_memory[INITIAL_KMALLOC_MEMORY_SIZE];

Enabling the `MULTIBOOT_VIDEO_MODE` Flag

This is needed as I rely on Multiboot (which we use when booting in combination with GRUB) to prepare a framebuffer for the kernel. As far as I understand, QEMU & co. use a different approach, so it's not enabled by default.

--- a/Kernel/Prekernel/Arch/x86/multiboot.S
+++ b/Kernel/Prekernel/Arch/x86/multiboot.S
@@ -3,7 +3,7 @@
 .set MULTIBOOT_PAGE_ALIGN,    0x1
 .set MULTIBOOT_MEMORY_INFO,   0x2
 .set MULTIBOOT_VIDEO_MODE,    0x4
-.set multiboot_flags,         MULTIBOOT_PAGE_ALIGN | MULTIBOOT_MEMORY_INFO
+.set multiboot_flags,         MULTIBOOT_PAGE_ALIGN | MULTIBOOT_MEMORY_INFO | MULTIBOOT_VIDEO_MODE
 .set multiboot_checksum,      -(MULTIBOOT_MAGIC + multiboot_flags)

 .section .multiboot, "a"

Changing the WindowServer Screen Resolution

The default WindowServer screen resolution is 1024x768. However, the framebuffer we ask Multiboot to set up for us has a resolution of 1280x1024 by default:

/* for MULTIBOOT_VIDEO_MODE */
.long 0x00000000    /* mode_type */
.long 1280          /* width */
.long 1024          /* height */
.long 32            /* depth */

While it's possible to change these values (which would be great, as the monitor resolution is much larger), any attempts to do so resulted in a 640x480 framebuffer. Checking vbeinfo in a GRUB console revealed that no larger resolution is supported by this particular VBE setup:

grub> vbeinfo
List of supported video modes:
Legend: mask/position=red/green/blue/reserved
Adapter `VESA BIOS Extension Video Driver':
  VBE info:   version: 3.0  OEM software rev: 144.6
              total memory: 16384 KiB
  Ox101  640 x  480 x  8 ( 640)  Paletted
  0x103  800 x  600 x  8 (1024)  Paletted
  Ox105 1024 x  768 x  8 (1024)  Paletted
  Ox107 1280 x 1024 x  8 (1280)  Paletted
  Ox111  640 x  480 x 16 (1280)  Direct color, mask: 5/6/5/0  pos: 11/5/0/0
  Ox112  640 x  480 x 32 (2560)  Direct color, mask: 8/8/8/8  pos: 16/8/0/24
  Ox114  800 x  600 x 16 (2048)  Direct color, mask: 5/6/5/0  pos: 11/5/0/0
  0x115  800 x  600 x 32 (4096)  Direct color, mask: 8/8/8/8  pos: 16/8/0/24
  Ox117 1024 x  768 x 16 (2048)  Direct color, mask: 5/6/5/0  pos: 11/5/0/0
  Ox118 1024 x  768 x 32 (4096)  Direct color, mask: 8/8/8/8  pos: 16/8/0/24
  Ox11a 1280 x 1024 x 16 (2560)  Direct color, mask: 5/6/5/0  pos: 11/5/0/0
  Ox11b 1280 x 1024 x 32 (5120)  Direct color, mask: 8/8/8/8  pos: 16/8/0/24
grub>

Changing the resolution in DisplaySettings doesn't do anything either (see #4414).

This situation will likely improve in the future with more native graphics drivers (we already have one for Intel graphics), but for now I set the monitor to a 5:4 aspect ratio and apply this patch to make use of the full available resolution:

--- a/Base/etc/WindowServer.ini
+++ b/Base/etc/WindowServer.ini
@@ -5,8 +5,8 @@ MainScreen=0
 Device=/dev/fb0
 Left=0
 Top=0
-Width=1024
-Height=768
+Width=1280
+Height=1024
 ScaleFactor=1

 [Fonts]

Demo

With all of these changes, it finally boots! The only major issue I found is that time advances way too quickly — we'd have to adjust it via NTP every now and then. Everything else seems to work just fine, including networking :^)

Thanks!

None of this would've been possible without Idan, Liav, Luke & all the other folks in the #bare-metal channel on our Discord server. I needed quite a bit of hand holding to understand the various panics and find workarounds or fixes for them!

Implementing the Temporal Proposal in LibJS

2021-11-29T00:00:00Z

Over the past five months we've implemented the Temporal stage 3 proposal in LibJS, the SerenityOS JavaScript engine. In this case, we is Idan, Luke, and yours truly.

We're just three functions (& a bunch of bugfixes 🐛) short of having the minimum implementation completed, so I figured I'd write down some thoughts about the whole process :^)

Motivation

After finishing my rewrite of the Object implementation back in June, I was looking for another project to work on. I first discovered the Temporal proposal around the same time; and since we had already experimented with implementing proposals and not just the stage 4 ECMA-262 spec before, I decided that was going to be it.

commit 8269921212875b75c918bde5d318f0c5de152dac
Author: Linus Groh <mail@linusgroh.de>
Date:   Tue Jul 6 19:14:47 2021 +0100

    LibJS: Add the Temporal namespace object :^)

    Currently empty, but we gotta start somewhere! This is the start of
    implementing the Temporal proposal (currently stage 3).

    I have decided to start a new subdirectory (Runtime/Temporal/) as well
    as a new C++ namespace (JS::Temporal) for this so we don't have to
    prefix all the files and classes with "Temporal" - there will be a lot.

    https://tc39.es/proposal-temporal/

I can't speak for others, but Idan and Luke joined me shortly after; and it's been a team effort ever since :^)

OK, How Much Code?

$ git rev-parse --short HEAD
7a27ecc135
$ scc Userland/Libraries/LibJS/Runtime/Temporal
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
C Header                    34      2049      343       215     1491          6
C++                         34     17693     3700      5467     8526       1695
───────────────────────────────────────────────────────────────────────────────
Total                       68     19742     4043      5682    10017       1701
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop (organic) $303,647
Estimated Schedule Effort (organic) 8.744090 months
Estimated People Required (organic) 3.085114
───────────────────────────────────────────────────────────────────────────────
Processed 996444 bytes, 0.996 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────

What's Special

When we started working on Temporal, LibJS was just a little over a year old, and in that year we learned a lot about what works well for us, and what doesn't.

Here are a couple of things that didn't follow the status quo:

Code Structure

In LibJS all of the ECMAScript built-in functions and objects reside within the Runtime/ directory — clearly inspired by JavaScriptCore, as Andreas worked on WebKit for years. As mentioned in the commit message above, I decided to make a new subdirectory for Temporal specifically, and we ended up doing the same for Intl. In both cases it nicely matches the JS API, as both of them have namespace objects, and avoids polluting the top-level directory with even more files.

JSC did not end up doing the same, FWIW :^)

Additionally, everything is enclosed in a new C++ namespace, JS::Temporal (we only had JS so far: JS::Object, JS::Value, etc.):

Within the Temporal namespace, objects are referred to by their name only, e.g. PlainDateTime. This is clear enough in all cases.
Outside of that but within LibJS, we need to qualify the namespace, e.g. Temporal::PlainDateTime — just like Temporal.PlainDate in JS.
Outside of LibJS, e.g. in the js(1) REPL, the full namespace is required, e.g. JS::Temporal::PlainDateTime.

Again, we also did the same for Intl, and would do it for any other large namespace objects added to the language specification.

Finally, we strictly put all functions in their respective files split up by spec sections. No special reason other than consistency and zero need for bikeshedding.

Spec comments

Shortly before all of this I really started to embrace using spec comments in the respective implementation code, i.e. literally copying the whole spec text into the implementation. Yes, seriously.

Looks like this:

// 2.3.4 SystemDateTime ( temporalTimeZoneLike, calendarLike ), https://tc39.es/proposal-temporal/#sec-temporal-systemdatetime
ThrowCompletionOr<PlainDateTime*> system_date_time(GlobalObject& global_object, Value temporal_time_zone_like, Value calendar_like)
{
    Object* time_zone;

    // 1. If temporalTimeZoneLike is undefined, then
    if (temporal_time_zone_like.is_undefined()) {
        // a. Let timeZone be ! SystemTimeZone().
        time_zone = system_time_zone(global_object);
    }
    // 2. Else,
    else {
        // a. Let timeZone be ? ToTemporalTimeZone(temporalTimeZoneLike).
        time_zone = TRY(to_temporal_time_zone(global_object, temporal_time_zone_like));
    }

    // 3. Let calendar be ? ToTemporalCalendar(calendarLike).
    auto* calendar = TRY(to_temporal_calendar(global_object, calendar_like));

    // 4. Let instant be ! SystemInstant().
    auto* instant = system_instant(global_object);

    // 5. Return ? BuiltinTimeZoneGetPlainDateTimeFor(timeZone, instant, calendar).
    return builtin_time_zone_get_plain_date_time_for(global_object, time_zone, *instant, *calendar);
}

We didn't always do that. In fact, the majority of LibJS's code is still without any spec annotations. It's useful for a number of reasons though, so I wanted to take it to the next level in Temporal — 100% of its code is annotated like this, and I'm not going back.

While the usefulness of // 9. Else, specifically remains a source of debate within the SerenityOS community, we have started to also use this approach in parts of LibWeb, for example.

Initial Lack of Tests in Test262

For a while now we've used test262 to aid with testing our own implementation. Despite that, we still use and continue to expand our own test suite, which became especially important again for Temporal: a considerable amount of tests initially written for the polyfill only got merged on October 1. Before that, test coverage was really scarce, so we wrote plenty of tests ourselves:

$ git ls-files Userland/Libraries/LibJS/Tests/builtins/Temporal | wc -l
298
$ git grep 'test(' Userland/Libraries/LibJS/Tests/builtins/Temporal | wc -l
995

Test262 coverage of Temporal will be expanded further in the future, and currently contains a number of broken tests, so take this with a grain of salt: at the time of writing, we pass ~85% of their tests.

Exposure to the ECMAScript Proposal Process & TC39

In the past we've only ever taken finished, mostly stable specs & proposals for our implementation. Temporal still being Stage 3 meant that we eventually found, reported, and in some cases fixed more than a dozen issues with the specification itself, which we encountered during development (as well as relying on other implementors doing the same).

While most of the actual decision making for any normative changes happens within TC39 meetings, it was nice to get involved a little bit and see who is making the specs and how they do it :^)

What's not Special

What absolutely did not change was how we approached this project. As you might know, the only plan in the SerenityOS project is not having a plan, so while one of the spec's champions kindly pointed us to a dependency graph he made for the JSC implementation, we still ended up picking functions at random and implementing the missing Abstract Operations (AOs) on the fly.

Rinse, repeat, stop once all functions, and — by necessity — AOs are implemented.

Other than perhaps being slightly more consistent than other parts of LibJS, we still used the same language (C++), coding style, patterns (e.g. TRY()), existing engine mechanisms (PrototypeObject, ThrowCompletionOr, & co.). It will all feel very familiar if you already know the rest of the codebase.

What's Next

There's still a lot of work to do! The spec requires support for a single named timezone (UTC, numeric timezone offsets like +01:30 are a different story but part of the core functionality as well) and calendar (iso8601). To be more more widely useful, we'll need support for other named timezones (e.g. Europe/London) as well as calendars (e.g. gregory, the full list can be found on MDN). These would be sourced from the IANA Time Zone Database and the Unicode Common Locale Data Repository (CLDR) respectively, we already heavily utilize the latter for various functionality in ECMA-402 (Intl).

Speaking of Intl, there's a bunch of changes to be made to integrate Temporal with Intl objects, e.g. Intl.DateTimeFormat. ~~Since we don't have that implemented yet, we didn't make those changes either.~~ Just a day later, Tim opened a PR starting to implement exactly that. It's still based on the vanilla Intl spec, though.

Lastly, it's still a stage 3 proposal and will probably receive further small editorial and normative changes for several more months, which we're obviously keeping up to date with.

SpiderMonkey, JavaScriptCore, and V8 are all working on their implementations, and I imagine Temporal will eventually ship unflagged in the major browser engines sometime in 2022 — it's gonna be great. Until then, you can already use it in the SerenityOS Browser today :^)

Talking to SerenityOS Contributors About a Scratch-built C++ Developer’s Playground in Modern C++

2021-11-12T00:00:00Z

Learning C++ With Serenity

2021-09-02T00:00:00Z

My Journey With SerenityOS, So Far 🐞

2021-08-22T00:00:00Z

This is a timeline of personal highlights of my involvement in the SerenityOS project — from learning C++ to becoming a maintainer and beyond.

If you have any questions, feel free to get in touch, for example on the Discord Server. :^)

2019 — Discovery

I first heard about SerenityOS in late 2019, but cannot remember where and when exactly — probably Reddit, Twitter, or a YouTube recommendation. I started watching Andreas' hacking videos occasionally and found them very enjoyable.

2020-01-26 — First PR

My first ever contribution was PR #1137, titled LibVT: Rename escape functions. 12 commits, +57 −35 changeset.

This was inspired by a remark in one of the videos and subsequent commit. It seemed easy enough for me to attempt, so I did. It was a one-off though, and I went back to just watching.

2020-03-13 — First LibJS PR

PR #1439 was my first ever contribution to LibJS, or to any JavaScript engine for that matter. A simple change resolving a FIXME.

A long time before that, I — like so many others — believed that you can't build a browser from scratch anymore, they're too complex. Sure, SerenityOS already had a simple browser, but as far as I was concerned that already existed when I first saw it, like all the other browsers. When Andreas uploaded a YouTube video titled Browser hacking: Let's build a JavaScript engine for SerenityOS! it clicked for me. I had been wrong, I just watched a guy doing what I considered to be impossible, starting from nothing.

I still didn't know any C++ though, but after a week the urge to take on this challenge/chance became strong enough to dive into it. I could work on a brand new JS engine, where literally all the things still had to be implemented. I'm a web developer professionally, so I was excited — and that provided enough motivation to get started. Needless to say, I'm glad I did, otherwise I'd probably still be watching this project from the outside.

2020-04-08 — Starting the Promises PR

A few weeks into the first lockdown in the UK and with more free time on my hands than ever before, I needed something to work on. That something was implementing JavaScript Promises in LibJS.

At that point I still barely knew what I was doing when it comes to C++, so I spent countless hours trying to understand compiler errors and debugging silly mistakes. I got pretty far and opened PR #1706 after a few days, earning lots excitement, but then got stuck, and eventually closed the PR after having it sit around for a while, thinking I couldn't finish it (I probably could have, with some help).

Luckily this story will have a happy ending!

2020-05-17 — 100 Commits

About two months after starting to contribute regularly, I reached 100 commits in the project, and became the fifth person to join the list of contributors in the README.

70 of these commits are work on LibJS and the js REPL :^)

At this point things calmed down for a little while, as I went on vacation in July/August and was quite busy with work in September/October.

2020-12-29 — Crash Reporter

This is a fun little project I started during the last couple of days in 2020 (PR #4626).

Here's what the initial version looked like, and what it looks like now (as of August 2021):

Even though I've worked on various GUI applications in SerenityOS, the Crash Reporter remains the only one that I wrote myself.

2021-01-18 — Python 3.9 Port

For a long time I was a bit uncomfortable with contributing to Serenity's ports, as I had never ported software to a new system before. After working on a couple of small ports (gnuplot, neofetch, as well as fixes for other existing ones), I attempted a big one: Python 3.9.

At the time we already had a Python 3.6 port, but it was several releases behind upstream, experimental, and didn't compile with threading support enabled. So, I started over, and it was a success! The initial port was done in PR #5001, with several follow-up PRs afterwards.

Since then I've been keeping the port up to date with the latest Python releases and gradually enabled more features (which mostly boils down to getting as many modules as possible to compile). Python is the first programming language I learned and still the one I know the best, and as such it's my favourite non-core part of the system. 🐍

View tweet: https://twitter.com/linusgroh/status/1351251082601787396

2021-02-13 — Developer Interview

Andreas asked me if I'd be interested in doing an interview in early January, and without much hesitation I agreed. I had never done anything of this sort before, so I was slightly nervous, but it was a great experience :^)

Other developer interviews followed, you can watch those in this playlist.

2021-04-02 — Finishing the Promises PR

Mar 26 09:40:04 <kling> linusg: I used a JS promise at work the other day and it made me remember you had a LibJS promise thing going a while back
Mar 26 09:41:21 <linusg> I did never finish it, the "schedule this asap but not immediately" part threw me off IIRC
Mar 26 09:42:53 <linusg> I'll rebase it and then we can figure that out :)
...
Mar 29 20:02:04 <linusg> kling: thanks for nudging me towards picking up the promises PR again, it's been a fun couple of days :D

I rewrote some parts of it, fixed some other parts, posted an up-to-date screenshot — and then it got merged, almost exactly one year after starting it. I was stoked :^)

During this one year, the question of when LibJS would have Promise support came up several times, and the answer always was "when someone picks it up (again)". I still wanted to be that person.

This PR remains one of my all time favourites, not only because it's a really cool JS feature and basic requirement for many modern Web APIs, but also because of what it represents for me personally: going from poking at a new JS engine and trying to somehow make it work, to later turning that initial failure into a success and realising I can actually do this.

View tweet: https://twitter.com/awesomekling/status/1377724079696924676

2021-04-05 — 1000 Commits

A little over a year after starting to contribute regularly, I reached 1000 commits in the project. Andreas tweeted this screenshot of a DM I had sent him, and I couldn't be happier about proving myself wrong!

View tweet: https://twitter.com/awesomekling/status/1379127438823817220

2021-04-10 — Discord

For a long time all the SerenityOS developers were only on IRC — and it was great. Any requests to start a Discord server were dismissed, until we tried it one day after a good experience with a Discord server of the Zig community, and it turns out: we were so wrong.

There was a concern that Discord would have more memes and nonsense than productive discussion, and to some extent that appeared to be true — there was a lot of noise and excitement in the beginning. Now the Discord server has grown to almost 4000 people and about two dozen channels, and it's more active and productive than IRC ever was (it had less than 200 people on it). Oh, and it also has more yak emotes than IRC.

My ZNC is still running, but I haven't connected to it in a long time, especially since the freenode drama. It was fun while it lasted :^)

2021-04-16 — Becoming a Reviewer

Andreas did an interview with Jonathan Turner (Systems With JT), and the topic of dealing with the ever growing list list of pull requests came up. Later that day I Andreas asked me if I wanted to be a reviewer. Of course I did!

It took me a couple of days to get comfortable with this new position as well as overcoming the initial fear of somehow messing up. Over the next couple of weeks my code contributions plummeted as I spent more time on code reviews instead.

View tweet: https://twitter.com/awesomekling/status/1383081852857155592

If one person doesn't scale, two won't either, so we were soon after joined by Ali, and then Gunnar.

2021-06-30 — Montly Update Video

Much to my surprise Andreas asked me if I wanted to do a part of his monthly SerenityOS update video, specifically to talk about LibJS. It already was the last day of the month though, so there was very little preparation time 😅

It worked out nonetheless!

2021-07-31 — Montly Update Video, Again

This time with way more preparation, a proper microphone, and covering both LibJS and LibWeb.

The Future

As you know, there is no plan!

Well, that's not entirely true: I do plan to use Serenity as my primary OS eventually. I haven't managed to do a full successful boot on bare metal yet though, so that will definitely have to join this list one day.

This is the most fun and ambitious project I have ever worked on, and I'm proud to be a part of it. It's been keeping me busy during lockdown in 2020, and has been growing closer to my heart ever since. Being exposed to the exceptional can-do attitude has been invaluable. In other words: I'm here to stay :^)

SerenityOS developer interview: Linus Groh

2021-02-13T00:00:00Z