JavaScript in JavaScript
The ongoing project to run the Web Browser Engineering browser in your browser has already involved writing a Python-to-JS compiler and writing a mock networking library using async/await. That was enough to get Chapters 1–8 working.
But for Chapter 9 and 10, we needed to run JavaScript, and that was more of a challenge than expected.Not Eval
Basically, this should be easy—“how do I run JavaScript in a web browser”—that’s not hard, that’s what browsers do! JavaScript even has an eval function that runs JavaScript source code! It’s the Python version of the browser that needs additional libraries to run JavaScript.
But it turned out to be harder than expected. Running JavaScript isn’t hard, but isolating it from the rest of the browser is.
This gets a little confusing, so remember that there’s two browsers here: a host browser, which is Chrome or Firefox or whatever you’re running, and a virtual browser which is the compiled-to-JavaScript version of the WBE browser. We want to run JavaScript in the virtual browser by downloading a virtual script; set up a virtual execution environment by defining virtual constants like document and Node; and then run the virtual JavaScript in that virtual environment to modify the virtual page. But if I run the virtual JavaScript with eval, it’ll attempt to access the host document and Node variables and modify the host browser’s host page.
We can get around this using the JavaScript with command or the even weirder distinction between direct and indirect eval
document and Node are shadowed by the virtual definitions. But unless we shadow everything the host defines (and we won’t, browsers are constantly adding new host definitions) the virtual script can probably “escape” the scope and start mucking with global state. Not good.Isolating with Web Workers
Luckily, JavaScript does provide a workable isolation mechanism in the form of Web Workers. Web Workers are basically share-nothing threads for JavaScript; each Web Worker runs in an isolated environment and has no access to the environment that created it. That sounded perfect for my use case.
I set up a simple Web Worker based JavaScript execution library like so. First, when you create a JSInterpreter, I create a new Web Worker:
class JSInterpreter {
constructor() {
this.worker = new Worker("/widgets/dukpy.js");
this.worker.onmessage = this.onmessage.bind(this);
this.promise_stack = [];
}
}
To run a script in the interpreter, we just send it over with postMessage:
class JSInterpreter {
evaljs(code, replacements) {
return new Promise((resolve, reject) => {
this.promise_stack.push(resolve);
this.worker.postMessage({
"type": "eval",
"body": code,
"bindings": replacements,
});
});
}
}
I send over the source code we want executed and then return a Promise. That Promise will resolve when the virtual script finishes executing and wants to return a value; for that reason I save its resolve function for later.
On the WebWorker side, when we receive this message we just run eval:
addEventListener("message", (e) => {
switch (e.data.type) {
case "eval":
dukpy = e.data.bindings;
let val = eval?.(e.data.body);
if (val instanceof Function) val = null;
$$POSTMESSAGE({"type": "return", "data": val});
break;
}
});
Basically, we call eval on the body of the message, using the indirect eval style for reasons that I remember losing an hour to but I don’t remember the details of.
postMessage, where the weird name is to avoid a script overwriting postMessage and making everything break.Finally, back in the virtual browser, we receive this message and resolve the promise:
class JSInterpreter {
async onmessage(e) {
switch (e.data.type) {
case "return":
this.promise_stack.pop()(e.data.data);
break;
}
}
}This basically works, and lets me execute JavaScript in a Web Worker and receive the results.
The DOM
But the virtual browser running virtual scripts is just one half of JavaScript support. The other half is virtual scripts calling into the virtual browser. Specifically, we want virtual scripts to be able to call browser APIs like querySelectorAll or innerHTML.
The challenge here is pretty mundane: these APIs are synchronous, in that you call them without using await and expect results in the form of values, not Promises. We can send an API call request to the virtual browser no problem, using postMessage. But inside a Web Worker, we can’t receive data from the main script (which runs the virtual browser) except by receiving a postMessage back, and that requires us to first finish executing the current script. We need some form of synchronous cross-thread communication, and JavaScript mostly doesn’t provide that.
Mostly! It turns out there are at least two support methods for communicating across Web Workers without using messages: localStorage and SharedArrayBuffers.
I haven’t tried the localStorage route, and it seems janky, so let’s talk about SharedArrayBuffer. This feature was originally added to JavaScript to facilitate stuff like WebAssembly and WebGL, so it’s kind of obscure. Basically, you can allocate a SharedArrayBuffer and send it, via postMessage, to a Web Worker. Then both the main script and the Worker can modify it, and those modifications are immediately visible in both threads. Moreover, the Atomics set of functions allows various forms of locking, blocking, and atomically modifying the shared buffer.
This is perfect for making calls from virtual scripts to the virtual browser. The virtual script can send a postMessage describing the API call it wants to make, and then wait on a prearranged SharedArrayBuffer for the return value. In code, this looks like this:
function call_python() {
let args = Array.from(arguments);
let fn = args.shift();
$$POSTMESSAGE({
"type": "call",
"fn": fn,
"args": args,
});
Atomics.wait($$FLAGARRAY, 0, 0);
let len = $$FLAGARRAY[0];
Atomics.store($$FLAGARRAY, 0, 0);
if (len > 0) {
let buffer = new Uint8Array(len);
for (let i = 0; i < buffer.length; i++) {
buffer[i] = $$READARRAY[i];
}
let result = JSON.parse(new TextDecoder().decode(buffer));
return result;
}
}There’s a lot of code here, but it’s ultimately not that bad. The first block of code sends a API call request to the virtual browser. The second block, with an Atomics.wait instruction, blocks the virtual script until a pre-arranged “flag array” is changed. Once it is, a length is read from the flag array, and then that many bytes are read from a pre-arranged “read array” which are decoded from JSON to be the return value of the API call.
(Recall that in the book’s Python version of all of this, API requests like querySelectorAll just return a list of integer handles, which are then wrapped on the JavaScript side into Node objects, so JSON is more than rich enough to represent all the data we need to pass back and forth.)
On the virtual browser side, things are similarly not too bad: when the virtual browser receives a call message, it calls the requested function with the requested arguments, encodes the result to JSON, and stores it in the read buffer before updating the flag buffer:
class JSInterpreter {
async onmessage(e) {
switch (e.data.type) {
case "call":
let fn = this.function_table[e.data.fn]
let res = await fn.call(window, ... e.data.args);
let json_result = JSON.stringify(res);
let bytes = new TextEncoder().encode(json_result);
if (bytes.length > this.write_buffer.length)
throw new JSExecutionError(e.data.fn);
for (let i = 0; i < bytes.length; i++) {
Atomics.store(this.write_buffer, i, bytes[i]);
}
Atomics.store(this.flag_buffer, 0, bytes.length);
Atomics.notify(this.flag_buffer, 0);
break;
}
}
}Here there are four steps. First, run the requested function. Next, encode it to JSON and check that there’s enough space in the buffer to store it all.
Finally, store the result to the read buffer and then unblock the waiting Worker withAtomics.notify.By the way, in response to an API call, the virtual browser might execute more JavaScript—which is why the promises used when returning have to be a stack.
Security
So after a few hours of debugging this all worked, and things were basically good—except for the fact that every browser turns off SharedArrayBuffer for security! This is part of the fallout from Spectre, a class of vulnerability where very accurate timing measurements can reveal information about other processes executing on the same CPU. The current state of things is that you must choose between SharedArrayBuffer and iframes.
The Web Browser Engineering book uses iframes for a bunch of stuff, but most prominently it uses them to embed the JavaScript browser into the book. So we can’t turn off iframes. But we need SharedArrayBuffer to run JavaScript in the embedded browser. A dilemma.
For now, we’ve gone with a pretty limited solution. The JavaScript-enabled virtual browser isn’t embedded into the book pages. Instead, there’s a link to a separate page containing only the virtual browser. On that page, iframes aren’t used and SharedArrayBuffer is enabled instead.
Frankly, this isn’t an ideal solution, so one of the things on my todo list is to investigate using localStorage for synchronous communication between threads instead. If anyone has done that, I’d love to hear more (how long does it take? do you just busy wait?). Otherwise, uhh, please don’t use anything described in this post in production. It’s definitely a bad idea. But do enjoy the JavaScript-enabled browsers linked from Chapters 9 and 10!
Perhaps I’ll write about the mock Tkinter implementation another time.
Honestly, one of the ugliest features of the web platform that I’m aware of.
Basically I need to make sure that if one eval message defines a constant, another eval message can use it, and for that I need the definition to occur in the global scope, not the local scope of this one execution of the message handler.
You might be wondering why I test if the return value is a function. The only situation in our browser, and in real browsers mostly, where the return value of eval is examined, is the true/false return value of an event handler, which determines whether the default behavior is run. But functions would be returned accidentally if the last statement in a script were a function definition, so here I filter that out.
The shared read/write array has some fixed length, so it’s possible the return value overflows that, in which case this code throws an error. Given how limited the book’s browser’s capabilities are, I’m not too worried about this.
I think you probably don’t need to use Atomics.store for each byte here, but I didn’t want to learn about the SharedArrayBuffer memory model and performance is not a concern.
You make the choice via the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy HTTP headers. There’s some work ongoing to make these policies a little looser, but right now it’s pretty much a strict choice between SharedArrayBuffer and iframes.