The books are really nicely made, with a super cute cover on the Korean one and and some fun animal typography (it’s O’Reilly) in the Japanese one:

The translators also surely did a great job, because they found a lot of small typos and mistakes in the English-language version of the book and were nice enough to file GitHub issues.

We’re thrilled to have readers in so many countries and, if you’re a publisher, we’re happy to work with you on more translations as well!

I hope you’re enjoying Web Browser Engineering and maybe even considering gifting a copy to a friend for the coming holidays. We have a new Korean edition out in case you’d like one, and there’s more translations in the works.

Meanwhile, I did want to mention a few changes we made to the book to keep up with some of our dependencies changing or going obsolete.

First, Skia. Chapter 11 (on visual effects) and every later chapter uses the Skia 2D graphics library for drawing page content and the browser UI. Until now, we’ve specifically required Skia version 87, which is very, very old; the most recent release is milestone 144. So we’ve updated the text to work with more recent versions.

The main change is in Chapter 15 in the implementation of the CSS image-rendering property. Basically, in Skia 87, this used an API called FilterQuality, but more recent Skia versions instead use an API called SamplingOptions. It’s not a big change, and we have porting notes in case you want to keep using Skia 87, but the big benefit is that while FilterQuality was an unstable API, SamplingOptions is stable so it’s guaranteed to work in future Skia versions as well. This means we can now recommend readers install the current Skia version instead of asking you to install an old, obsolete version.

Second, playsound. This was a simple Python library to play audio, which we used in Chapter 14 (on accessibility) to implement a really simple screen reader. But unfortunately, it looks like playsound doesn’t work on some combination of recent Python versions and recent macOS releases, and we’ve heard some complaints from users on other OSes as well.

Since browsers don’t typically include their own screen readers, and since playing audio is actually totally secondary to the main point of the chapter, we’ve decided to remove playsound from the main chapter text, and just recommend users print the screen reader text to the console. There’s a new exercise that describes how to build a somewhat more sophisticated screen reader, including support for polite and assertive interruptions for live content.

As always, thank you all for reading, and do consider gifting Web Browser Engineering this holiday season. The book should be available on all of the typical sites like Bookshop.org, Barnes & Noble, and Amazon (referral link). And if you’ve got some free time over the holidays, consider hacking a bit on your favorite open source web browser!

Web Browser Engineering, our book about how web browsers work (and how to write your own, from scratch) has been out for about half a year now, and we’ve just got the first number from the publisher on how many we’ve sold.

The initial print run for the book was 75 hardback books and 305 paperbacks. We’ve now busted through both, having sold 76 hardbacks and 518 paperbacks (and hopefully a few more by the time you get this!), so we’re now on our second printing!

As always, thank you all for reading and buying the book. If you’ve been holding off, or if you have a friend you think would like it, now’s a good time! The book should be available on all of the typical sites like Bookshop.org, Barnes & Noble, and Amazon (referral link).

Alessandro Molina is the main developer behind DukPy, the Python bindings for the JavaScript interpreter we use in the book. Besides putting out new releases and making sure DukPy is available for modern Python versions, Alessandro very kindly accepted a patch improving one of the error messages we saw readers hitting. The underlying DukTape engine itself is maintained by Sami Vaarala.

Kota Yamaguchi and Hin-Tak Leung are the maintainers of skia-python, which the book uses for advanced graphics starting in Chapter 11. Maintaining skia-python is especially challenging, because Skia is a fast-moving project with a lot of breaking changes. Both helped determine which Skia version to target in WBE, and Hin-Tak made several special-purpose releases of the chosen version, m87, for newer Python versions. M87 is quite old and just building on modern Python and C++ needs patches, so I’m especially thankful for his expertise.

Austin Hurst seems to be the current maintainer of PySDL2, which the book uses as the base GUI library from Chapter 11 on. Mike C. Fletcher maintains the PyOpenGL library, which WBE uses in Chapter 13 to do GPU-accelerated rendering. Both libraries have vast APIs and yet Chris and I had no troubles with either libraries, nor have any of our readers reported issues, which really speaks to their stewardship.

The playsound and gTTS libraries are used in Chapter 14 to demonstrate a basic screen reader. Taylor S. Marks wrote playsound, though I don’t know if he still maintains it. Pierre Nicolas Durette maintains gTTS, quite actively in fact, as new languages are added to Google’s TTS service.

And of course, the Python language and standard library, maintained by the Python Software Foundation, is what makes WBE possible at all. In particular it is extremely helpful that Python comes with a large and stable standard library, especially the ssl, tkinter, socket, and threading packages. Of course, those packages are in turn built on open-source libraries like Tcl/Tk and OpenSSL.

So, again, a thank you to all of you!

The story of the web and the story of open source technology are fundamentally intertwined, as we discuss in the History chapter. All browsers today are open source, and part of the reason for this is that they rely on a vast ecosystem of other open source projects: graphics packages, networking code, cryptography libraries, image and video formats, user interfaces libraries, and so so much more.

Moreover, it’s the availability of open-source servers (Apache, NGINX), programming languages (Perl, PHP, Python), and operating systems (Linux, BSD) that have enabled the web to grow so large, diverse, and widely used. And these open source projects are often organized using the web! So, to open source in general, without which not only would WBE not be possible, it would not even be worth writing: thank you!

Buy from Oxford University Press

That said, the book is also available from plenty of other resellers. In the US, Bookshop.org, Barnes & Noble, and Amazon all sell the book, and popular bookstores in other countries likely carry it as well. We’ve confirmed that some readers have been able to purchase the book (through Amazon) in Canada, for example, and Amazon currently advertises delivery in about a week.

If you’re new to the book, you can preview the full book on our website and, if you like what you see, please order a copy. Or, if you’re already a fan, maybe forward this email to a friend who you think will like the book.

But if you haven’t ordered a copy, now’s a good time. The best place to order one is directly from the publisher; the paperback is $50 in the US (plus shipping; £28 in the UK).

Web Browser Engineering teaches you how web browsers work, in the best way possible: by building your own. Some of my personal favorite bits are:

• Implementing error-correct HTML parsing in Chapter 4

• Adding link navigation and multiple tabs in Chapter 7

• Implementing JavaScript click and submit callbacks in Chapter 9

• Adding a separate browser and main thread in Chapter 11

• Using transforms for iframe rendering and hit-testing in Chapter 15

We’ve written the book to make the code as transparent and simple as possible, and then we’ve listed fun exercises implementing real web browser features (like custom fonts, CORS, z-index, and :hover) that you’ll be able to do after reading the book.

Check out the Web Browser Engineering website to read it online and, if you like what you see, please do order a copy.

The book feels great—lightweight, easy to read, and the cover looks even better than I expected. My understanding is that these will start shipping out to pre-orders sooner rather than later, I think first in the UK and then later in the US and elsewhere.

If you’ve already pre-ordered, thank you so much! The demand has been quite a bit more than I expected, and I’m thrilled to see so many people excited to learn about what goes on inside a web browser. If you haven’t yet, you can do so on Oxford’s website or on a variety of other book-buying sites like Amazon. And if you’re not sure if you’ll like it, the whole book is available online, for free, and will continue to be.

Chris and I will now be shutting down the Patreon. The book is finished, and our Patreon supporters were a big part of how it happened. We appreciate the support over multiple years! Thank you Randy Naar, Min Lee, Zachary Tatlock, Jonas Treub, Alexandru Nedel, Adam Gutglick, Swav Rybak, Rishi Chopra, Yuanhang Xie, Shuhei Kagawa, Vitor Roriz, Maia X., Parker Henderson, Tiago Pereira, Liza Daly, Sangyeob Han, YongWoo Jeon, Jess, Martin Minkov, Peter Rushforth, Gowtham K, Ryo Ogawa, and JaviFML.

The proof process focused on a lot of things, like getting all the punctuation in the right place, making sure all the code blocks are correctly indented, and things like that. But one particularly tricky bit was making sure Chapter 11, where we talk about color mixing, makes sense both when the book is printed in black and white and when the book is viewed in full color online. This required modifying some figure captions and adding footnotes, where appropriate, reminding people to check the website for full-color images. The book will come out in print form with no color, a color version on Oxford University Press’s web page, and as always on web.browser.engineering.

As soon as we have a hard shipping date, I’ll publish an update for y’all. It’s been a long journey! Chris and I are extremely excited that soon we’ll hold Web Browser Engineering in our hands. Hopefully you are too.

And if you are extra-excited, please do preorder the book, either on the publisher’s web page (preferred) or from Amazon.

I suppose while I’m here I might as well relay a funny story. The book’s website is hosted on a Digital Ocean droplet (supported by our wonderful Patreons) for $4/mo. Amazingly, it handled the traffic flood just fine, with CPU usage at around 10–20% when I checked on it. But our log parser GoAccess, which we use to figure out how many visitors the site has, kept getting OOM-killed because the log was too big, so I was keeping track mostly by checking my network usage in the Digital Ocean dashboard.

Today traffic died down and I finally got the log parser running. It looks like about 22,000 people checked out the book, mostly people who’d never visited the site before. That’s a huge number of people getting excited about web browsers! Digging into the data a bit, most visitors of course only looked at the table of contents, but a big fraction started reading the first chapter and a sizable fraction of those presumably finished it and went on to the second.

So, again, hello new readers and a big thank you to long-time fans for supporting this work over the years!

Finally, a quick status update—we’re waiting for a second and hopefully final proof from the publishers, after which printing can start. I’m a little unclear on the exact timeline, but we still hope to have books sent out to people who pre-ordered before the end of the year.

If you pre-order, you’ll likely get your copy before the end of the year, perhaps as a present for your wintertime holiday of choice. The book comes in hardback and paperback, beautifully typeset with a lovely cover:

Besides the general theme of connecting the world through software, the cover has some easter egg references to the web’s history and major players, from centering the globe on CERN to using Fira Sans for the title.

To celebrate our progress toward publication, we’ve also pushed several updates to the website which fix a number of bugs in the text. The most important is probably a fix to iframe scrolling in Chapter 15, where we were failing to trigger a redraw when an iframe scrolled. We’ve also fixed all of the JavaScript widgets on the site (especially in Safari). The text of the book is also much more consistent in terms of terminology and typography, largely thanks to the work of our amazing copyeditor Richard Hutchinson.

Anyway, if you’d like to purchase a copy, please do pre-order, which helps promote the book. You can do so from OUP’s website, or through the numerous links on the book’s website.

Merging in Pixel pseudo-code: blending modes are kind of complicated, and it’s helpful to see their mathematical definitions. But of course a web browser doesn’t actually have those formulas in its code base; they instead live inside whatever 2D graphics library the browser uses, in our case Skia. In earlier drafts there were some sections, then, that had pseudo-code for the blending operations, but these sections stood out from the rest of the chapter. We’ve now integrated them a bit better and introduce them earlier, where we talk about drawing / blitting, as a way to explain how a surface is represented internally.

Introducing scrolling earlier: a simple and compelling use case for compositing is scrolling: rasterizing the whole page to one surface, and then just offsetting that surface during drawing, makes scrolling much, much faster than re-rasterizing on every frame. Making this one of the first things in the chapter, instead of one of the last, justifies the up-front work to switch from Tk to Skia and, I hope, gets the reader asking “what else can surfaces do”.

De-emphasizing atomicity: applying effects like opacity atomically means correctly grouping effects and applying them in the right order. In a previous draft of this chapter, we first introduced non-atomic effects, and then had to have a bunch of text to convince the reader of the value of atomicity. In the new draft, we take a new approach: introduce RGBA colors (which don’t need atomicity because they only apply to one drawing command) and then introduce atomicity at the same time we introduce opacity. RGBA makes it easier to explain atomicity, and now we can focus only on the correct approach instead of first introducing an incorrect one.

As part of this, we also fixed the compositing algorithm in Chapter 13 to handle effects atomically. This is a little tricky, and Chris and I debated for a while whether the added complexity is worth it, but ultimately we decided that always handling atomicity correctly made the book simpler—we wouldn’t have to explain the effects of doing it wrong.

Renaming the effects: In a previous draft of the chapter, more or less all visual effects were applied via a single “Swiss army knife” drawing command called SaveLayer. That’s important for efficiency but the name was opaque and it made the chapter harder to read. We now introduce separate Blend and Opacity operations before merging them later in the chapter when we talk about optimization. I think this makes the chapter a bit easier to follow.

Removing ClipRR: In previous drafts, the chapter introduced destination-in blending for clips, but then switches to using Skia’s native ClipRR. The native operation is faster, but it’s a bit weird to introduce blending and then not use it for anything significant. So in the latest draft, we remove this optimization from the chapter text. As a result, the book browser now uses blend operations for clipping, which I think better justifies their importance. It also provides another opportunity to talk about blend order and atomicity.

Anyway, I think the new chapter draft is shorter and clearer. As mentioned, the changes are live on the site and should make it into the published book. Please do recommend the book to anyone you think might appreciate it, and if you’re interested in updates, follow this blog and Mastodon (or Twitter, whatever it might be called) to hear about future updates, and participate in the Github Discussions with questions and answers for the community. And as always, a huge thank you to our readers and our supporters on Patreon.

The old way

The browser chrome is a little complicated because it works one way before Chapter 11 and another way in Chapter 11 and onward, though there are similarities. Before Chapter 11, the browser chrome was drawn as part of the Browser’s draw method by directly invoking methods on the Tk Canvas object that the browser is drawn to. The arguments to those calls were just hard-coded numbers. This had the benefit of conceptual simplicity—the chrome is drawn “raw”, just method calls and constants.

After Chapter 11 switched to Skia that “raw”ness became hard to maintain, because the Skia API is more verbose, so instead the chrome code made calls to wrapper functions like draw_line and similar. One benefit of this old approach was that we didn’t need to define DrawLine and DrawOutline classes until much later in the book, since when the chrome was first implemented page content only contained rectangles and text.

But I didn’t like the draw_line functions—it felt weird to have a DrawLine class that calls a draw_line function—and also using “raw” APIs meant that the transition to Skia required a lot of quite tedious porting from one API to another. That porting was annoying enough that the book left it to the reader, which made it harder for readers to make it through Chapter 11. Ultimately, the “raw” API approach, while it had its benefits, just introduced needless complexity to the Skia port, and that wasn’t worth it.

The new way

The new way avoids all of these issues by first painting the browser chrome to a display list, which is then drawn to the Canvas or Surface just like page content. This does mean that we need the DrawLine and DrawOutline paint commands earlier, as soon as they’re used in the browser chrome, and it also means we have to accept some minor hits to aesthetics, like the back button now using the less-than character instead of a manually painted triangle to represent the back arrow. But using the same paint commands for page content and browser chrome means that nothing major has to be ported to Skia, and the DrawLine command is also useful for cursor drawing in Chapter 8.1

As we were doing this, we made another big change to the browser chrome: we added layout. You see, previously we gave exact pixel coordinates for the corner of every button and text field, and that worked fine because the browser chrome was basically static. But it caused problems across different operating systems, because different operating systems use different fonts of different sizes. The book asked readers to adjust the coordinates for their system, but actually observing some readers made it clear that that was difficult, and most readers weren’t learning much from doing it.

So now the browser chrome uses font measurements and metrics to determine the position and size of each browser chrome element. This means it’ll look right on every operating system from the start, and also reinforce layout reasoning, which is a big part of the book. One downside of this change is that hit testing for the browser chrome used to be trivial, since the size and position of each button and input field was hard-coded. After this change, the layout depends on font metrics, so we chose to compute the browser chrome layout in a layout phase and then retain that layout for later paint and hit-test phases. In fact, the browser chrome ends up structured very similarly to ordinary layout objects,2 further emphasizing that browser chrome and page content are more alike than not.3

The advantages

Basically, this change made Chapter 11, the port to Skia, a lot less laborious, and while it added a lot of conceptual complexity—a chrome display list, a Chrome class, a hit-testing method—a lot of that conceptual complexity is already familiar from layout objects and seeing it again might actually help readers review how layout and paint works and why both phases are needed.

The new changes are live on the site, so do check them out and let us know (by commenting below or by emailing us) whether you like them.

Web Browser Engineering will continue to be available on the current website, but Oxford will also be publishing it in book form, and it should ultimately be available from your favorite online and local book stores. We don’t yet have an exact date, pre-order page, pricing or anything like that, but we’ll make that all public as soon as we have it.

We also hope to have translations come out in other languages, though no firm promises on that front.

More details to come, and as always please do recommend the book to anyone you think might appreciate it, and if you’re interested in updates, follow this blog and Mastodon (or Twitter, whatever it might be called) to hear about future updates, and participate in the Github Discussions with questions and answers for the community. And as always, a huge thank you to our readers and our supporters on Patreon.

How Chapter 1 was written

Chapter 1 is pretty old—it was published in July 2020, just as the world was starting to first emerge from Covid lockdowns, and in fact a month before Chris joined the project! This means it was before we had any real sense of the tone or style of the book. For example, some of the “Go Further” blocks in that chapter were pretty strange by the standards of later chapters: a short sentence linking to a standards document, nothing more.

The presentation was also odd—it was breezier, less concerned with writing or organizing code. For example, most of the code was written as if https://example.org/index.html was the only URL you might be requesting, and it was up to the reader to actually generalize it to handle multiple URLs. That was confusing, and it was easy to get it wrong (for example, by not updating the Host header). In fact, that style wouldn’t last even a second chapter, because as the browser gets bigger and more complex, it becomes more and more important to know where each line of code lives and when it is executed.

So anyway, the point is that Chapter 1 tried out some stylistic ideas that didn’t quite work, and I was itching to rewrite it anyway.

Why one big function

Besides style, there was a key technical issues with Chapter 1. As originally published, Chapter 1 basically writes one big function: request.1 The issue with request is that it’s quite long, which makes it kind of annoying to talk about later in the book, when we need to add features (like cookies). The obvious choice would be to break it up into multiple steps:

• parse_url to parse a URL and return pieces like the host and path

• request_url to send an HTTP request

• Maybe parse_response to parse the resulting HTTP response

In fact, this isn’t hypothetical, because it’s how I initially wrote this chapter (sadly, pre-git, so lost to the mists of time). However, this design was pretty verbose and over time I merged pieces. For example, while splitting the HTTP request and response is nice, the object passed between the two would be a live socket connection. That seemed dangerous: different functions open and close the socket, and errors can happen in either function. So in the first version of the chapter that I committed, I had already combined the request and response logic into a single request function. So it was request and parse.

Of course I couldn’t really call the parse_url function, just parse. Browsers parse lots of things! But even parse_url was awkward! It would return a four-tuple, which I then had to pass to request. Since I didn’t want to use fancy Python features like *args, that meant every HTTP request would take up two lines (one to parse and unpack into variables, and another to call request). Plus those lines were just long and ugly. So eventually I just combined all of it into one big request function. This worked fine for the first chapter (where I could just walk through each piece of it step by step) and also for the next few, because we don’t modify the request function until Chapter 8, and because the most complex modifications don’t happen until Chapter 10.

Issues with one big function

That said, as we wrote more and more of the book, we started to dislike the one big function. First of all, over time, we changed Chapter 10 to focus more on browser-internal security policies (like CSP) and that meant adding functions like url_origin. These duplicate pieces of the URL parsing logic.

There was also a sense that much of the parsing code was eventually “organized” into classes like CSSParser and HTMLParser, but URL parsing ended up scattered across multiple functions called from multiple classes.

I initially tried a Networking class that could be used as a namespace for all the URL-focused classes. The issue with that was there wasn’t any state for the Networking class to represent until Chapter 10 (when cookies are added), which would make it weird to create the class in Chapter 1. (It would have only static methods, something we’ve avoided in the rest of the book.) And splitting parsing into its own function would reintroduce the issue that its return value was an ugly four-tuple.

Eventually I hit up a variation that would work: a class to represent URLs.

A URL class

Upon reconsideration, the theme of the book is browsers, not web protocols like HTTP. So while the first chapter does focus on networking, we need to think about networking from a browser-focused perspective, which to me means a procedure to convert URLs to resources. So the key object to represent is the URL, not networking in abstract. A URL class would fix a lot of issues.

First, the URL parser could be the constructor for URL. Instead of returning a four-tuple, it would just return a URL object with proper fields. Instead of passing all the fields to request, request could be a method on URLs. In Chapter 10 we need to pass more arguments to request, including a second URL, and distinguishing the URL being requested by making it the object whose request method is called would make that code clearer.

Things like url_origin and resolve_url could become other methods on URLs. Plus, it would become possible to test URL parsing directly by just constructing a URL and querying its fields, instead of inspecting the HTTP request. (This would help my students, who use the browser unit tests as part of their assignments.)

The networking chapter could also be refocused on URLs instead of networking in general. This just required moving some paragraphs around, not a full rewrite, but Chris & I felt it made the chapter more concrete. It also means that we can get to code (for URL parsing) right away, which is more in keeping with the rest of the book.

Conclusion

Anyway, we’ve now rewritten the first chapter, and the rest of the book, to introduce a URL class. The updated chapter is live on the site. We’ll be making more modifications to the book, and pushing those as we go, with blog posts when we can. And hopefully soon we’ll be able to share news about an on-paper publication sometime in 2024!

Since we’re not writing more chapters, we’re now working on wrapping up the book. You’ll see edits (including some major ones) to already-published chapters, and also technical changes to the code that we think will make it clearer. A lot of these involve back-porting changes from later chapters to earlier ones, and we’ll blog about them as they happen.

We’re also working on finishing up various related content in the book. We’ve just released a post-script, a glossary, and a one-page version of the book. We’re working on similar things too. That should set us up to publish a physical copy of the book sometime soon—stay tuned for an announcement.

Thanks as always to readers, and our supporters on Patreon. Please continue to give us feedback and tell your friends all about it. And follow this blog and Mastodon (or Twitter, whatever it might be called) to hear about future updates, and participate in the Github Discussions with questions and answers for the community.

Invalidation is well known to be a nest of bugs, performance issues, and subtle interactions. In fact, it’s one of Phil Karlton’s famous two hardest problems in computer science—so it’s no surprise that he worked on Netscape. Accordingly, this chapter is the most complex one that I’ve written.

My key goal was to make sure we not only described the basic techniques of validation, but that our implementation was correct or, better yet, obviously correct. Unfortunately, existing browsers actually don’t reach this bar—perhaps Chrome’s recent LayoutNG architecture is the closest—so this chapter introduces a new abstraction called ProtectedField that, I hope, makes implementing complex invalidation schemes safe and easy.

Chris and I went through a huge number of variations in the course of writing this chapter. There are many, many ways to implement invalidation—it’s just that lots of them are subtly incorrect, or wouldn’t scale to a real browser. I feel pretty hopeful that we got it right in the end: ProtectedField is relatively easy to use, allows readers to implement invalidation step by step, and still makes it hard to forget a dependency or forget to invalidate something.

Another challenge throughout this chapter was the need to make a number of small- and medium-sized changes to previous chapters to make Chapter 16 possible. It turns out that over the four years we’ve been writing this book, we made a few mistakes. But in software engineering and web-published books, mistakes can be fixed, so we did just that in a few cases. If you look carefully at some of the earlier chapters, you might notice the changes.

This is the last planned technical chapter of the book. Our hope is now to add some nontechnical content, clean up various design issues that have been bugging us, and move toward a complete, finished book sometime soon. More on that when we can!

Thanks as always to readers, and our supporters on Patreon. Please continue to give us feedback and tell your friends all about it. And follow our blog and Mastodon (or Twitter if you need it) to hear about future updates, and participate in the Github Discussions with questions and answers for the community.

What are computed style values?

In CSS, an element has a "computed style" as well as a "specified style". The specified styles are what you get after matching and cascading assigns a value to each CSS property for that element. But computed styles differ from that because they resolve relative font sizes and ensure properties like padding only have positive lengths.

I haven’t worked much with the modern CSS specifications, but in CSS 2, it always felt to me like computed styles were mostly there for resolving percentage font-size values, so that “em” units could be converted into “px” units during styling.1 Note that most percentage sizes are resolved during layout; it’s just font-size that’s special and has to be resolved during styling.

How do browsers handle computed styles?

Logically, computed styles are separate from specified styles, so computing2 them is a separate pass, applied after all of the other elements of the cascade have happened and determined the element’s specified style.

However, in the WBE browser, the only computed value is font-size: percentage font sizes resolve into pixel values, so we had some flexibility to implement it in a different way. We chose to write a compute_style function, which you’d call before storing any value in the style dictionary.

The advantage of this is that the style dictionary only ends up containing computed styles, which is a kind of conceptual simplification. But watching my students work through the style chapter, I realized that this approach created more problems than it solved.

What was the problem?

First, it wasn’t clear why you would do something in compute_style instead of in style itself. I suppose the CSS spec does provide the answer (when it defines the differences between computed and specified styles) but readers and students aren’t going to be intimately familiar with those steps, and the boundary between these functions then looks pretty vague.

Second, compute_style had a weird API. Apparently it could output None to mean not to store the value at all, which I wanted to use in case someone used an illegal unit for font size. But this was neither explained in the book nor even correctly handled by all of the code. I think I was trying to shove value parsing into this function, too, but conceptually that’s a different step of the process (it normally happens during parsing) and adding it here just made things confusing.

Finally, and most importantly, this design got in the way of Chapter 16. In Chapter 16,3 we need to track dependencies between CSS values, which means mainly working with style fields, not style values. The compute_style API hindered that because it accessed style fields but returned style values. If the API were otherwise great, I could have worked around it, but given that the API was already bad, it seemed ripe for replacement.

The new approach

Given all of this, I went back and removed the compute_value function, replacing it with an extra “phase” of style that runs after all of the cascading is done but before we recurse to style the children. This avoids having an API boundary at all, which avoids all the challenges with designing a good one and also doesn’t hinder Chapter 16. As a bonus, it’s a bit more like a real browser.

I’ve merged this change to the book and updated the website, so you can check out Chapter 6 to see the new approach and update your browser to follow suit if you like it.

See you all again soon!

You might be surprised that iframes can be implemented in just one chapter’s worth of text and code. That’s because it builds on all of the powerful technology built up in prior chapters. I really enjoyed the way this chapter tied into almost every previous chapter’s code, leveraging it to make a complex feature come together quite quickly.

Images and iframes are almost the last core feature / architectural concept that we want to cover—we are nearly done with the book! In fact, unless we change plans, this looks like the last chapter with me as lead author. Honestly, I can’t believe how far this book has come, nor that it’s almost done. It’s been quite an adventure.

Thanks as always to readers, and our supporters on Patreon. Please continue to give us feedback and tell your friends all about it. And follow our blog and Mastodon (or Twitter if you need it) to hear about future updates, and participate in the GitHub Discussions with questions and answers for the community.

How Layout Works in WBE

Chapter 5 is about browser layout: building the layout tree, computing sizes and positions of the layout nodes, and using that information to paint the web page to the screen.

Until this change, the layout tree had two key types of nodes: BlockLayout and InlineLayout. BlockLayout nodes were used for things like <div> or <body>, elements that contain a sequence of vertically stacked blocks, while InlineLayout nodes were used for things like <p> or <h1>, which contain text laid out left to right in lines.

There are basically two key things a layout node needs to be able to do: construct its children, and compute its own size and position.1 BlockLayout and InlineLayout construct their children differently (BlockLayout creates BlockLayouts and InlineLayouts; InlineLayouts create LineLayouts and TextLayouts) but compute their sizes and positions more or less identically.2 So it was kind of a judgement call whether these should be one class, thereby sharing the size and position computation but with ugly conditionals for the tree construction, or two classes, avoiding the conditionals at the cost of some duplication.

When we were first writing Chapter 5, it looked like splitting the two classes lead to cleaner code overall, so we went with that.

Why Change?

First of all, since we wrote Chapter 5, we’ve written a lot more chapters, and that’s made the costs and duplication versus conditionals much clearer. Chapters 11 and 13 about the graphics pipeline, for example, required lots of changes to painting for BlockLayout and InlineLayout. Typically, those changes have to be duplicated across both classes (ugly!), and it’s pretty easy to forget change both in the same way, meaning that it’s easy to end up with rare bugs or to have to fix the same bug twice. That’s frustrating.

Second, I’ve taught the material in the book a couple of times, and I’ve found that it’s hard for students to keep in mind the distinction between BlockLayout and InlineLayout. It’s not that it’s hard to differentiate between paragraphs of text and a section containing paragraphs; it’s just that a paragraph is still a block, so if you’re implementing, say, background images, it seems natural to do that in BlockLayout, even if your test page uses a paragraph. That leads to false starts and confusions. It doesn’t help, by the way, that CSS has the concept of display: block and display: inline, and those don’t correspond to BlockLayout versus InlineLayout (the book doesn’t implement the display property).

Third, and ultimately what was decisive, is the ongoing work on invalidation in Chapter 16. Without writing a whole chapter here on what invalidation is and how it works, one important step is caching the layout tree instead of recreating it every time the browser does layout. This requires only re-creating a layout node if its dependencies change. But this creates a problem with the split between InlineLayout and BlockLayout: it is a node’s parent that creates the InlineLayout or BlockLayout, so the node’s parent needs to keep track of changes to its children to recreate them as the correct layout node type. This was very hard to explain, in what’s already going to prove a very complicated chapter, and would be avoided entirely by unifying the InlineLayout and BlockLayout classes.

Making the Change

I know many readers are following along with the book, implementing their own web browsers and sometimes even going way beyond the book. It’s up to you whether you want to update your browser with the unified BlockLayout, but you may want to, and if you do it’ll probably be easier to incorporate Chapter 16 when it is eventually published.

I think the change won’t be too bad to incorporate. I recommend doing it in a few steps.

First, track down where InlineLayouts are created in your code. Most likely, they are only created by BlockLayout’s layout method, inside the tree creation for loop. Remove that code; once we unify BlockLayout and InlineLayout, that for loop will only create BlockLayouts.

Second, copy all the InlineLayout methods into BlockLayout. That includes recurse, text, input, new_line, and get_font.

Third, verify that the paint method and the constructor are the same for InlineLayout or BlockLayout. It’s possible that they’re different—maybe only InlineLayout has backgrounds, or only BlockLayout supports visual effects. In that case, you’ll want to merge the two paint methods. It should be possible to support all the paint features that either supports.

At this point the only thing left unmerged should be the layout methods of the two classes. These two methods should have a lot of duplication—they should compute sizes and positions in the same way, and both should recursively call layout on their children—but they will have different tree creation code. You’ll want to use a conditional to unify the tree creation code, like this:

class BlockLayout:
    def layout(self, zoom):
        # ...
        mode = layout_mode(self.node)
        if mode == "block":
            # BlockLayout tree creation code
        else:
            # InlineLayout tree creation code
        # ...

The BlockLayout tree creation code here is a for loop, while the InlineLayout tree creation code calls new_line followed by recurse.

Once you’ve made these changes, verify that your browser works, that InlineLayout has no references in your code, and that all its methods have been copied into BlockLayout, and then delete the InlineLayout class.

Future Changes

As much as I would love to be “done” with the already-published chapters of WBE, we’ll probably need to continue pushing fixes or changing the code of earlier chapters as we work on later ones. Once we reach the end of the book, we’ll probably do a clean-up pass as well. So there are probably more book-wide changes coming up. So far those changes have been minor (changing something from a local variable to a field, for example) and we’ve pushed the changes without announcement, but if we need any more major changes we’ll put up a blog post.

But don’t let these changes keep you from hacking away on your own browser. One of the goals of this book is to give you ownership of a web browser. If you don’t like our changes, it’s your browser and your code! Or you may well have found your own workarounds to these problems, or written enough code on the old abstractions that it’s hard to port. That, too, is part of working on web browsers, and as good a reason as any to skip the updates. Happy hacking!

Not Eval

Basically, this should be easy—“how do I run JavaScript in a web browser”—that’s not hard, that’s what browsers do! JavaScript even has an eval function that runs JavaScript source code! It’s the Python version of the browser that needs additional libraries to run JavaScript.

But it turned out to be harder than expected. Running JavaScript isn’t hard, but isolating it from the rest of the browser is.

This gets a little confusing, so remember that there’s two browsers here: a host browser, which is Chrome or Firefox or whatever you’re running, and a virtual browser which is the compiled-to-JavaScript version of the WBE browser. We want to run JavaScript in the virtual browser by downloading a virtual script; set up a virtual execution environment by defining virtual constants like document and Node; and then run the virtual JavaScript in that virtual environment to modify the virtual page. But if I run the virtual JavaScript with eval, it’ll attempt to access the host document and Node variables and modify the host browser’s host page.

We can get around this using the JavaScript with command or the even weirder distinction between direct and indirect eval2 to make the virtual JavaScript run inside a scope, potentially one where host definitions of document and Node are shadowed by the virtual definitions. But unless we shadow everything the host defines (and we won’t, browsers are constantly adding new host definitions) the virtual script can probably “escape” the scope and start mucking with global state. Not good.

Isolating with Web Workers

Luckily, JavaScript does provide a workable isolation mechanism in the form of Web Workers. Web Workers are basically share-nothing threads for JavaScript; each Web Worker runs in an isolated environment and has no access to the environment that created it. That sounded perfect for my use case.

I set up a simple Web Worker based JavaScript execution library like so. First, when you create a JSInterpreter, I create a new Web Worker:

class JSInterpreter {
    constructor() {
        this.worker = new Worker("/widgets/dukpy.js");
        this.worker.onmessage = this.onmessage.bind(this);
        this.promise_stack = [];
    }
}

To run a script in the interpreter, we just send it over with postMessage:

class JSInterpreter {
    evaljs(code, replacements) {
        return new Promise((resolve, reject) => {
            this.promise_stack.push(resolve);
            this.worker.postMessage({
                "type": "eval",
                "body": code,
                "bindings": replacements,
            });
        });
    }
}

I send over the source code we want executed and then return a Promise. That Promise will resolve when the virtual script finishes executing and wants to return a value; for that reason I save its resolve function for later.

On the WebWorker side, when we receive this message we just run eval:

addEventListener("message", (e) => {
    switch (e.data.type) {
    case "eval":
        dukpy = e.data.bindings;
        let val = eval?.(e.data.body);
        if (val instanceof Function) val = null;
        $$POSTMESSAGE({"type": "return", "data": val});
        break;
    }
});

Basically, we call eval on the body of the message, using the indirect eval style for reasons that I remember losing an hour to but I don’t remember the details of.3 Then we send that return value4 back to the virtual browser with postMessage, where the weird name is to avoid a script overwriting postMessage and making everything break.

Finally, back in the virtual browser, we receive this message and resolve the promise:

class JSInterpreter {
    async onmessage(e) {
        switch (e.data.type) {
        case "return":
            this.promise_stack.pop()(e.data.data);
            break;
        }
    }
}

This basically works, and lets me execute JavaScript in a Web Worker and receive the results.

The DOM

But the virtual browser running virtual scripts is just one half of JavaScript support. The other half is virtual scripts calling into the virtual browser. Specifically, we want virtual scripts to be able to call browser APIs like querySelectorAll or innerHTML.

The challenge here is pretty mundane: these APIs are synchronous, in that you call them without using await and expect results in the form of values, not Promises. We can send an API call request to the virtual browser no problem, using postMessage. But inside a Web Worker, we can’t receive data from the main script (which runs the virtual browser) except by receiving a postMessage back, and that requires us to first finish executing the current script. We need some form of synchronous cross-thread communication, and JavaScript mostly doesn’t provide that.

Mostly! It turns out there are at least two support methods for communicating across Web Workers without using messages: localStorage and SharedArrayBuffers.

I haven’t tried the localStorage route, and it seems janky, so let’s talk about SharedArrayBuffer. This feature was originally added to JavaScript to facilitate stuff like WebAssembly and WebGL, so it’s kind of obscure. Basically, you can allocate a SharedArrayBuffer and send it, via postMessage, to a Web Worker. Then both the main script and the Worker can modify it, and those modifications are immediately visible in both threads. Moreover, the Atomics set of functions allows various forms of locking, blocking, and atomically modifying the shared buffer.

This is perfect for making calls from virtual scripts to the virtual browser. The virtual script can send a postMessage describing the API call it wants to make, and then wait on a prearranged SharedArrayBuffer for the return value. In code, this looks like this:

function call_python() {
    let args = Array.from(arguments);
    let fn = args.shift();
    $$POSTMESSAGE({
        "type": "call",
        "fn": fn,
        "args": args,
    });

    Atomics.wait($$FLAGARRAY, 0, 0);

    let len = $$FLAGARRAY[0];
    Atomics.store($$FLAGARRAY, 0, 0);
    if (len > 0) {
        let buffer = new Uint8Array(len);
        for (let i = 0; i < buffer.length; i++) {
            buffer[i] = $$READARRAY[i];
        }
        let result = JSON.parse(new TextDecoder().decode(buffer));
        return result;
    }
}

There’s a lot of code here, but it’s ultimately not that bad. The first block of code sends a API call request to the virtual browser. The second block, with an Atomics.wait instruction, blocks the virtual script until a pre-arranged “flag array” is changed. Once it is, a length is read from the flag array, and then that many bytes are read from a pre-arranged “read array” which are decoded from JSON to be the return value of the API call.

(Recall that in the book’s Python version of all of this, API requests like querySelectorAll just return a list of integer handles, which are then wrapped on the JavaScript side into Node objects, so JSON is more than rich enough to represent all the data we need to pass back and forth.)

On the virtual browser side, things are similarly not too bad: when the virtual browser receives a call message, it calls the requested function with the requested arguments, encodes the result to JSON, and stores it in the read buffer before updating the flag buffer:

class JSInterpreter {
    async onmessage(e) {
        switch (e.data.type) {
        case "call":
            let fn = this.function_table[e.data.fn]
            let res = await fn.call(window, ... e.data.args);

            let json_result = JSON.stringify(res);
            let bytes = new TextEncoder().encode(json_result);
            if (bytes.length > this.write_buffer.length)
                throw new JSExecutionError(e.data.fn);

            for (let i = 0; i < bytes.length; i++) {
                Atomics.store(this.write_buffer, i, bytes[i]);
            }

            Atomics.store(this.flag_buffer, 0, bytes.length);
            Atomics.notify(this.flag_buffer, 0);
            break;
        }
    }
}

Here there are four steps. First, run the requested function. Next, encode it to JSON and check that there’s enough space in the buffer to store it all.5 Finally, store the result to the read buffer6 and then unblock the waiting Worker with Atomics.notify.

By the way, in response to an API call, the virtual browser might execute more JavaScript—which is why the promises used when returning have to be a stack.

Security

So after a few hours of debugging this all worked, and things were basically good—except for the fact that every browser turns off SharedArrayBuffer for security! This is part of the fallout from Spectre, a class of vulnerability where very accurate timing measurements can reveal information about other processes executing on the same CPU. The current state of things is that you must choose between SharedArrayBuffer and iframes.7

The Web Browser Engineering book uses iframes for a bunch of stuff, but most prominently it uses them to embed the JavaScript browser into the book. So we can’t turn off iframes. But we need SharedArrayBuffer to run JavaScript in the embedded browser. A dilemma.

For now, we’ve gone with a pretty limited solution. The JavaScript-enabled virtual browser isn’t embedded into the book pages. Instead, there’s a link to a separate page containing only the virtual browser. On that page, iframes aren’t used and SharedArrayBuffer is enabled instead.

Frankly, this isn’t an ideal solution, so one of the things on my todo list is to investigate using localStorage for synchronous communication between threads instead. If anyone has done that, I’d love to hear more (how long does it take? do you just busy wait?). Otherwise, uhh, please don’t use anything described in this post in production. It’s definitely a bad idea. But do enjoy the JavaScript-enabled browsers linked from Chapters 9 and 10!