Reading the modern web

An agent that browses has to answer one question constantly: what does this page say? The obvious implementation is document.querySelector('main') and a walk of its children. That implementation is wrong on most of the web, and it is wrong in the worst possible way — it returns something, confidently.

The shape of the bug

querySelectorAll does not enter shadow roots. On any site built from web components — and that is YouTube, Reddit, Gmail, and a great deal of documentation — the interesting half of the page is invisible to it. It does not enter iframes either, so checkout forms and embedded editors are invisible too.

Helix's page toolkit knew this. Every walk in it is a composed walk: document, then each open shadow root, then each same-origin iframe, recursively. Elements, forms, tables, links — all composed.

Except the one that reads the page as prose. That one took node.childNodes and never crossed a shadow boundary in its life.

So asked to read the DOM documentation, the agent got the prose and none of the code, and reported it as the page. There was nothing in the reply to suggest anything was missing — which is why this kind of bug survives: it does not look like a bug, it looks like a short page.

The flattened tree has exactly three rules

Reading a page as it is rendered, rather than as it was authored, is a solved problem with three rules:

• An element with a shadow root renders its shadow children, not its light ones — so descend into the shadow root instead.
• A <slot> renders whatever was assigned to it, so descend into assignedNodes. This is how light-DOM content that is displayed gets read, and why skipping light children above loses nothing.
• A same-origin frame renders its own document. A cross-origin one genuinely cannot be read, and saying so is useful where pretending it was empty is not.

Getting the first two right together is what avoids the obvious trap: walking both the host's light children and its shadow root reads every slotted string twice. The test fixture now has prose behind a slot specifically to assert it appears exactly once.

A correction we owe the record

The first version of this investigation reported that 57,531 characters were hidden inside shadow roots. That number was wrong. It counted textContent, which on that page is dominated by the CSS inside <style> elements. The real figure — rendered text, measured before and after in the same session on the same document — is the 14,472 → 25,033 above.

We mention it because the correction came from the same discipline as the fix: measure it in a real browser, then measure it again when the result looks too good.

The other three

Reading was not the only thing wrong down there.

A page call that answered false was treated as a missing toolkit. The bridge evaluated (window.__helix && expr) and read a falsy result as proof the toolkit was absent — so it re-injected 63KB and ran the expression a second time. Several expressions legitimately answer false; the focus call inside type returns false whenever focus did not stick. So on exactly the stubborn pages where focusing was already failing, scrollIntoView, focus() and the selection reset all happened twice.

Root discovery was paid for once per query. Finding where the shadow roots are means walking querySelectorAll("*") over the whole composed tree, and that was being redone for every single selector — four times for one obstacle check. It is now computed once per synchronous call and dropped on the next microtask, which is what makes it safe: page JavaScript is single-threaded, so the DOM cannot change between two queries inside one function.

Nothing ever let go of the working tab. A run opened it, drove it, finished — and the tab stayed: pinned, with the debugger still attached, so Chrome kept showing "is debugging this browser" and the console and network buffers kept filling for a task that had ended hours ago. Closing at the end of a run would be wrong, because a follow-up lands seconds later and wants the same page and the same element refs. So the trigger is idleness — ten minutes, and never mid-call.