architecture / the-pipeline

architecture / the-pipeline

The pipeline

v0.12.0 · 01 / 05 · 4 min read · docs/ARCHITECTURE.md

Resolve. normalizeInput(in, engine) (nav.go) decides what an entry

target ─► normalizeInput ─► load ─► toMarkdown ─► renderDoc ─► model ─► screen
(url |      (nav.go)       (fetch)  (convert+       (render.go) (tui.go)  (View/render)
 file |                       │       extract)         ▲
 words)                      │           │            └─ guessLang (guess.go)
                             │           └─ a page drawn by scripts? ─► alternate text
                             │              version ─► headless render (headless.go)
                             └─ cache (~/.cache/wr)      store (~/.local/state/wr)

              model ◄── update check (update.go): GitHub releases, interactive reader only
  1. Resolve. normalizeInput(in, engine) (nav.go) decides what an entry is: an http(s):///file:// URL, a local path (always file://), a looksLikeHost shorthand, or search words through the configured engine (engineTemplate, duckduckgo/wikipedia/github or a custom %s URL).
  2. Load. load(src) → loadFrom(src, fetchURL) (fetch.go; loadWith is the same with a plain fetcher injected, which is why extraction is testable without a network): fetch (20 s timeout, one retry on a reset or a timeout, 8 MB cap, UA fixed, Accept-Language from the locale, TLS verified, charset → UTF-8) and report the final address after redirects, which is what relative links are resolved against. An HTTP error names who answered (the <title> of the error page) and what to do. Then convert dispatches by media type: HTML → toMarkdown, Markdown passes through sanitize, JSON/XML/plain is fenced. A page that came out as a JS shell (see shell.go) is retried in order: its advertised text version (alternateLink), its AMP version (ampLink), then renderedMarkdown (headless.go: an installed Chromium/Chrome run headless, its own profile, the whole process group killed on timeout, output capped, images off, skipped when render_js is off), else the page’s own summary under a notice. On success it writes the cache when it is an http page, except for a bare shell.
  3. Extract. toMarkdown (convert.go): collectHints first saves what a shell page says outside its body (<noscript>, description, JSON-LD); then pickRoot scores <article>/<main>/[role=main]/<body> by text weight, dropTags removes <form> and friends, dropHidden what the page hides (including aria-hidden), dropHeaders/dropChrome strip the page chrome (with a 50% size guard so a chrome-shaped document is not emptied), flattenCodeLines turns block-line code (Expressive Code) into text, code blocks keep their language (language-*, data-language, Sphinx highlight-*, MediaWiki mw-highlight-lang-*), tables are flattened, headings lose their permalink anchors (cleanHeadings), anchors are tidied (tidyAnchors: empty and skip links go, cards keep the title as the link) and absolutized, then html-to-markdown v2 plus fence-aware cleanups (tidyMarkdown, dedupeLinkLines, dropLeadingCrumbs) and sanitize (control characters, including ESC, are removed — page content is untrusted).
  4. Render. renderDoc(md, width, cfg) (render.go) parses the Markdown with goldmark, highlights code with chroma (memoized per block, one goroutine per block; a block with no language gets guessLang’s answer when it is sure, marked ? internally and labelled “(guessed)”), and emits self-contained ANSI lines in a Doc: Lines, Plain, Heads, Links (as LinkSpan + URLs), per-line Meta for selection. Invisible APC markers placed during rendering are scanned into link spans afterwards, so links keep their screen positions. renderDocCached (render_cache.go) wraps it in a bounded LRU keyed by the Markdown, the width, the layout options and the palette, so resizing back and going back do not parse or highlight again.
  5. Interact. The Bubble Tea v2 model (tui.go): keys, mouse, resize, spring scrolling (harmonica), search, selection, toasts, panels (the update window is one more: update_ui.go). Navigation (browse.go) keeps back/fwd stacks of navEntry and runs every load through startNav/loadCmd; the pages of the session also live in a small in-memory LRU (page_memory.go), so back and forward do not touch the disk. An overtaken load is dropped by id, and a failed load never replaces the page on screen. newModel + Update + render() work with no terminal at all — that single property is what makes the deterministic UI tests and the agent sessions in mcp_session.go possible.
  6. Update. startUpdateCheck runs once at launch in the interactive reader only (updateEnabled is switched on by runTUI, so tests, pipes, MCP and eval never reach the network for it). checkForUpdate trusts the last answer for 12 hours, remembers it in update.json, and reports a newer release, whether it was already announced, and, on the first launch after an update, that release’s notes. applyUpdate downloads the archive from the project’s release URL only, verifies its SHA-256 against checksums.txt, runs the new binary (--version must match the tag) and renames it over the running one.

edit this page on GitHub ↗report a problem ↗