​

Tier Pipeline

Differens parses files at six levels of abstraction, T0 through T5. Each tier is a parser that produces a Node tree. Each tier can fall back to the tier below it. The tier number is the abstraction ladder: T0 is bytes, T5 is code.

​

The tiers

​

T0: binary (hash-only)

Byte-level hash comparison. Used for images, fonts, archives, and anything else that fails the text sniff. Output is a single Update when the hash changes, with the byte-size delta as context. T0 never attempts to parse.

​

T1: raw (LCS line diff)

The safety net: a line-level longest-common-subsequence diff over the raw text. It is the fallback for every tier above it. It is capped at 2,000 lines. Beyond that, T1 returns a whole-file Update action instead of running an unbounded LCS.

​

T2: prose (word-level)

Word-level diff for plain text and logs. Paragraphs and sentences become tree nodes, so a moved paragraph is a Move instead of a delete-plus-insert. Punctuation and whitespace changes update leaf nodes rather than rewriting whole paragraphs.

​

T3: markup (lenient HTML/XML)

A lenient HTML/XML tokenizer that tolerates malformed documents: unclosed tags, misnested elements, unescaped entities. Element hierarchy becomes the node tree. Attribute order changes are reorders, not updates.

​

T4: data (JSON / YAML / TOML)

Value trees for JSON, the YAML subset Differens supports, and the TOML subset. The diff is key-path based: nested objects become nested nodes, so a key moved between two objects reports as a Move with the key path in the label. Array reordering is a Reorder.

​

T5: code (tree-sitter)

A tree-sitter CST with per-language semantic extractors. The extractors map raw tree-sitter node types to canonical concepts such as Function, Class, Method, and Import. The narration can then say “function” whether the source said fn, def, or func. Currently: TypeScript/JavaScript, Python, Rust, and Go. Languages without an extractor still work at the generic level: structural diff with raw tree-sitter node type labels. The extractor only adds human-readable concept names.

​

Graceful degradation

The tiers form a chain. Each tier can hand off to the tier below it:

1. T5 parse fails (syntax error, unsupported construct) → fall back to a generic tree-sitter CST diff, still structural.
2. Generic CST fails (no tree-sitter grammar, garbage input) → T1 line diff.
3. T1 hits the 2,000-line cap → returns a whole-file Update action.

The worst case for text is a line diff. It is less informative than a structural diff, but it still runs.

​

Classification rules

The content router classifies by extension, then by content sniff: Extension-based classification is the first cut; the content sniff is the backstop. A .txt file that is actually minified JSON still gets a T2 word-level diff. That is acceptable, because the tier below can take over.

​

Tier summary