Phase 3g: Recursive document walker + ProseFieldCompressor (parked)

[chopratejas]

Status: parked, scheduled to land alongside Phase 6 (ONNX migration)

Parking this so we can finish the Rust proxy (Phase 1) and per-provider cutover (Phase 4) first. The architecture below was designed in the same conversation that landed PRs #330 and #331; capturing it here so it can be picked up cleanly.

The gap

The pipeline orchestrator landed in Phase 3g handles "flat" content well: a tool output that is one of {JSON array, log, diff, search results} flows through the matching transforms cleanly. What it does not handle is nested structure:

{
  "results": [...],
  "summary": "long prose paragraph here that should be Kompress-compressed",
  "raw_logs": "INFO worker-1 ...\nINFO worker-2 ...\n... 800 more lines",
  "diff": "diff --git a/foo.py b/foo.py\n@@ ..."
}

Today this top-level object hits JsonOffload (SmartCrusher), which deduplicates the array but leaves the prose / logs / diff fields untouched.

The design

A recursive document walker that, at each leaf, runs Magika content detection and dispatches to the matching transform.

Three sub-PRs, stacked

PR3a — DocumentWalker (the recursion architecture)

• Walk JSON tree; at each leaf string, run content detection (Magika is already in the codebase).
• Dispatch by detected ContentType:
  • BuildOutput → LogTemplate + LogOffload, replace string in-place.
  • GitDiff → DiffNoise + DiffOffload.
  • JsonArray (stringified JSON inside JSON) → recurse.
  • PlainText AND token_count > threshold → flag as prose candidate (compression hookup in PR3b).
  • Otherwise → pass through.
• For arrays-of-dicts at any node, delegate to JsonOffload (the existing SmartCrusher wrapper).
• Walker config: max depth, prose token threshold (default 50, configurable per #315 conventions).

PR3b — ProseFieldCompressor wrapping Kompress-base

• Plugs into the walker's prose-candidate hook.
• Calls Kompress-base (the dual-head BERT at https://huggingface.co/chopratejas/kompress-base).
• Initial implementation: PyO3 shim into the existing Python Kompress runtime — ships fast, reuses the loaded model.
• Token threshold (configurable, default 30-50 tokens) below which we don't bother — overhead exceeds savings on short fields.
• Lossless via CCR: original prose stashed under cache_key; LLM retrieves on demand.

PR3c — Native Rust Kompress (collapses into Phase 6)

• Replace the PyO3 shim with native ort ONNX inference (already in deps via fastembed/magika).
• Model download via hf-hub (already in deps).
• Tokenizer via tokenizers (already in deps).
• Drop the Python runtime dependency for prose compression.

Architectural decision recorded

Walker vs SmartCrusher's internal process_value recursion: the new walker delegates to SmartCrusher for array-of-dicts nodes only. SmartCrusher's internal walking stays untouched — preserves the 51 parity fixtures. Two recursion paths but no behavior change. (Decided in conversation; alternative was to refactor SmartCrusher into a leaf compressor, which would require re-validating all parity fixtures.)

Why parked

1. Existing pipeline already covers the 80/20. Tool outputs that are one shape — pure log, pure diff, pure JSON array — flow through cleanly. Nested-shape outputs are real but less common.
2. Phase 1 (Rust proxy) is the critical-path blocker. Until the proxy is in Rust, retiring the Python ContentRouter is risky.
3. Phase 6 (ONNX migration) is Kompress's natural home. Building a PyO3 shim now is throwaway code that gets replaced when we do native ort inference for ALL ML transforms.
4. Validation comes from real usage. Once the Rust proxy sees traffic, we'll know whether nested-prose-in-JSON is a real pattern or a theoretical one — informs threshold defaults.

Mitigation for the gap

• At Python-proxy retirement (Phase 7) gate: if telemetry shows nested-prose payloads matter, this issue must close before cutover.
• For users who hit the gap today: existing Python ContentRouter still does basic LLMLingua-style prose handling end-to-end; the Rust pipeline only kicks in via PyO3 bridge for the shapes it covers.

Scheduling

Land alongside Phase 6 (ONNX migration). Both share infra:

• ort ONNX runtime
• tokenizers HF tokenizer crate
• hf-hub model download
• ONNX model export workflow

Doing them together avoids two separate dependency-stabilization passes.

Acceptance

• [ ] PR3a: DocumentWalker walks JSON, detects leaves, dispatches to existing transforms. Tests cover nested logs/diffs/JSON in JSON.
• [ ] PR3b: ProseFieldCompressor (PyO3 shim) plugs into walker. Tests cover prose detection + token threshold.
• [ ] PR3c: Native ort Kompress replaces the shim; Python runtime no longer needed for prose compression.
• [ ] Pipeline TOML config for walker depth / prose threshold under [walker] section.
• [ ] Phase 7 (proxy retirement) cannot close until this issue closes OR telemetry shows nested-prose is non-critical.