Engineering Research & Post-Mortems

1. Net Task Tokens vs. Gross Server Spend

Most industry benchmarks only report the happy-path tokens of tasks that succeeded. Over 14 days in production (850 model calls, 40.6M input tokens, 17.6M output tokens), we measure both metrics transparently: 586,759 net tokens on merged task rows versus 4,476,923 gross tokens when unmerged retries, background daemons, and planning revisions are included.

Metric                                  | 14d Baseline         | Post-Optimization Target
----------------------------------------+----------------------+---------------------------
Net tokens on merged task rows / PR     | 586,759              | ~400,000
Gross server tokens / merged PR         | 4,476,923            | <= 2,800,000 (-38%)
Developer CLI turns / merged PR         | 9.5 calls (35.0M)    | < 6.0 calls (< 2.0M/merge)
Planning tokens / merged PR (5 roles)   | 650,000 (8.4M total) | < 300,000 (SHA-256 cache)

2. Eliminating Transcript Replay in Resumed CLI Sessions

When a coding agent completes a vertical slice and spawns sequential follow-up processes for completeness, test coverage, typechecking, and linting, each resumed turn re-ingest the entire conversation transcript. Executing all deterministic static gates upfront and batching the union of failures into a single resumed turn eliminates three redundant context replays per slice.

1. Privilege Separation Between Builders and Judges

Code-authoring steps execute inside isolated git worktrees via sandboxed CLIs (Claude Code, Codex, or Antigravity). Conversely, all eight evaluation and external-input roles (QA Engineer, Plan Critic, Security Gate, Market Researcher, Competitor Analyst, Product Manager, and Growth Strategist) execute strictly over tool-less HTTP APIs with zero filesystem write access.

2. One-Sided Z = 1.64 Statistical Promotion

Candidate model configurations run on 50% of production tasks for up to 14 days. A challenger is promoted to default only if total tokens per merged PR drops by at least 10% at Z >= 1.64 (95% one-sided confidence) with non-inferior merge velocity. Learned heuristics that fail to reduce their target error signature by >= 30% are pruned automatically.

1. Federated Pattern Promotion Without Sharing Code or Secrets

When an instance discovers a reusable engineering pattern (such as Next.js 16 + Prisma connection pooling or Playwright flake elimination), a deterministic refusal filter scans the candidate rule against the workspace's forbidden literals, domains, and paths. Only stack-level patterns corroborated across at least two independent pseudonymized workspaces reach the operator's approval gate.

1. Quantified Blast Radius and Default-Action Escalations

Rather than streaming noisy terminal logs into chat, Brevio Harness enforces a strict weekly interrupt budget. High-risk changes touching billing, authentication, or database migrations emit a structured decision card stating the exact file blast radius, the cost of waiting, the automatic fallback action on timeout, and 1-tap controls ([Approve], [/hint], [/rule]).

1. Layer 0 Auto-Healing vs. Multi-Turn LLM Repair

Instead of paying an LLM 3,500 tokens to fix a missing threshold in an acceptance criterion or replace a banned sub-12px font class, deterministic Python healers repair the artifact on disk before the linter runs. Across test suites, PM node invocations dropped from 3 LLM calls to exactly 1 call while maintaining 1.0 AC testability.

Engine Module                   | Catalog Size | Layer 0 Zero-Token Auto-Healer                | Net Token Delta
--------------------------------+--------------+-----------------------------------------------+-------------------
brevio/pipeline/dev_rules.py    | 42 Rules     | Git diff added-line scanner + targeted brief  | 0 tokens on pass
brevio/pipeline/design_rules.py | 40 Rules     | Auto-heal text-[10px]/h-6/gradients in <1ms   | -1,800 tokens/run
brevio/pipeline/pm_rules.py     | 40 Rules     | Auto-heal weak ACs, (source: F#) & 3-tier KPI | -4,150 tokens/run