AIStoreHub Package Vault: Headroom
Package Slug: headroom
Archetype: mcp_server
Upstream Repository: https://github.com/headroomlabs-ai/headroom
Workspace Files & Specifications (67 files)
- .claude-plugin/marketplace.json (2210 bytes)
{ "name": "headroom-marketplace", "owner": { "name": "Headroom Contributors" }, "metadata": { "description": "Headroom marketplace for Claude Code and GitHub Copilot CLI plugins.", "version": "0.44.0" }, "plugins": [ { "name": "headroom", "source": "./plugins/headroom-agent-hooks", "description": "Headroom startup hooks for Claude Code and GitHub Copilot CLI.", "version": "0.44.0", "author": { "name": "Headroom Contributors", "url": "https://github.com/headroomlabs-ai/headroom" }, "homepage": "https://github.com/headroomlabs-ai/headroom", "repository": "https://github.com/headroomlabs-ai/headroom", "keywords": [ "headroom", "hooks", "claude-code", "copilot-cli" ] }, { "name": "headroom-skill", "source": "./plugins/headroom-skill", "description": "Skill: add headroom-ai compress() to a Python or TypeScript app to cut LLM input tokens (tool outputs, logs, JSON, chat history). No API key.", "version": "0.44.0", "author": { "name": "Headroom Contributors", "url": "https://github.com/headroomlabs-ai/headroom" }, "homepage": "https://github.com/headroomlabs-ai/headroom", "repository": "https://github.com/headroomlabs-ai/headroom", "keywords": [ "headroom", "skill", "token-cost", "tool-outputs", "context-window", "claude-code", "codex", "copilot-cli" ] }, { "name": "headroom-snip", "source": "./plugins/headroom-snip", "description": "Claude Code mod: watch Headroom snip tokens off every request, with an animated band, a /headroom log and a running total.", "version": "0.44.0", "author": { "name": "Headroom Contributors", "url": "https://github.com/headroomlabs-ai/headroom" }, "homepage": "https://github.com/headroomlabs-ai/headroom", "repository": "https://github.com/headroomlabs-ai/headroom", "keywords": [ "headroom", "claude-code", "tokens", "compression", "mod" ] } ] } - AGENTS.md (3494 bytes)
# AGENTS.md ## If you landed here to use Headroom in an app The PyPI package is `headroom-ai`; the import and the CLI are `headroom`. ```bash pip install headroom-ai ``` ```python from headroom import compress result = compress(messages, model="gpt-4o") # OpenAI or Anthropic message dicts # send result.messages instead of messages; log result.tokens_before / tokens_after / tokens_saved ``` In-process, no API key, no model download. `HEADROOM_OFFLINE=1` makes the run zero-network (no update check, no model or pricing-table downloads, no beacon). Verify without a provider key: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` - FastAPI example: https://docs.headroomlabs.ai/docs/fastapi - Full docs index for agents: https://docs.headroomlabs.ai/llms.txt (hand-curated subset: `llms.txt` in this repo) - Coding agents (Claude Code, Codex, Cursor, Copilot): `uv tool install --python 3.13 "headroom-ai[all]"` then `headroom wrap claude` - `headroom.integrations.asgi.CompressionMiddleware` is for apps that *serve* `/v1/chat/completions`-style routes, not for an app that *calls* a provider; use `compress()` for that - Canonical repo: https://github.com/headroomlabs-ai/headroom. `github.com/chopratejas/headroom` redirects here; there is no `headroom-sdk` org. The PyPI distribution named `headroom` is a different project. ## If you are changing this repo - Layout: Python package in `headroom/`, Rust core and PyO3 extension in `crates/` (built by maturin), TypeScript SDK in `sdk/typescript/`, docs site (Next.js + Fumadocs) in `docs/content/docs/`, agent plugins in `plugins/`, examples in `examples/`. - Setup: `uv sync --extra dev` and `make install-git-hooks` (pre-commit, commitlint, pre-push). Python checks: `uv run ruff check .`, `uv run ruff format .`, `uv run pytest`. Rust checks: `make ci-precheck-rust`. All gates: `make ci-precheck`. See `CONTRIBUTING.md`. - Commits and PR titles are Conventional Commits (`feat:`, `fix:`, `docs:`, ...). Never edit `CHANGELOG.md`; release-please writes it from PR titles. - Every external PR needs a "Real behavior proof" section (setup, exact commands, observed output, what was not tested). Bug fixes need a test that fails before the fix. - One docs site: `docs/`, deployed by Vercel to docs.headroomlabs.ai. `wiki/` and the `gh-pages` branch are not published; do not add content there. - Public API examples in `README.md`, `llms.txt`, `AGENTS.md` and `docs/` must use names that exist in `headroom/compress.py` (`compress`, `CompressResult.messages`, `.tokens_before`, `.tokens_after`, `.tokens_saved`, `.compression_ratio`, `.transforms_applied`). - Savings claims anywhere in the repo must match the measured table under "Proof" in `README.md`: 21-57% on the four agent scenarios, 90%+ on repeated JSON arrays and log lines, little on prose. - Version-carrying files (`pyproject.toml`, plugin and marketplace manifests, `sdk/typescript/package.json`, `server.json`) are kept in step by `scripts/version-sync.py` and gated by `scripts/verify-versions.py`; add any new one to both. - aistorehub.json (564 bytes)
{ "name": "Headroom", "slug": "headroom", "version": "1.0.0", "type": "mcp_server", "description": "Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/headroomlabs-ai/headroom", "tags": [ "trending", "headroomlabs-ai", "python", "github", "open-source" ], "imported_at": "2026-11-22T00:08:35.474Z" } - benchmarks/__init__.py (1204 bytes)
"""Headroom SDK Benchmark Suite. This package provides performance benchmarks for Headroom transforms and relevance scorers. Benchmarks use pytest-benchmark for accurate timing measurements. Usage: # Run all benchmarks pytest benchmarks/ --benchmark-only # Run specific suite pytest benchmarks/bench_transforms.py --benchmark-only # Generate comparison report python benchmarks/run_benchmarks.py --suite all --output report.md Performance Targets: - SmartCrusher: < 10ms for 1000 items - CacheAligner: < 1ms for date extraction - BM25Scorer: < 1ms for 100 items - HybridScorer: < 50ms for 100 items (with embeddings) """ __version__ = "0.2.0" from .scenarios.conversations import ( generate_agentic_conversation, generate_rag_conversation, ) from .scenarios.tool_outputs import ( generate_api_responses, generate_database_rows, generate_log_entries, generate_search_results, ) __all__ = [ # Data generators "generate_search_results", "generate_log_entries", "generate_api_responses", "generate_database_rows", # Conversation generators "generate_agentic_conversation", "generate_rag_conversation", ] - benchmarks/adversarial_ccr_tests.py (62319 bytes)
#!/usr/bin/env python3 """ Adversarial CCR Tests - Designed to BREAK Our Assumptions These tests are intentionally malicious, edge-casey, and designed to expose weaknesses in our compression and retrieval logic. Categories: 1. SEMANTIC ATTACKS: Data that tricks our heuristics 2. BOUNDARY CONDITIONS: Edge cases at limits 3. INJECTION ATTACKS: Malformed data designed to break parsing 4. RACE CONDITIONS: Concurrency attacks 5. MEMORY PRESSURE: Resource exhaustion 6. DECEPTIVE DATA: Items that look like one thing but are another Run with: python benchmarks/adversarial_ccr_tests.py """ from __future__ import annotations import concurrent.futures import gc import hashlib import json import random import sys import threading import time import uuid from dataclasses import dataclass, field from typing import Any from headroom.cache.compression_feedback import ( get_compression_feedback, reset_compression_feedback, ) from headroom.cache.compression_store import ( CompressionStore, RetrievalEvent, get_compression_store, reset_compression_store, ) from headroom.transforms.smart_crusher import ( SmartCrusherConfig, smart_crush_tool_output, ) @dataclass class AdversarialResult: """Result from an adversarial test.""" name: str category: str passed: bool = False expected_behavior: str = "" actual_behavior: str = "" severity: str = "medium" # low, medium, high, critical details: dict[str, Any] = field(default_factory=dict) def run_test(func) -> AdversarialResult: """Run a test and catch any exceptions.""" try: return func() except Exception as e: return AdversarialResult( name=func.__name__, category="exception", passed=False, expected_behavior="Test should complete without exception", actual_behavior=f"Exception: {type(e).__name__}: {str(e)[:200]}", severity="critical", ) # ============================================================================= # CATEGORY 1: SEMANTIC ATTACKS # ============================================================================= def test_all_items_are_errors() -> AdversarialResult: """ ATTACK: Every single item is an error. If we keep ALL errors, we keep everything = no compression. What SHOULD happen? Keep all? Sample errors? Fail gracefully? """ result = AdversarialResult( name="All Items Are Errors", category="semantic", expected_behavior="Should handle gracefully, possibly skip compression", severity="high", ) # 1000 items, ALL are errors items = [ { "id": i, "status": "error", "error_code": 500 + (i % 50), "message": f"Error at position {i}: something went wrong", } for i in range(1000) ] config = SmartCrusherConfig(max_items_after_crush=15) original_json = json.dumps(items) compressed_json, was_modified, reason = smart_crush_tool_output(original_json, config) compressed = json.loads(compressed_json) # What happened? if len(compressed) == 1000: result.actual_behavior = "Kept ALL 1000 items (no compression when all errors)" result.passed = True # This is actually correct behavior! elif len(compressed) == 15: result.actual_behavior = f"Compressed to 15 items, lost {1000 - 15} errors!" result.passed = False else: result.actual_behavior = f"Compressed to {len(compressed)} items" result.passed = len(compressed) >= 100 # Should keep most errors result.details = { "original": 1000, "compressed": len(compressed), "reason": reason, } return result def test_error_keyword_in_normal_data() -> AdversarialResult: """ ATTACK: Normal items contain "error" keyword in benign context. "The error rate for this metric is 0.001%" - NOT an error! "Error handling documentation" - NOT an error! """ result = AdversarialResult( name="Error Keyword False Positive", category="semantic", expected_behavior="Should NOT treat benign 'error' mentions as errors", severity="medium", ) items = [] # 100 normal items with "error" in benign context for i in range(100): items.append( { "id": i, "status": "success", # Clearly success! "message": random.choice( [ f"Error rate: 0.00{i}%", f"Error handling improved by {i}%", f"Zero errors detected in batch {i}", f"Error-free operation for {i} hours", "Documentation: How to handle errors", ] ), "value": i, } ) # Add 3 REAL errors real_error_ids = [25, 50, 75] for idx in real_error_ids: items[idx] = { "id": idx, "status": "error", # This is a REAL error "message": f"CRITICAL: System failure at {idx}", "error_code": 500, } config = SmartCrusherConfig(max_items_after_crush=15) original_json = json.dumps(items) compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) compressed = json.loads(compressed_json) # Count how many items with "error" in message were kept items_with_error_word = len( [item for item in compressed if "error" in str(item.get("message", "")).lower()] ) # Count real errors kept real_errors_kept = len([item for item in compressed if item.get("status") == "error"]) # False positives (keeping non-errors) are OK - conservative is good # False negatives (missing real errors) are NOT OK if real_errors_kept < 3: result.actual_behavior = ( f"Only kept {real_errors_kept}/3 real errors - missed actual errors!" ) result.passed = False else: # Keeping extra items with "error" word is fine - better safe than sorry result.actual_behavior = f"Kept all {real_errors_kept} real errors (+ {items_with_error_word} with 'error' word - conservative is OK)" result.passed = True result.details = { "total_compressed": len(compressed), "real_errors_kept": real_errors_kept, "items_with_error_word": items_with_error_word, } return result def test_needle_looks_exactly_like_hay() -> AdversarialResult: """ ATTACK: The critical item has NO distinguishing features. In a list of 1000 users, user #456 is the one we need. User #456 looks EXACTLY like every other user. """ result = AdversarialResult( name="Needle Identical to Hay", category="semantic", expected_behavior="CCR retrieval should still find specific item by ID", severity="high", ) reset_compression_store() store = get_compression_store() # 1000 identical-looking users target_id = 456 items = [ { "user_id": i, "name": f"User {i}", "status": "active", "created": "2025-01-01", } for i in range(1000) ] original_json = json.dumps(items) config = SmartCrusherConfig(max_items_after_crush=15) compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) # Store for CCR hash_key = store.store( original=original_json, compressed=compressed_json, original_item_count=1000, compressed_item_count=15, tool_name="user_search", ) # Recover the target user via CCR retrieval (hash-only → full content) entry_for_search = store.retrieve(hash_key) search_results = json.loads(entry_for_search.original_content) if entry_for_search else [] found_target = any(item.get("user_id") == target_id for item in search_results) if found_target: result.actual_behavior = "Found target user via CCR retrieval" result.passed = True else: # Try full retrieval as fallback entry = store.retrieve(hash_key) if entry: all_items = json.loads(entry.original_content) target_in_original = any(item.get("user_id") == target_id for item in all_items) if target_in_original: result.actual_behavior = "Search failed, but full retrieval works" result.passed = True # CCR still provides recovery path else: result.actual_behavior = "Data lost entirely!" result.passed = False else: result.actual_behavior = "CCR cache miss - data not found" result.passed = False result.details = { "target_id": target_id, "search_results": len(search_results), "found_target": found_target, } return result def test_anomaly_in_string_not_number() -> AdversarialResult: """ ATTACK: Anomaly is in a string field, not numeric. 999 items: region="us-east-1" 1 item: region="DEPRECATED-DO-NOT-USE" SmartCrusher detects numeric anomalies, but what about string outliers? """ result = AdversarialResult( name="String Anomaly Detection", category="semantic", expected_behavior="Should detect or preserve string outliers", severity="medium", ) items = [] anomaly_idx = 500 for i in range(1000): if i == anomaly_idx: items.append( { "id": i, "region": "DEPRECATED-DO-NOT-USE-CRITICAL-MIGRATION-REQUIRED", "status": "active", } ) else: items.append( { "id": i, "region": "us-east-1", "status": "active", } ) config = SmartCrusherConfig(max_items_after_crush=20) original_json = json.dumps(items) compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) compressed = json.loads(compressed_json) # Check if anomaly was preserved anomaly_preserved = any("DEPRECATED" in str(item.get("region", "")) for item in compressed) if anomaly_preserved: result.actual_behavior = "String anomaly was preserved" result.passed = True else: result.actual_behavior = "String anomaly was LOST - only numeric anomalies detected" result.passed = False result.details = { "compressed_count": len(compressed), "anomaly_preserved": anomaly_preserved, } return result # ============================================================================= # CATEGORY 2: BOUNDARY CONDITIONS # ============================================================================= def test_empty_array() -> AdversarialResult: """ ATTACK: Empty array input. """ result = AdversarialResult( name="Empty Array", category="boundary", expected_behavior="Should return empty array unchanged", severity="low", ) config = SmartCrusherConfig() compressed_json, was_modified, reason = smart_crush_tool_output("[]", config) if compressed_json == "[]" and not was_modified: result.actual_behavior = "Correctly handled empty array" result.passed = True else: result.actual_behavior = f"Unexpected result: {compressed_json[:100]}" result.passed = False return result def test_single_item_array() -> AdversarialResult: """ ATTACK: Array with exactly 1 item. """ result = AdversarialResult( name="Single Item Array", category="boundary", expected_behavior="Should return single item unchanged", severity="low", ) items = [{"id": 1, "value": "only_one"}] config = SmartCrusherConfig() compressed_json, was_modified, _ = smart_crush_tool_output(json.dumps(items), config) compressed = json.loads(compressed_json) if len(compressed) == 1 and compressed[0].get("id") == 1: result.actual_behavior = "Single item preserved" result.passed = True else: result.actual_behavior = f"Unexpected: {len(compressed)} items" result.passed = False return result def test_exactly_max_items() -> AdversarialResult: """ ATTACK: Array with exactly max_items_after_crush items. """ result = AdversarialResult( name="Exactly Max Items", category="boundary", expected_behavior="Should not compress when at exact limit", severity="low", ) config = SmartCrusherConfig(max_items_after_crush=15) items = [{"id": i} for i in range(15)] # Exactly 15 compressed_json, was_modified, _ = smart_crush_tool_output(json.dumps(items), config) compressed = json.loads(compressed_json) if len(compressed) == 15: result.actual_behavior = "Kept all 15 items as expected" result.passed = True else: result.actual_behavior = f"Changed count: {len(compressed)}" result.passed = False return result def test_max_items_plus_one() -> AdversarialResult: """ ATTACK: Array with max_items + 1. IMPORTANT: If data has high uniqueness and no importance signal, crushability analysis correctly skips compression to avoid data loss. This is the RIGHT behavior - don't blindly compress unique entities. """ result = AdversarialResult( name="Max Items Plus One", category="boundary", expected_behavior="Skip compression for unique entities OR compress with signal", severity="low", ) config = SmartCrusherConfig(max_items_after_crush=15, min_items_to_analyze=5) # Create items WITH a score field so compression can determine importance items = [{"id": i, "value": f"item_{i}", "score": 1.0 - (i / 100)} for i in range(16)] compressed_json, was_modified, reason = smart_crush_tool_output(json.dumps(items), config) compressed = json.loads(compressed_json) result.actual_behavior = f"Compressed to {len(compressed)} items ({reason})" # With a score signal, we should compress to max_items result.passed = len(compressed) <= 15 return result def test_hash_collision_attempt() -> AdversarialResult: """ ATTACK: Try to create hash collisions in CCR store. We use SHA256[:16] - what if two different contents hash the same? """ result = AdversarialResult( name="Hash Collision Attack", category="boundary", expected_behavior="Different content should not collide", severity="high", ) reset_compression_store() get_compression_store() # Store many different contents hashes = set() collisions = 0 for i in range(10000): content - benchmarks/agent_cost_benchmark.py (32273 bytes)
#!/usr/bin/env python3 """ Agent Cost Crisis Benchmark - The Compelling Story This benchmark demonstrates WHY Headroom matters by showing: 1. THE PROBLEM: Context explosion in real-world agent workloads - Tokens grow exponentially with conversation length - Tool outputs dominate context (often 70%+ of tokens) - Dynamic content breaks cache efficiency 2. THE SOLUTION: Headroom's impact on real workloads - Token reduction from SmartCrusher (50-80% on tool outputs) - Cache alignment improvement (10x+ potential savings) - Context windowing (stay within limits without losing info) 3. THE PROOF: Quality preservation - Critical information retained (errors, anomalies, relevant items) - Agent task completion unaffected - Information retrieval accuracy maintained Usage: python benchmarks/agent_cost_benchmark.py python benchmarks/agent_cost_benchmark.py --format markdown > BENCHMARK.md python benchmarks/agent_cost_benchmark.py --scenario coding-agent """ from __future__ import annotations import argparse import json import statistics import time from dataclasses import dataclass, field from typing import Any # Benchmark scenario imports from benchmarks.scenarios.conversations import ( generate_agentic_conversation, generate_rag_conversation, ) from benchmarks.scenarios.tool_outputs import ( generate_log_entries, generate_search_results, ) # Headroom imports from headroom.transforms.smart_crusher import SmartCrusherConfig, smart_crush_tool_output # ============================================================================= #: Share of turns in a steady-state agent session whose prefix is already #: cached. A 50-turn session writes its prefix once and reads it on the other #: 49 turns, so 0.98 is the arithmetic rather than a guess; the round 0.95 below #: leaves room for the TTL expiries a real session hits mid-run. Used ONLY by #: this synthetic benchmark -- the proxy never assumes a hit rate, it reads the #: provider's reported mix per request (headroom.pricing.counterfactual). WARM_CACHE_SHARE = 0.95 # PRICING DATA (as of 2025) # ============================================================================= PRICING = { # Anthropic Claude 3.5 Sonnet "claude-3.5-sonnet": { "input": 3.00 / 1_000_000, # $3 per 1M tokens "output": 15.00 / 1_000_000, # $15 per 1M tokens "cached_input": 0.30 / 1_000_000, # 90% discount on cache hit "cache_write": 3.75 / 1_000_000, # 25% premium to write cache }, # OpenAI GPT-4o "gpt-4o": { "input": 2.50 / 1_000_000, "output": 10.00 / 1_000_000, "cached_input": 1.25 / 1_000_000, # 50% discount }, # Google Gemini 1.5 Pro "gemini-1.5-pro": { "input": 1.25 / 1_000_000, "output": 5.00 / 1_000_000, "cached_input": 0.3125 / 1_000_000, # 75% discount }, } # Approximate tokens per character (GPT-4 tokenizer average) CHARS_PER_TOKEN = 4 def _warm_savings(tokens_saved: int, pricing: dict) -> float: """Value ``tokens_saved`` for a session whose prefix is mostly already cached. The share given by :data:`WARM_CACHE_SHARE` would have been billed as cache READS had those tokens been sent; the rest as ordinary input. For Anthropic that is a 10x difference per token, which is why the cold-only figure this benchmark used to print was the single most misleading number it produced. Falls back to the list rate for a pricing row with no cached rate, which is the correct answer for a provider that does not discount cache reads. """ if tokens_saved <= 0: return 0.0 cached_rate = pricing.get("cached_input", pricing["input"]) warm = tokens_saved * WARM_CACHE_SHARE * cached_rate cold = tokens_saved * (1.0 - WARM_CACHE_SHARE) * pricing["input"] return warm + cold @dataclass class CostAnalysis: """Cost analysis for a workload.""" tokens_input: int = 0 tokens_output: int = 0 tokens_cached: int = 0 cost_baseline: float = 0.0 cost_optimized: float = 0.0 cost_with_cache: float = 0.0 savings_from_compression: float = 0.0 savings_from_caching: float = 0.0 total_savings_percent: float = 0.0 # Compression savings under a WARM cache -- the realistic steady state for # an agent session, where the prefix is written once and read on every # later turn. `savings_from_compression` above is the COLD figure: it # prices every removed token as a cache miss, which is true only for the # first request of a cache window. Reporting the cold number alone # overstates a long session's saving by up to 10x, so both are carried and # the report names which assumption each one is under. savings_from_compression_warm: float = 0.0 #: Fraction of turns whose prefix was already cached. See WARM_CACHE_SHARE. cache_profile: str = "cold" @dataclass class BenchmarkResult: """Result from a single benchmark scenario.""" name: str description: str # Token metrics tokens_original: int = 0 tokens_optimized: int = 0 compression_ratio: float = 0.0 # Cache metrics cache_hit_rate_baseline: float = 0.0 cache_hit_rate_optimized: float = 0.0 # Quality metrics critical_items_retained: int = 0 critical_items_total: int = 0 retention_rate: float = 0.0 # Cost analysis cost_analysis: CostAnalysis = field(default_factory=CostAnalysis) # Performance optimization_latency_ms: float = 0.0 # Details details: dict[str, Any] = field(default_factory=dict) # ============================================================================= # SCENARIO 1: Coding Agent Context Explosion # ============================================================================= def benchmark_coding_agent_explosion() -> BenchmarkResult: """ Simulate a Claude Code / Cursor style coding agent session. Shows how context explodes as the agent: - Searches codebase (100s of file snippets) - Reads documentation (large text blocks) - Makes tool calls (grep, find, read) - Accumulates conversation history """ result = BenchmarkResult( name="Coding Agent Context Explosion", description="50-turn coding session with file search, grep, and documentation lookups", ) # Generate realistic coding agent conversation messages = generate_agentic_conversation( turns=50, tool_calls_per_turn=2, items_per_tool_response=100, # 100 search results per tool call ) # Calculate original tokens original_content = json.dumps(messages) result.tokens_original = len(original_content) // CHARS_PER_TOKEN # Apply Headroom transforms using convenience function config = SmartCrusherConfig(max_items_after_crush=20) start = time.perf_counter() optimized_messages = [] critical_retained = 0 critical_total = 0 for msg in messages: if msg.get("role") == "tool": # Parse tool content as JSON array try: original_content = msg.get("content", "[]") content = json.loads(original_content) if isinstance(content, list) and len(content) > 10: # Count critical items (errors, high-relevance) for item in content: if isinstance(item, dict): if item.get("error") or item.get("status") == "failed": critical_total += 1 if item.get("is_needle"): critical_total += 1 # Compress with SmartCrusher convenience function compressed_str, was_modified, _ = smart_crush_tool_output( original_content, config ) if was_modified: compressed = json.loads(compressed_str) # Count retained critical items for item in compressed: if isinstance(item, dict): if item.get("error") or item.get("status") == "failed": critical_retained += 1 if item.get("is_needle"): critical_retained += 1 msg = {**msg, "content": compressed_str} except (json.JSONDecodeError, TypeError): pass optimized_messages.append(msg) result.optimization_latency_ms = (time.perf_counter() - start) * 1000 # Calculate optimized tokens optimized_content = json.dumps(optimized_messages) result.tokens_optimized = len(optimized_content) // CHARS_PER_TOKEN # Calculate metrics result.compression_ratio = 1 - (result.tokens_optimized / result.tokens_original) result.critical_items_total = critical_total result.critical_items_retained = critical_retained result.retention_rate = critical_retained / critical_total if critical_total > 0 else 1.0 # Cost analysis (using Claude 3.5 Sonnet pricing) pricing = PRICING["claude-3.5-sonnet"] result.cost_analysis = CostAnalysis( tokens_input=result.tokens_original, cost_baseline=result.tokens_original * pricing["input"], cost_optimized=result.tokens_optimized * pricing["input"], savings_from_compression=(result.tokens_original - result.tokens_optimized) * pricing["input"], savings_from_compression_warm=_warm_savings( result.tokens_original - result.tokens_optimized, pricing ), cache_profile="cold+warm", ) result.cost_analysis.total_savings_percent = result.compression_ratio * 100 result.details = { "turns": 50, "tool_calls": 100, "items_per_response": 100, "items_after_compression": 20, } return result # ============================================================================= # SCENARIO 2: Cache Alignment Impact # ============================================================================= def benchmark_cache_alignment() -> BenchmarkResult: """ Show how dynamic content breaks caching and how CacheAligner fixes it. Simulates 100 requests with same base prompt but different dates. Without alignment: 0% cache hits With alignment: 90%+ cache hits """ from headroom.cache import DetectorConfig, DynamicContentDetector result = BenchmarkResult( name="Cache Alignment Impact", description="100 requests with dynamic dates - cache hit improvement", ) # Base system prompt with dynamic date base_prompt = """You are Claude, an AI assistant by Anthropic. Today is {date}. Current time: {time}. Session ID: {session_id} Request ID: {request_id} You are a helpful coding assistant. Follow these guidelines: 1. Write clean, readable code 2. Add appropriate comments 3. Handle errors gracefully 4. Follow best practices Be concise and helpful.""" import datetime import uuid # Use DynamicContentDetector to extract static content detector = DynamicContentDetector(DetectorConfig(tiers=["regex"])) # Simulate 100 requests over a day prompts_original = [] prompts_aligned = [] base_date = datetime.datetime(2025, 1, 15, 9, 0, 0) for i in range(100): # Each request has different timestamp request_time = base_date + datetime.timedelta(minutes=i * 5) prompt = base_prompt.format( date=request_time.strftime("%A, %B %d, %Y"), time=request_time.strftime("%I:%M %p"), session_id=f"sess_{uuid.uuid4().hex[:24]}", request_id=f"req_{uuid.uuid4().hex[:24]}", ) prompts_original.append(prompt) # Extract static content for cache alignment detection_result = detector.detect(prompt) prompts_aligned.append(detection_result.static_content) # Calculate cache hits # Baseline: all prompts are different (dynamic dates) unique_original = len(set(prompts_original)) cache_hits_baseline = 100 - unique_original # Aligned: static prefixes should be identical unique_aligned = len(set(prompts_aligned)) cache_hits_aligned = 100 - unique_aligned result.cache_hit_rate_baseline = cache_hits_baseline / 100 result.cache_hit_rate_optimized = cache_hits_aligned / 100 # Token calculation result.tokens_original = sum(len(p) // CHARS_PER_TOKEN for p in prompts_original) # Cost analysis with caching pricing = PRICING["claude-3.5-sonnet"] tokens_per_request = len(prompts_original[0]) // CHARS_PER_TOKEN # Baseline: pay full price every time (no cache hits) cost_baseline = 100 * tokens_per_request * pricing["input"] # Optimized: first request is cache write, rest are cache hits first_request_cost = tokens_per_request * pricing["cache_write"] cached_requests_cost = 99 * tokens_per_request * pricing["cached_input"] cost_optimized = first_request_cost + cached_requests_cost result.cost_analysis = CostAnalysis( tokens_input=result.tokens_original, cost_baseline=cost_baseline, cost_with_cache=cost_optimized, savings_from_caching=cost_baseline - cost_optimized, total_savings_percent=((cost_baseline - cost_optimized) / cost_baseline) * 100, ) result.details = { "total_requests": 100, "unique_prompts_baseline": unique_original, "unique_prompts_aligned": unique_aligned, "cache_improvement_factor": f"{(cache_hits_aligned - cache_hits_baseline)}x", } return result # ============================================================================= # SCENARIO 3: RAG Context Scaling # ============================================================================= def benchmark_rag_scaling() -> BenchmarkResult: """ Show how RAG context grows and how Headroom manages it. Simulates large RAG context with multiple queries. """ result = BenchmarkResult( name="RAG Context Scaling", description="Large RAG context (~50K tokens) with compression" ) # Generate RAG conversation with ~50K tokens of context messages = generate_rag_conversation( context_tokens=50000, num_queries=10, ) original_content = json.dumps(messages) result.tokens_original = len(original_content) // CHARS_PER_TOKEN # Apply transforms - compress tool outputs in messages config = SmartCrusherConfig(max_items_after_crush=10) start = time.perf_counter() # Compress tool outputs in messages optimized_messages = [] for msg in messages: if msg.get("role") == "tool": try: original_content_msg = msg.get("content", "[]") compressed_str, was_modified, _ = smart_crush_tool_output( - benchmarks/bench_latency.py (45665 bytes)
#!/usr/bin/env python3 """Latency benchmark for Headroom compression pipeline. Measures compression overhead across content types and input sizes, profiles individual transform stages, and computes cost-benefit analysis to answer: "Does the token savings outweigh added processing time?" Usage: # Run with terminal output (default) python benchmarks/bench_latency.py # Save markdown report python benchmarks/bench_latency.py --output docs/LATENCY_BENCHMARKS.md # Save JSON results python benchmarks/bench_latency.py --json latency_results.json # Custom iterations python benchmarks/bench_latency.py --iterations 50 # Run specific content type only python benchmarks/bench_latency.py --scenario json python benchmarks/bench_latency.py --scenario code python benchmarks/bench_latency.py --scenario text python benchmarks/bench_latency.py --scenario logs python benchmarks/bench_latency.py --scenario agentic Scenarios: json - JSON arrays via SmartCrusher (100-5K items) code - Python source via CodeCompressor (50-1000 lines) text - Plain text/RAG via Kompress fallback (1K-50K tokens) logs - Structured logs via LogCompressor (100-5K entries) agentic - Multi-turn agent conversations (10-100 turns) rag - RAG conversations with large context (5K-50K tokens) """ from __future__ import annotations import argparse import json import math import platform import random import statistics import sys import time from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any # --------------------------------------------------------------------------- # Ensure benchmarks package is importable when running as script # --------------------------------------------------------------------------- _repo_root = Path(__file__).resolve().parent.parent if str(_repo_root) not in sys.path: sys.path.insert(0, str(_repo_root)) from benchmarks.scenarios.conversations import ( # noqa: E402 generate_agentic_conversation, generate_rag_conversation, ) from benchmarks.scenarios.tool_outputs import ( # noqa: E402 generate_api_responses, generate_database_rows, generate_log_entries, generate_search_results, ) # --------------------------------------------------------------------------- # Constants # --------------------------------------------------------------------------- # LLM prefill rates (ms per input token) for cost-benefit analysis. # These are conservative estimates based on published benchmarks and represent # the incremental TTFT contribution per additional input token. MODEL_PROFILES: dict[str, dict[str, float]] = { "gpt-4o-mini": { "ms_per_token": 0.01, "price_per_mtok_input": 0.15, "label": "GPT-4o Mini", }, "gpt-4o": { "ms_per_token": 0.03, "price_per_mtok_input": 2.50, "label": "GPT-4o", }, "claude-sonnet-4-5": { "ms_per_token": 0.03, "price_per_mtok_input": 3.00, "label": "Claude Sonnet 4.5", }, "claude-opus-4": { "ms_per_token": 0.08, "price_per_mtok_input": 15.00, "label": "Claude Opus 4", }, } # Reference model for the main report table REFERENCE_MODEL = "claude-sonnet-4-5" # --------------------------------------------------------------------------- # Data classes # --------------------------------------------------------------------------- @dataclass class Scenario: """A benchmark scenario to measure.""" name: str content_type: str # json, code, text, logs, agentic, rag size_label: str # Human-readable size (e.g., "100 items", "50 turns") messages: list[dict[str, Any]] model_limit: int = 200_000 # Context limit for pipeline @dataclass class TransformTiming: """Timing for a single transform within the pipeline.""" name: str durations_ms: list[float] = field(default_factory=list) @property def p50_ms(self) -> float: if not self.durations_ms: return 0.0 s = sorted(self.durations_ms) return s[len(s) // 2] @property def mean_ms(self) -> float: return statistics.mean(self.durations_ms) if self.durations_ms else 0.0 @dataclass class LatencyResult: """Result of benchmarking a single scenario.""" scenario_name: str content_type: str size_label: str tokens_before: int tokens_after: int tokens_saved: int compression_ratio: float num_messages: int timings_ms: list[float] # Full pipeline timings per iteration transform_timings: dict[str, TransformTiming] = field(default_factory=dict) transforms_applied: list[str] = field(default_factory=list) @property def p50_ms(self) -> float: s = sorted(self.timings_ms) return s[len(s) // 2] @property def p95_ms(self) -> float: s = sorted(self.timings_ms) idx = int(math.ceil(0.95 * len(s))) - 1 return s[max(0, idx)] @property def p99_ms(self) -> float: s = sorted(self.timings_ms) idx = int(math.ceil(0.99 * len(s))) - 1 return s[max(0, idx)] @property def mean_ms(self) -> float: return statistics.mean(self.timings_ms) @property def stddev_ms(self) -> float: return statistics.stdev(self.timings_ms) if len(self.timings_ms) > 1 else 0.0 @property def min_ms(self) -> float: return min(self.timings_ms) @property def max_ms(self) -> float: return max(self.timings_ms) def to_dict(self) -> dict[str, Any]: return { "scenario_name": self.scenario_name, "content_type": self.content_type, "size_label": self.size_label, "tokens_before": self.tokens_before, "tokens_after": self.tokens_after, "tokens_saved": self.tokens_saved, "compression_ratio": self.compression_ratio, "num_messages": self.num_messages, "iterations": len(self.timings_ms), "p50_ms": round(self.p50_ms, 3), "p95_ms": round(self.p95_ms, 3), "p99_ms": round(self.p99_ms, 3), "mean_ms": round(self.mean_ms, 3), "stddev_ms": round(self.stddev_ms, 3), "min_ms": round(self.min_ms, 3), "max_ms": round(self.max_ms, 3), "transforms_applied": self.transforms_applied, "transform_breakdown": { name: { "p50_ms": round(tt.p50_ms, 3), "mean_ms": round(tt.mean_ms, 3), } for name, tt in self.transform_timings.items() }, } # --------------------------------------------------------------------------- # Code generation (for CodeCompressor scenarios) # --------------------------------------------------------------------------- def _generate_python_function(name: str, lines: int) -> str: """Generate a realistic Python function.""" parts = [f"def {name}(data: list[dict], config: dict | None = None) -> dict:"] parts.append(f' """Process {name.replace("_", " ")} and return results."""') parts.append(" if config is None:") parts.append(" config = {}") parts.append(f' results = {{"function": "{name}", "items": []}}') parts.append(" errors = []") parts.append("") # Fill body to target line count for i in range(max(0, lines - 12)): kind = i % 5 if kind == 0: parts.append(f" for item in data[{i}:{i + 10}]:") parts.append(f' value = item.get("field_{i}", None)') elif kind == 1: parts.append(f" if len(results['items']) > {i * 10}:") parts.append(' results["overflow"] = True') elif kind == 2: parts.append(" try:") parts.append(f" computed = sum(x.get('value', 0) for x in data[:{i + 5}])") parts.append(f' results["computed_{i}"] = computed') elif kind == 3: parts.append(" except (KeyError, TypeError) as exc:") parts.append(f' errors.append({{"step": {i}, "error": str(exc)}})') else: parts.append(f" # Step {i}: aggregate intermediate results") parts.append(f' results["step_{i}"] = len(data)') parts.append("") parts.append(' results["errors"] = errors') parts.append(" return results") return "\n".join(parts) def generate_python_code(target_lines: int) -> str: """Generate a realistic Python module of approximately `target_lines` lines.""" sections = [ '"""Auto-generated benchmark module for code compression testing."""', "", "from __future__ import annotations", "", "import json", "import logging", "import os", "from dataclasses import dataclass, field", "from typing import Any", "", "logger = logging.getLogger(__name__)", "", "", "@dataclass", "class ProcessingConfig:", ' """Configuration for data processing."""', "", " batch_size: int = 100", " max_retries: int = 3", " timeout_seconds: float = 30.0", " output_format: str = 'json'", " debug: bool = False", "", "", ] current_lines = len(sections) func_idx = 0 while current_lines < target_lines: remaining = target_lines - current_lines func_lines = min(remaining, random.randint(15, 40)) func_name = f"process_batch_{func_idx}" func_code = _generate_python_function(func_name, func_lines) sections.append(func_code) sections.append("") sections.append("") current_lines += func_lines + 2 func_idx += 1 return "\n".join(sections[:target_lines]) # --------------------------------------------------------------------------- # Plain text generation # --------------------------------------------------------------------------- def generate_plain_text(target_tokens: int) -> str: """Generate realistic plain text content (technical documentation).""" # ~4 chars per token target_chars = target_tokens * 4 paragraphs = [ "The system architecture follows a microservices pattern with clear separation of concerns. " "Each service owns its data store and communicates through well-defined APIs. Event-driven " "messaging handles asynchronous workflows, while synchronous REST APIs serve real-time " "requests. The API gateway handles routing, authentication, and rate limiting at the edge.", "Database optimization is critical for maintaining low-latency responses under load. " "We use connection pooling with a minimum of 10 and maximum of 100 connections per service. " "Read replicas handle analytics queries to avoid impacting transactional workloads. " "Indexes are maintained on frequently queried columns with regular analysis of query plans.", "The caching layer uses a tiered approach: L1 in-memory caches with a 60-second TTL for " "hot data, L2 Redis caches with a 5-minute TTL for frequently accessed resources, and L3 " "CDN caching for static assets. Cache invalidation follows a pub/sub pattern to ensure " "consistency across service instances without requiring cache stampede protection.", "Monitoring and observability are built into every service from day one. Structured logging " "with correlation IDs enables distributed tracing across service boundaries. Metrics are " "collected via Prometheus and visualized in Grafana dashboards. Alerts are configured for " "SLO violations with appropriate severity levels and escalation paths.", "The deployment pipeline uses blue-green deployments with automated canary analysis. Each " "deployment is validated against health checks, latency percentiles, and error rate thresholds " "before traffic is shifted. Rollback is automated if any SLO is breached during the canary " "window, typically set to 15 minutes for non-critical services.", "Security follows a defense-in-depth strategy with multiple layers of protection. All " "inter-service communication uses mTLS with certificate rotation every 90 days. API " "authentication uses short-lived JWT tokens with refresh token rotation. Secrets are " "managed through HashiCorp Vault with automatic rotation policies.", "Error handling follows a consistent pattern across all services. Transient errors trigger " "exponential backoff with jitter, starting at 100ms and capping at 30 seconds. Circuit " "breakers prevent cascade failures by opening after 5 consecutive failures and attempting " "a half-open state after 60 seconds. All errors are classified by severity and tracked " "as structured events for post-incident analysis.", "Performance testing is integrated into the CI/CD pipeline. Load tests run against a " "staging environment that mirrors production topology. Baseline metrics are captured for " "each release candidate and compared against the previous stable release. Regressions " "greater than 10% in p99 latency automatically block the deployment.", ] result: list[str] = [] current_chars = 0 while current_chars < target_chars: para = random.choice(paragraphs) result.append(para) result.append("") current_chars += len(para) + 1 return "\n".join(result)[:target_chars] # --------------------------------------------------------------------------- # Scenario generators # --------------------------------------------------------------------------- def _wrap_as_tool_message(content: str) -> list[dict[str, Any]]: """Wrap content as a minimal tool-call conversation.""" return [ {"role": "system", "content": "You are a helpful assistant.\n\nCurrent date: 2025-01-06"}, {"role": "user", "content": "Analyze the following data."}, { "role": "assistant", "content": None, "tool_calls": [ { "id": "call_bench_1", "type": "function", "function": {"name": "get_data", "arguments": "{}"}, } ], }, {"role": "tool", "tool_call_id": "call_bench_1", "content": content}, ] def generate_scenarios(content_types: list[str] | None = None) -> list[Scenario]: """Generate all benchmark scenarios. Args: content_types: Limit to specific types. None = all. Returns: List of Scenario objects ready for benchmarking. """ all_types = {"json", "code", "text", "logs", "agenti - benchmarks/bench_relevance.py (13502 bytes)
"""Relevance scorer benchmarks for Headroom SDK. This module contains performance benchmarks for relevance scorers: - BM25Scorer: Zero-dependency keyword matching - HybridScorer: BM25 + embedding fusion (with graceful fallback) Performance Targets: BM25Scorer: - Single item: < 0.1ms - Batch 100: < 1ms - Batch 1000: < 10ms HybridScorer (BM25 fallback): - Single item: < 0.2ms - Batch 100: < 2ms HybridScorer (with embeddings): - Single item: < 5ms - Batch 100: < 50ms Run with: pytest benchmarks/bench_relevance.py --benchmark-only -v """ from __future__ import annotations import json import pytest def _check_embedding_available() -> bool: """Check if sentence-transformers is available for embedding tests.""" try: import sentence_transformers # noqa: F401 return True except ImportError: return False class TestBM25Benchmarks: """Benchmarks for BM25 keyword relevance scorer. BM25Scorer performs: - Text tokenization (regex-based) - IDF computation - BM25 score calculation - Long-token bonus (UUIDs, IDs) Expected performance: - O(n*m) where n=tokens in item, m=tokens in query - Single item: < 0.1ms - Batch operations are linear with items """ @pytest.fixture def scorer(self): """Create BM25 scorer instance.""" from headroom.relevance.bm25 import BM25Scorer return BM25Scorer() def test_single_item( self, benchmark, scorer, json_items_100, query_context_uuid, ): """Benchmark scoring a single item. Target: < 0.1ms Tests basic scoring overhead. """ item = json_items_100[0] result = benchmark(scorer.score, item, query_context_uuid) assert result.score >= 0.0 assert result.score <= 1.0 def test_batch_100( self, benchmark, scorer, json_items_100, query_context_uuid, ): """Benchmark scoring 100 items in batch. Target: < 1ms Tests typical batch size for SmartCrusher. """ results = benchmark(scorer.score_batch, json_items_100, query_context_uuid) assert len(results) == 100 assert all(0.0 <= r.score <= 1.0 for r in results) def test_batch_1000( self, benchmark, scorer, json_items_1000, query_context_uuid, ): """Benchmark scoring 1000 items in batch. Target: < 10ms Tests larger batch for stress testing. """ results = benchmark(scorer.score_batch, json_items_1000, query_context_uuid) assert len(results) == 1000 def test_uuid_matching( self, benchmark, scorer, json_items_100, query_context_uuid, ): """Benchmark UUID pattern matching. Target: < 1ms Tests regex efficiency for UUID detection. """ # Query contains UUID - tests that BM25 can handle long token patterns results = benchmark(scorer.score_batch, json_items_100, query_context_uuid) # Verify scoring completes - specific matches depend on generated data assert len(results) == 100 assert all(r.score >= 0.0 for r in results) def test_semantic_query( self, benchmark, scorer, json_items_100, query_context_semantic, ): """Benchmark semantic query (BM25 limitations). Target: < 1ms Tests keyword matching on semantic queries. """ # BM25 will only match literal terms results = benchmark(scorer.score_batch, json_items_100, query_context_semantic) assert len(results) == 100 def test_empty_context( self, benchmark, scorer, json_items_100, ): """Benchmark with empty query context. Target: < 0.5ms Tests early-exit optimization. """ results = benchmark(scorer.score_batch, json_items_100, "") # All scores should be 0 with no context assert all(r.score == 0.0 for r in results) def test_long_items( self, benchmark, scorer, log_entries_1000, query_context_semantic, ): """Benchmark scoring longer items (log entries). Target: < 15ms Tests performance with larger text per item. """ json_items = [json.dumps(entry) for entry in log_entries_1000] results = benchmark(scorer.score_batch, json_items, query_context_semantic) assert len(results) == 1000 class TestHybridBenchmarks: """Benchmarks for Hybrid BM25+Embedding scorer. HybridScorer performs: - BM25 scoring (always) - Embedding scoring (if available) - Adaptive alpha computation - Score fusion Without embeddings (fallback mode): - Single item: < 0.2ms - Batch 100: < 2ms With embeddings (full mode): - Single item: < 5ms (model inference) - Batch 100: < 50ms (batched inference) """ @pytest.fixture def scorer_fallback(self): """Create hybrid scorer without embeddings (BM25 fallback).""" from headroom.relevance.bm25 import BM25Scorer from headroom.relevance.hybrid import HybridScorer # Force BM25-only mode by not providing embedding scorer scorer = HybridScorer( alpha=0.5, adaptive=True, bm25_scorer=BM25Scorer(), embedding_scorer=None, ) # Ensure we're in fallback mode scorer._embedding_available = False return scorer @pytest.fixture def scorer_full(self): """Create hybrid scorer with embeddings (if available).""" from headroom.relevance.hybrid import HybridScorer scorer = HybridScorer(alpha=0.5, adaptive=True) return scorer def test_single_item_fallback( self, benchmark, scorer_fallback, json_items_100, query_context_uuid, ): """Benchmark single item scoring (BM25 fallback). Target: < 0.2ms Tests fallback mode overhead. """ item = json_items_100[0] result = benchmark(scorer_fallback.score, item, query_context_uuid) assert "BM25 only" in result.reason def test_batch_100_fallback( self, benchmark, scorer_fallback, json_items_100, query_context_uuid, ): """Benchmark batch scoring (BM25 fallback). Target: < 2ms Tests fallback batch performance. """ results = benchmark(scorer_fallback.score_batch, json_items_100, query_context_uuid) assert len(results) == 100 def test_adaptive_alpha_uuid( self, benchmark, scorer_fallback, json_items_100, query_context_uuid, ): """Benchmark adaptive alpha with UUID query. Target: < 2ms Tests alpha computation overhead. """ results = benchmark(scorer_fallback.score_batch, json_items_100, query_context_uuid) # UUID query should favor BM25 (but we're in fallback mode) assert len(results) == 100 def test_adaptive_alpha_semantic( self, benchmark, scorer_fallback, json_items_100, query_context_semantic, ): """Benchmark adaptive alpha with semantic query. Target: < 2ms Tests alpha computation for semantic queries. """ results = benchmark(scorer_fallback.score_batch, json_items_100, query_context_semantic) assert len(results) == 100 @pytest.mark.skipif( not _check_embedding_available(), reason="sentence-transformers not installed", ) def test_single_item_full( self, benchmark, scorer_full, json_items_100, query_context_uuid, ): """Benchmark single item with embeddings. Target: < 5ms Tests full hybrid mode (requires sentence-transformers). """ if not scorer_full.has_embedding_support(): pytest.skip("Embeddings not available") item = json_items_100[0] result = benchmark(scorer_full.score, item, query_context_uuid) # Should show hybrid scoring assert "Hybrid" in result.reason @pytest.mark.skipif( not _check_embedding_available(), reason="sentence-transformers not installed", ) def test_batch_100_full( self, benchmark, scorer_full, json_items_100, query_context_uuid, ): """Benchmark batch scoring with embeddings. Target: < 50ms Tests batched embedding inference. """ if not scorer_full.has_embedding_support(): pytest.skip("Embeddings not available") results = benchmark(scorer_full.score_batch, json_items_100, query_context_uuid) assert len(results) == 100 class TestScorerFactoryBenchmarks: """Benchmarks for scorer factory and initialization.""" def test_create_bm25_scorer(self, benchmark): """Benchmark BM25 scorer creation. Target: < 0.1ms Tests initialization overhead. """ from headroom.relevance import create_scorer scorer = benchmark(create_scorer, tier="bm25") assert scorer is not None def test_create_hybrid_scorer(self, benchmark): """Benchmark hybrid scorer creation. Target: < 1ms (without embedding model load) Tests initialization with fallback. """ from headroom.relevance import create_scorer scorer = benchmark(create_scorer, tier="hybrid") assert scorer is not None class TestRelevanceInSmartCrusher: """Benchmarks for relevance scoring within SmartCrusher context. Tests the realistic scenario where SmartCrusher uses relevance scoring to determine which items to preserve during compression. """ @pytest.fixture def crusher_with_bm25(self, smart_crusher_config): """SmartCrusher with BM25 relevance scorer.""" from headroom.config import RelevanceScorerConfig from headroom.transforms.smart_crusher import SmartCrusher return SmartCrusher( config=smart_crusher_config, relevance_config=RelevanceScorerConfig(tier="bm25"), ) @pytest.fixture def crusher_with_hybrid(self, smart_crusher_config): """SmartCrusher with hybrid relevance scorer.""" from headroom.config import RelevanceScorerConfig from headroom.transforms.smart_crusher import SmartCrusher return SmartCrusher( config=smart_crusher_config, relevance_config=RelevanceScorerConfig(tier="hybrid"), ) def test_crush_with_bm25_relevance( self, benchmark, crusher_with_bm25, mock_tokenizer, items_100, ): """Benchmark crushing with BM25 relevance scoring. Target: < 3ms Tests BM25 integration overhead. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Find user 550e8400-e29b-41d4-a716-446655440000"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_100), }, ] result = benchmark(crusher_with_bm25.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_crush_with_hybrid_relevance( self, benchmark, crusher_with_hybrid, mock_tokenizer, items_100, ): """Benchmark crushing with hybrid relevance scoring. Target: < 60ms (with embeddings) or < 3ms (fallback) Tests hybrid integration. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Show me failed requests and errors"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_100), }, ] result = benchmark(crusher_with_hybrid.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_crush_large_with_relevance( self, benchmark, crusher_with_bm25, mock_tokenizer, items_1000, ): """Benchmark crushing 1000 items with relevance. Target: < 15ms Tests scalability of relevance scoring. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Search for Alice and find any errors"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_1000), }, ] result = benchmark(crusher_with_bm25.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def _check_embedding_available() -> bool: """Check if embedding scorer is available.""" try: from headroom.relevance.embedding import EmbeddingScorer return EmbeddingScorer.is_available() except ImportError: return False - benchmarks/bench_transforms.py (13438 bytes)
"""Transform benchmarks for Headroom SDK. This module contains performance benchmarks for Headroom transforms: - SmartCrusher: Statistical tool output compression - CacheAligner: Cache-aligned prefix optimization Performance Targets: SmartCrusher: - 100 items: < 2ms - 1000 items: < 10ms - 10000 items: < 100ms CacheAligner: - Date extraction: < 1ms - Hash computation: < 0.5ms Run with: pytest benchmarks/bench_transforms.py --benchmark-only -v """ from __future__ import annotations import json import pytest class TestSmartCrusherBenchmarks: """Benchmarks for SmartCrusher statistical compression. SmartCrusher performs: - Array analysis (field statistics, pattern detection) - Change point detection for numeric fields - Relevance scoring against query context - Strategic sampling (first K, last K, errors, anomalies) Expected performance: - O(n) for array analysis - O(n) for relevance scoring (BM25) - Total: < 10ms for 1000 items """ @pytest.fixture def crusher(self, smart_crusher_config): """Create SmartCrusher instance.""" from headroom.transforms.smart_crusher import SmartCrusher return SmartCrusher(config=smart_crusher_config) def test_compress_100_items( self, benchmark, crusher, mock_tokenizer, items_100, ): """Benchmark crushing 100 search results. Target: < 2ms This is the typical size for API responses. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Search for users"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_100), }, ] result = benchmark(crusher.apply, messages, mock_tokenizer) # Verify compression occurred assert result.tokens_after < result.tokens_before assert len(result.transforms_applied) > 0 def test_compress_1000_items( self, benchmark, crusher, mock_tokenizer, items_1000, ): """Benchmark crushing 1000 search results. Target: < 10ms This tests larger tool outputs from extensive searches. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Search for all users"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_1000), }, ] result = benchmark(crusher.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_compress_10000_items( self, benchmark, crusher, mock_tokenizer, items_10000, ): """Benchmark crushing 10000 search results. Target: < 100ms Stress test for very large tool outputs. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Export all data"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_10000), }, ] result = benchmark(crusher.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_analyze_log_entries( self, benchmark, crusher, mock_tokenizer, log_entries_1000, ): """Benchmark crushing log entries (cluster detection). Target: < 15ms Tests cluster sampling strategy for repetitive logs. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Show recent logs"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(log_entries_1000), }, ] result = benchmark(crusher.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_analyze_metrics_with_anomalies( self, benchmark, crusher, mock_tokenizer, database_rows_1000, ): """Benchmark crushing metrics data (anomaly detection). Target: < 15ms Tests change point detection and anomaly preservation. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Get CPU metrics"}, { "role": "tool", "tool_call_id": "call_1", "content": json.dumps(database_rows_1000), }, ] result = benchmark(crusher.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before def test_multiple_tool_outputs( self, benchmark, crusher, mock_tokenizer, items_100, log_entries_100, ): """Benchmark crushing multiple tool outputs in one pass. Target: < 5ms Tests realistic scenario with multiple tool calls. """ messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Search users and get logs"}, { "role": "assistant", "content": None, "tool_calls": [ { "id": "call_1", "type": "function", "function": {"name": "search", "arguments": "{}"}, }, { "id": "call_2", "type": "function", "function": {"name": "logs", "arguments": "{}"}, }, ], }, {"role": "tool", "tool_call_id": "call_1", "content": json.dumps(items_100)}, {"role": "tool", "tool_call_id": "call_2", "content": json.dumps(log_entries_100)}, ] result = benchmark(crusher.apply, messages, mock_tokenizer) assert result.tokens_after < result.tokens_before class TestCacheAlignerBenchmarks: """Benchmarks for CacheAligner prefix optimization. CacheAligner performs: - Date pattern detection and extraction - Whitespace normalization - Stable prefix hash computation Expected performance: - Date extraction: < 1ms (regex matching) - Hash computation: < 0.5ms (MD5) - Total: < 2ms for typical system prompts """ @pytest.fixture def aligner(self, cache_aligner_config): """Create CacheAligner instance.""" from headroom.transforms.cache_aligner import CacheAligner return CacheAligner(config=cache_aligner_config) def test_date_extraction( self, benchmark, aligner, mock_tokenizer, messages_with_system_date, ): """Benchmark date extraction from system prompt. Target: < 1ms Tests regex-based date pattern matching. """ result = benchmark(aligner.apply, messages_with_system_date, mock_tokenizer) # Verify date was extracted assert "cache_align" in str(result.transforms_applied) def test_hash_computation( self, benchmark, aligner, mock_tokenizer, system_prompt_long, ): """Benchmark stable prefix hash computation. Target: < 0.5ms Tests hash stability for cache hit prediction. """ messages = [ {"role": "system", "content": system_prompt_long}, {"role": "user", "content": "Hello"}, ] result = benchmark(aligner.apply, messages, mock_tokenizer) # Verify hash was computed assert result.cache_metrics is not None assert result.cache_metrics.stable_prefix_hash def test_whitespace_normalization( self, benchmark, aligner, mock_tokenizer, ): """Benchmark whitespace normalization. Target: < 0.5ms Tests string processing for consistent formatting. """ messy_content = """You are a helpful assistant. Current date: 2025-01-06 This has excessive whitespace. And multiple blank lines.""" messages = [ {"role": "system", "content": messy_content}, {"role": "user", "content": "Hi"}, ] result = benchmark(aligner.apply, messages, mock_tokenizer) assert result.messages[0]["content"] != messy_content # Was normalized def test_long_system_prompt( self, benchmark, aligner, mock_tokenizer, system_prompt_long, ): """Benchmark processing long system prompts. Target: < 2ms Tests performance with larger instruction sets. """ # Add date to trigger alignment content_with_date = system_prompt_long + "\n\nCurrent date: 2025-01-06" messages = [ {"role": "system", "content": content_with_date}, {"role": "user", "content": "Help me with code"}, ] result = benchmark(aligner.apply, messages, mock_tokenizer) assert result.cache_metrics is not None def test_multiple_system_messages( self, benchmark, aligner, mock_tokenizer, ): """Benchmark with multiple system messages. Target: < 3ms Tests edge case of multiple system prompts. """ messages = [ { "role": "system", "content": "You are a helpful assistant.\n\nCurrent date: 2025-01-06", }, {"role": "system", "content": "Additional context: Technical support mode."}, {"role": "user", "content": "Hello"}, ] benchmark(aligner.apply, messages, mock_tokenizer) # RollingWindow benchmarks were retired in PR-B1 along with the # RollingWindow transform itself. Live-zone-only compression # (PR-B2..B7) does not drop messages, so message-count-based # benchmarks no longer have a baseline to measure. Phase B's own # performance suite lives alongside the live-zone dispatcher. class TestTransformPipelineBenchmarks: """Benchmarks for full transform pipeline. Tests the complete flow: CacheAligner -> SmartCrusher Expected performance: - Simple conversation: < 5ms - Agentic with tools: < 30ms - Large RAG context: < 50ms """ @pytest.fixture def mock_provider(self, mock_token_counter): """Create mock provider for pipeline.""" from unittest.mock import Mock provider = Mock() provider.get_token_counter.return_value = mock_token_counter return provider @pytest.fixture def pipeline(self, smart_crusher_config, cache_aligner_config, mock_provider): """Create transform pipeline. PR-B1 retired RollingWindow; the live-zone-only architecture runs CacheAligner → SmartCrusher (followed by ContentRouter in production, omitted here to keep the fixture pure-stage). """ from headroom.transforms.cache_aligner import CacheAligner from headroom.transforms.pipeline import TransformPipeline from headroom.transforms.smart_crusher import SmartCrusher return TransformPipeline( transforms=[ CacheAligner(cache_aligner_config), SmartCrusher(smart_crusher_config), ], provider=mock_provider, ) def test_pipeline_simple( self, benchmark, pipeline, messages_with_system_date, ): """Benchmark pipeline on simple conversation. Target: < 5ms Tests minimal overhead scenario. """ benchmark( pipeline.apply, messages_with_system_date, "benchmark-model", model_limit=100000, ) def test_pipeline_agentic( self, benchmark, pipeline, conversation_50_turns, ): """Benchmark pipeline on agentic conversation. Target: < 30ms Tests realistic agentic workload. """ result = benchmark( pipeline.apply, conversation_50_turns, "benchmark-model", model_limit=50000, ) assert result.tokens_after < result.tokens_before def test_pipeline_rag( self, benchmark, pipeline, rag_conversation_20k, ): """Benchmark pipeline on RAG conversation. Target: < 50ms Tests large context handling. Note: CacheAligner may add small markers (e.g., "[Dynamic Context]"), so we allow up to 1% token increase. """ result = benchmark( pipeline.apply, rag_conversation_20k, "benchmark-model", model_limit=30000, ) # Allow for small overhead from cache alignment markers assert result.tokens_after <= result.tokens_before * 1.01 - benchmarks/cache_bust_trace_report.py (13908 bytes)
#!/usr/bin/env python3 """Trace and report concrete cache-busting turns from local Claude session replays.""" from __future__ import annotations import json import os import subprocess import sys import tempfile from dataclasses import asdict, dataclass from pathlib import Path from typing import Any if __package__ in {None, ""}: sys.path.insert(0, str(Path(__file__).resolve().parents[1])) DEFAULT_OUTPUT_DIR = Path("benchmark_results") / "cache_bust_trace" @dataclass class BustEvent: branch: str mode: str session_id: str project: str request_id: str timestamp: str first_diff_index: int | None prev_len: int curr_len: int prev_msg: dict[str, Any] | None curr_msg: dict[str, Any] | None prev_tail: list[dict[str, Any]] curr_tail: list[dict[str, Any]] retroactive_rewrite: bool def _run_git(args: list[str], cwd: Path) -> str: completed = subprocess.run( ["git", *args], cwd=cwd, check=True, capture_output=True, text=True, ) return completed.stdout.strip() def _ref_slug(ref: str) -> str: return "".join(ch if ch.isalnum() else "-" for ch in ref).strip("-").lower() or "ref" def _first_diff_index(prev: list[dict[str, Any]], curr: list[dict[str, Any]]) -> int | None: for i, (a, b) in enumerate(zip(prev, curr)): if a != b: return i if len(prev) != len(curr): return min(len(prev), len(curr)) return None def _trace_branch( repo_root: Path, ref: str, label: str, *, recent_turns_per_session: int, max_events_per_mode: int = 10, ) -> list[BustEvent]: worktree_root = Path(tempfile.mkdtemp(prefix="headroom-bust-trace-")) worktree_dir = worktree_root / _ref_slug(label) _run_git(["worktree", "add", "--detach", str(worktree_dir), ref], repo_root) try: env = os.environ.copy() env["PYTHONPATH"] = str(worktree_dir) code = """ import copy, json from datetime import timedelta from pathlib import Path import importlib.util import os import sys module_path = Path(os.environ['BUST_TRACE_SCRIPT']) spec = importlib.util.spec_from_file_location('branch_benchmark', module_path) mod = importlib.util.module_from_spec(spec) assert spec and spec.loader sys.modules[spec.name] = mod spec.loader.exec_module(mod) PROXY_MODE_CACHE = mod.PROXY_MODE_CACHE PROXY_MODE_TOKEN = mod.PROXY_MODE_TOKEN PrefixCacheTracker = mod.PrefixCacheTracker _apply_mode_to_messages = mod._apply_mode_to_messages _cache_gap_within_ttl = mod._cache_gap_within_ttl _rewrite_scope = mod._rewrite_scope get_tokenizer = mod.get_tokenizer load_session_replay = mod.load_session_replay select_session_files = mod.select_session_files trim_replay_to_recent_turns = mod.trim_replay_to_recent_turns _make_proxy = mod._make_proxy from headroom.cache.compression_cache import CompressionCache ROOT = Path.home() / '.claude' / 'projects' TTL = timedelta(minutes=5) recent_turns_per_session = int(__import__('os').environ['BUST_TRACE_RECENT']) max_events_per_mode = int(__import__('os').environ['BUST_TRACE_MAX']) def first_diff_index(prev, curr): for i, (a, b) in enumerate(zip(prev, curr)): if a != b: return i if len(prev) != len(curr): return min(len(prev), len(curr)) return None def trace_mode(mode): proxy = _make_proxy(mode) session_files = select_session_files(ROOT) events = [] for session_file in session_files: replay = load_session_replay(session_file) if replay is None: continue replay = trim_replay_to_recent_turns(replay, recent_turns_per_session) prefix_tracker = PrefixCacheTracker('anthropic') comp_cache = CompressionCache() if mode == PROXY_MODE_TOKEN else None conversation = [] conversation_token_total = 0 previous_forwarded = [] previous_original_context = None previous_forwarded_context = None previous_timestamp = None pending = None for turn in replay.turns: tokenizer = get_tokenizer(turn.model) turn_input_token_total = sum(tokenizer.count_message(msg) for msg in turn.input_messages) prior_context_message_count = len(conversation) conversation.extend(turn.input_messages) raw_input_tokens = conversation_token_total + turn_input_token_total forwarded = _apply_mode_to_messages( proxy, mode, conversation, model=turn.model, prefix_tracker=prefix_tracker, comp_cache=comp_cache, previous_original_messages=previous_original_context, previous_forwarded_messages=previous_forwarded_context, ) if pending is not None: eligible = _cache_gap_within_ttl(pending.turn.timestamp, previous_timestamp, ttl=TTL) if eligible and previous_forwarded: prefix_preserved = ( len(pending.forwarded) >= len(previous_forwarded) and pending.forwarded[: len(previous_forwarded)] == previous_forwarded ) if not prefix_preserved: idx = first_diff_index(previous_forwarded, pending.forwarded) _, retro = _rewrite_scope( pending.request_messages, pending.forwarded, stable_prefix_message_count=max(len(previous_forwarded) - 1, 0), ) events.append({ 'mode': mode, 'session_id': replay.session_id, 'project': replay.decoded_project_path, 'request_id': pending.turn.request_id, 'timestamp': pending.turn.timestamp.isoformat(), 'first_diff_index': idx, 'prev_len': len(previous_forwarded), 'curr_len': len(pending.forwarded), 'prev_msg': previous_forwarded[idx] if idx is not None and idx < len(previous_forwarded) else None, 'curr_msg': pending.forwarded[idx] if idx is not None and idx < len(pending.forwarded) else None, 'prev_tail': previous_forwarded_context[-4:] if previous_forwarded_context else [], 'curr_tail': pending.request_messages[-4:], 'retroactive_rewrite': retro, }) if len(events) >= max_events_per_mode: return events previous_forwarded = copy.deepcopy(pending.forwarded) previous_timestamp = pending.turn.timestamp try: prefix_tracker.update_from_response( cache_read_tokens=0, cache_write_tokens=0, messages=forwarded, message_token_counts=[tokenizer.count_message(msg) for msg in forwarded], original_messages=conversation, ) except TypeError: prefix_tracker.update_from_response( cache_read_tokens=0, cache_write_tokens=0, messages=forwarded, message_token_counts=[tokenizer.count_message(msg) for msg in forwarded], ) class Pending: pass pending = Pending() pending.turn = turn pending.request_messages = copy.deepcopy(conversation) pending.forwarded = forwarded conversation.append(turn.assistant_message) conversation_token_total = raw_input_tokens + tokenizer.count_message(turn.assistant_message) previous_original_context = copy.deepcopy(conversation) previous_forwarded_context = copy.deepcopy(forwarded) + [copy.deepcopy(turn.assistant_message)] return events print(json.dumps({ 'token': trace_mode(PROXY_MODE_TOKEN), 'cache': trace_mode(PROXY_MODE_CACHE), }, indent=2)) """ env["BUST_TRACE_RECENT"] = str(recent_turns_per_session) env["BUST_TRACE_MAX"] = str(max_events_per_mode) script_path = worktree_dir / "benchmarks" / "claude_session_mode_benchmark.py" if not script_path.exists(): script_path = repo_root / "benchmarks" / "claude_session_mode_benchmark.py" env["BUST_TRACE_SCRIPT"] = str(script_path) completed = subprocess.run( [sys.executable, "-c", code], cwd=worktree_dir, check=True, capture_output=True, text=True, env=env, ) payload = json.loads(completed.stdout) events: list[BustEvent] = [] for mode in ("token", "cache"): for item in payload[mode]: events.append(BustEvent(branch=label, **item)) return events except subprocess.CalledProcessError as exc: raise RuntimeError( f"trace failed for {label} ({ref})\nSTDOUT:\n{exc.stdout}\nSTDERR:\n{exc.stderr}" ) from exc finally: subprocess.run( ["git", "worktree", "remove", "--force", str(worktree_dir)], cwd=repo_root, check=True, ) def _render_markdown(events: list[BustEvent], recent_turns_per_session: int) -> str: lines = [ "# Cache Bust Trace Report", "", f"- Sampling: most recent {recent_turns_per_session} turns per session", "", ] for branch in ("main", "pr"): lines.append(f"## {branch}") lines.append("") branch_events = [e for e in events if e.branch == branch] for mode in ("token", "cache"): lines.append(f"### {mode}") mode_events = [e for e in branch_events if e.mode == mode] if not mode_events: lines.append("") lines.append("- No bust events captured.") lines.append("") continue for event in mode_events: lines.append("") lines.append( f"- `{event.project}` `{event.session_id}` `{event.request_id}` " f"{event.timestamp} diff_index={event.first_diff_index} " f"retroactive={event.retroactive_rewrite}" ) lines.append("") return "\n".join(lines) def _render_html(events: list[BustEvent], recent_turns_per_session: int) -> str: sections = [] for branch in ("main", "pr"): rows = [] branch_events = [e for e in events if e.branch == branch] for mode in ("token", "cache"): mode_events = [e for e in branch_events if e.mode == mode] if not mode_events: rows.append( f"<tr><td>{mode}</td><td colspan='6'>No bust events captured.</td></tr>" ) continue for event in mode_events: rows.append( "<tr>" f"<td>{mode}</td>" f"<td>{event.project}</td>" f"<td>{event.session_id}</td>" f"<td>{event.request_id}</td>" f"<td>{event.timestamp}</td>" f"<td>{event.first_diff_index}</td>" f"<td>{event.retroactive_rewrite}</td>" "</tr>" ) sections.append( f"<section><h2>{branch}</h2><table><thead><tr>" "<th>Mode</th><th>Project</th><th>Session</th><th>Request</th>" "<th>Timestamp</th><th>First Diff</th><th>Retroactive</th>" f"</tr></thead><tbody>{''.join(rows)}</tbody></table></section>" ) return f"""<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Cache Bust Trace Report</title> <style> body {{ font-family: 'Segoe UI', system-ui, sans-serif; margin: 0; background: #f8fafc; color: #0f172a; }} .shell {{ max-width: 1280px; margin: 0 auto; padding: 32px 16px 48px; }} h1, h2 {{ letter-spacing: -0.02em; }} section {{ background: white; border: 1px solid #e2e8f0; border-radius: 16px; padding: 20px; margin-top: 16px; }} table {{ width: 100%; border-collapse: collapse; }} th, td {{ padding: 10px 12px; border-bottom: 1px solid #e2e8f0; text-align: left; white-space: nowrap; }} th {{ background: #f1f5f9; }} </style> </head> <body> <div class="shell"> <h1>Cache Bust Trace Report</h1> <p>Most recent {recent_turns_per_session} turns per session.</p> {"".join(sections)} </div> </body> </html>""" def main() -> int: repo_root = Path(__file__).resolve().parents[1] output_dir = DEFAULT_OUTPUT_DIR output_dir.mkdir(parents=True, exist_ok=True) recent_turns_per_session = 200 events = _trace_branch( repo_root, "upstream/main", "main", recent_turns_per_session=recent_turns_per_session ) events.extend( _trace_branch(repo_root, "HEAD", "pr", recent_turns_per_session=recent_turns_per_session) ) md_path = output_dir / "cache_bust_trace.md" json_path = output_dir / "cache_bust_trace.json" html_path = output_dir / "cache_bust_trace.html" md_path.write_text(_render_markdown(events, recent_turns_per_session), encoding="utf-8") json_path.write_text( json.dumps([asdict(event) for event in events], indent=2), encoding="utf-8" ) html_path.write_text(_render_html(events, recent_turns_per_session), encoding="utf-8") print(md_path) print(json_path) print(html_path) return 0 if __name__ == "__main__": raise SystemExit(main()) - benchmarks/cache_validation_bundle.py (32088 bytes)
#!/usr/bin/env python3 """Generate a reproducible local cache-validation report bundle.""" from __future__ import annotations import argparse import copy import hashlib import html import json import logging import platform import subprocess import sys from dataclasses import asdict from datetime import timedelta from pathlib import Path from typing import Any if __package__ in {None, ""}: sys.path.insert(0, str(Path(__file__).resolve().parents[1])) import benchmarks.claude_session_mode_benchmark as real_bench import benchmarks.synthetic_long_cache_suite_report as long_suite import benchmarks.synthetic_token_cache_bust_report as token_bust from benchmarks.claude_session_mode_benchmark import ( PROXY_MODE_CACHE, PROXY_MODE_TOKEN, _apply_mode_to_messages, _cache_gap_within_ttl, _rewrite_scope, build_dataset_and_observed_from_files, determine_winners, format_currency, get_tokenizer, load_session_replay, resolve_checkpoint_dir, select_session_files, simulate_session_files, trim_replay_to_recent_turns, write_report, ) from headroom.cache.compression_cache import CompressionCache from headroom.cache.prefix_tracker import PrefixCacheTracker DEFAULT_OUTPUT_DIR = Path("benchmark_results") / "cache_validation_bundle" def _excerpt_content(content: Any, *, max_chars: int) -> str: if isinstance(content, str): text = content.replace("\n", " ") return text[:max_chars] + ("..." if len(text) > max_chars else "") if isinstance(content, list): parts = [] for block in content[:4]: if isinstance(block, dict): btype = str(block.get("type", "unknown")) bcontent = block.get("content", "") if isinstance(bcontent, str): bcontent = bcontent.replace("\n", " ") bcontent = bcontent[:max_chars] + ("..." if len(bcontent) > max_chars else "") parts.append(f"[{btype}] {bcontent}") else: parts.append(str(block)[:max_chars]) return " | ".join(parts) return str(content)[:max_chars] def _message_preview(msg: dict[str, Any], *, max_chars: int) -> dict[str, str]: return { "role": str(msg.get("role")), "content_excerpt": _excerpt_content(msg.get("content"), max_chars=max_chars), } def _stable_hash(value: str) -> str: return hashlib.sha256(value.encode("utf-8")).hexdigest()[:12] def _redact_text(value: str, *, prefix: str) -> str: return f"{prefix}-{_stable_hash(value)}" def _redact_path(value: str) -> str: path = Path(value) suffix = path.suffix return f"path-{_stable_hash(value)}{suffix}" def _git_output(args: list[str], cwd: Path) -> str | None: try: completed = subprocess.run( ["git", *args], cwd=cwd, check=True, capture_output=True, text=True, ) return completed.stdout.strip() except Exception: return None def _runtime_metadata(repo_root: Path) -> dict[str, Any]: return { "git_sha": _git_output(["rev-parse", "HEAD"], repo_root), "git_dirty": bool(_git_output(["status", "--porcelain"], repo_root)), "python_version": sys.version, "platform": platform.platform(), "implementation": platform.python_implementation(), } def _corpus_fingerprint( *, root: Path, session_files: list[Path], max_sessions: int | None, recent_turns_per_session: int | None, cache_ttl_minutes: int, ) -> dict[str, Any]: normalized_files = [str(p.resolve()) for p in session_files] payload = { "root": str(root.resolve()), "session_files": normalized_files, "max_sessions": max_sessions, "recent_turns_per_session": recent_turns_per_session, "cache_ttl_minutes": cache_ttl_minutes, } digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode("utf-8")).hexdigest() return { "root": str(root.resolve()), "session_file_count": len(session_files), "session_files_sha256": digest, "max_sessions": max_sessions, "recent_turns_per_session": recent_turns_per_session, "cache_ttl_minutes": cache_ttl_minutes, } def _collect_real_processed_events( *, root: Path, recent_turns_per_session: int | None, max_events_per_mode: int, ttl_minutes: int, max_chars: int, include_content: bool, ) -> dict[str, Any]: ttl = timedelta(minutes=ttl_minutes) events: list[dict[str, Any]] = [] session_files = select_session_files(root) for mode in (PROXY_MODE_TOKEN, PROXY_MODE_CACHE): proxy = real_bench._make_proxy(mode) collected = 0 for session_file in session_files: replay = load_session_replay(session_file) if replay is None: continue replay = trim_replay_to_recent_turns(replay, recent_turns_per_session) prefix_tracker = PrefixCacheTracker("anthropic") comp_cache = CompressionCache() if mode == PROXY_MODE_TOKEN else None conversation: list[dict[str, Any]] = [] previous_original_context: list[dict[str, Any]] | None = None previous_forwarded_context: list[dict[str, Any]] | None = None previous_forwarded: list[dict[str, Any]] = [] previous_timestamp = None pending = None for turn in replay.turns: tokenizer = get_tokenizer(turn.model) prior_context_message_count = len(conversation) conversation.extend(turn.input_messages) forwarded = _apply_mode_to_messages( proxy, mode, conversation, model=turn.model, prefix_tracker=prefix_tracker, comp_cache=comp_cache, previous_original_messages=previous_original_context, previous_forwarded_messages=previous_forwarded_context, ) rewrite, retro = _rewrite_scope( conversation, forwarded, stable_prefix_message_count=prior_context_message_count, ) if rewrite: prior_forwarded = ( pending.forwarded if pending is not None else previous_forwarded ) prior_ts = pending.turn.timestamp if pending is not None else previous_timestamp eligible = bool( prior_ts is not None and _cache_gap_within_ttl(turn.timestamp, prior_ts, ttl=ttl) and prior_forwarded ) prefix_preserved = None first_diff_index = None if eligible: prefix_preserved = ( len(forwarded) >= len(prior_forwarded) and forwarded[: len(prior_forwarded)] == prior_forwarded ) if not prefix_preserved: for idx, (a, b) in enumerate(zip(prior_forwarded, forwarded)): if a != b: first_diff_index = idx break if first_diff_index is None: first_diff_index = min(len(prior_forwarded), len(forwarded)) events.append( { "mode": mode, "session_id": replay.session_id if include_content else _redact_text(replay.session_id, prefix="session"), "project": replay.decoded_project_path if include_content else _redact_path(replay.decoded_project_path), "request_id": turn.request_id if include_content else _redact_text(turn.request_id, prefix="request"), "timestamp": turn.timestamp.isoformat(), "cache_eligible": eligible, "prefix_preserved": prefix_preserved, "retroactive_rewrite": retro, "first_diff_index": first_diff_index, "original_tail": [ _message_preview(m, max_chars=max_chars) if include_content else { "role": str(m.get("role")), "content_excerpt": "[redacted]", } for m in conversation[max(0, len(conversation) - 4) :] ], "forwarded_tail": [ _message_preview(m, max_chars=max_chars) if include_content else { "role": str(m.get("role")), "content_excerpt": "[redacted]", } for m in forwarded[max(0, len(forwarded) - 4) :] ], } ) collected += 1 if collected >= max_events_per_mode: break if pending is not None: previous_forwarded = copy.deepcopy(pending.forwarded) previous_timestamp = pending.turn.timestamp real_bench._update_prefix_tracker( prefix_tracker, cache_read_tokens=0, cache_write_tokens=0, messages=forwarded, message_token_counts=[tokenizer.count_message(msg) for msg in forwarded], original_messages=conversation, ) class Pending: pass pending = Pending() pending.turn = turn pending.forwarded = forwarded conversation.append(turn.assistant_message) previous_original_context = copy.deepcopy(conversation) previous_forwarded_context = copy.deepcopy(forwarded) + [ copy.deepcopy(turn.assistant_message) ] if collected >= max_events_per_mode: break return {"events": events} def _write_processed_event_reports( output_dir: Path, payload: dict[str, Any] ) -> tuple[Path, Path, Path]: out_dir = output_dir / "real_processed" out_dir.mkdir(parents=True, exist_ok=True) json_path = out_dir / "real_processed_rewrite_report.json" md_path = out_dir / "real_processed_rewrite_report.md" html_path = out_dir / "real_processed_rewrite_report.html" json_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") md = [ "# Real Processed Rewrite Report", "", "Local-only report from real Claude transcript replays. Do not commit.", "", ] for mode in (PROXY_MODE_TOKEN, PROXY_MODE_CACHE): mode_events = [e for e in payload["events"] if e["mode"] == mode] md.extend([f"## `{mode}`", ""]) if not mode_events: md.extend(["No rewrite events captured.", ""]) continue for i, e in enumerate(mode_events, start=1): md.extend( [ f"### Event {i}", "", f"- session: `{e['session_id']}`", f"- request: `{e['request_id']}`", f"- cache eligible: `{e['cache_eligible']}`", f"- prefix preserved: `{e['prefix_preserved']}`", f"- retroactive rewrite: `{e['retroactive_rewrite']}`", f"- first diff index: `{e['first_diff_index']}`", "", "**Original Tail**", "", ] ) for msg in e["original_tail"]: md.append(f"- `{msg['role']}`: {msg['content_excerpt']}") md.extend(["", "**Forwarded Tail**", ""]) for msg in e["forwarded_tail"]: md.append(f"- `{msg['role']}`: {msg['content_excerpt']}") md.extend(["", ""]) md_path.write_text("\n".join(md), encoding="utf-8") sections = [] for mode in (PROXY_MODE_TOKEN, PROXY_MODE_CACHE): mode_events = [e for e in payload["events"] if e["mode"] == mode] cards = [] for i, e in enumerate(mode_events, start=1): orig = "".join( f"<li><code>{html.escape(str(m['role']))}</code>: " f"{html.escape(str(m['content_excerpt']))}</li>" for m in e["original_tail"] ) fwd = "".join( f"<li><code>{html.escape(str(m['role']))}</code>: " f"{html.escape(str(m['content_excerpt']))}</li>" for m in e["forwarded_tail"] ) cards.append( "<div class='event'>" f"<h3>Event {i}</h3>" f"<p><strong>session</strong>: <code>{html.escape(e['session_id'])}</code><br>" f"<strong>request</strong>: <code>{html.escape(e['request_id'])}</code><br>" f"<strong>cache eligible</strong>: <code>{e['cache_eligible']}</code><br>" f"<strong>prefix preserved</strong>: <code>{e['prefix_preserved']}</code><br>" f"<strong>retroactive rewrite</strong>: <code>{e['retroactive_rewrite']}</code><br>" f"<strong>first diff index</strong>: <code>{e['first_diff_index']}</code></p>" f"<div class='cols'><div><h4>Original Tail</h4><ul>{orig}</ul></div>" f"<div><h4>Forwarded Tail</h4><ul>{fwd}</ul></div></div>" "</div>" ) sections.append( f"<section class='card'><h2>{html.escape(mode)}</h2>" + ("".join(cards) if cards else "<p>No rewrite events captured.</p>") + "</section>" ) html_doc = ( "<!doctype html><html><head><meta charset='utf-8'>" "<meta name='viewport' content='width=device-width, initial-scale=1'>" "<title>Real Processed Rewrite Report</title>" "<style>" "body{font-family:ui-sans-serif,system-ui,sans-serif;max-width:1200px;margin:40px auto;padding:0 20px;line-height:1.55;color:#111827;background:#f8fafc}" ".card,.event{background: - benchmarks/ccr_regression_benchmark.py (25894 bytes)
#!/usr/bin/env python3 """ CCR Regression Benchmark - Verify No Information Loss This benchmark tests that the CCR (Compress-Cache-Retrieve) architecture does not cause any regression in agent behavior. Specifically: 1. NEEDLE RETENTION: Critical items survive compression - Errors, exceptions, failures - Specific IDs/UUIDs mentioned in user query - Anomalies and outliers 2. RETRIEVAL ACCURACY: When retrieval is needed, correct items are returned - Retrieval is by hash and always returns the full original content 3. FEEDBACK LEARNING: System learns from retrieval patterns - High retrieval rate triggers less aggressive compression Usage: python benchmarks/ccr_regression_benchmark.py python benchmarks/ccr_regression_benchmark.py --verbose python benchmarks/ccr_regression_benchmark.py --scenario needle-in-haystack """ from __future__ import annotations import argparse import json import time import uuid from dataclasses import dataclass, field from typing import Any from headroom.cache.compression_feedback import ( get_compression_feedback, reset_compression_feedback, ) from headroom.cache.compression_store import ( get_compression_store, reset_compression_store, ) from headroom.transforms.smart_crusher import ( SmartCrusherConfig, smart_crush_tool_output, ) @dataclass class RegressionResult: """Result from a regression test.""" name: str description: str passed: bool = False # Default to False, set to True when test passes # Metrics total_needles: int = 0 needles_retained: int = 0 retention_rate: float = 0.0 # CCR metrics items_compressed: int = 0 items_retrieved: int = 0 retrieval_accuracy: float = 0.0 # Performance latency_ms: float = 0.0 # Details details: dict[str, Any] = field(default_factory=dict) failures: list[str] = field(default_factory=list) def _ccr_retrieve_items(store: Any, hash_key: str) -> list[dict[str, Any]]: """Full CCR retrieval (hash-only) → parsed original items. Retrieval is by hash and always returns the complete original content, so any "needle" present at compression time is guaranteed to survive the round-trip. Returns the parsed list, or [] on a miss / non-list payload. """ entry = store.retrieve(hash_key) if not entry: return [] try: data = json.loads(entry.original_content) except (json.JSONDecodeError, TypeError): return [] return data if isinstance(data, list) else [] # ============================================================================= # TEST 1: Needle in Haystack - Error Retention # ============================================================================= def test_error_retention() -> RegressionResult: """ Test that errors are NEVER lost during compression. This is critical: if an API returns 1000 results with 3 errors, those 3 errors MUST be in the compressed output. """ result = RegressionResult( name="Error Retention", description="Verify all errors survive compression regardless of position", ) # Generate 1000 items with errors at various positions items = [] error_indices = [5, 47, 123, 456, 789, 999] # Spread throughout for i in range(1000): if i in error_indices: items.append( { "id": i, "status": "error", "message": f"Connection failed: timeout at {i}", "error_code": 500 + (i % 10), } ) else: items.append( { "id": i, "status": "success", "message": "OK", "data": {"value": i * 2}, } ) result.total_needles = len(error_indices) # Compress with SmartCrusher config = SmartCrusherConfig(max_items_after_crush=15) original_json = json.dumps(items) start = time.perf_counter() compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) result.latency_ms = (time.perf_counter() - start) * 1000 # Count errors in compressed output compressed = json.loads(compressed_json) errors_found = [item for item in compressed if item.get("status") == "error"] result.needles_retained = len(errors_found) result.retention_rate = result.needles_retained / result.total_needles result.items_compressed = len(compressed) # Check if ALL errors were retained result.passed = result.needles_retained == result.total_needles if not result.passed: result.failures.append( f"Lost {result.total_needles - result.needles_retained} errors during compression" ) result.details = { "original_items": 1000, "compressed_items": len(compressed), "error_positions": error_indices, "errors_retained": result.needles_retained, } return result # ============================================================================= # TEST 2: Needle in Haystack - UUID Lookup # ============================================================================= def test_uuid_retrieval() -> RegressionResult: """ Test that specific UUIDs can be found via CCR retrieval. Scenario: User asks "find transaction abc123..." The system compresses, but user should be able to retrieve the specific item. """ result = RegressionResult( name="UUID Retrieval via CCR", description="Verify specific UUIDs can be retrieved from compressed cache", ) reset_compression_store() store = get_compression_store() # Generate 1000 transactions with UUIDs target_uuid = str(uuid.uuid4()) items = [] for i in range(1000): item_uuid = target_uuid if i == 456 else str(uuid.uuid4()) items.append( { "transaction_id": item_uuid, "amount": 100 + (i % 1000), "status": "completed", "timestamp": f"2025-01-{(i % 28) + 1:02d}T10:00:00Z", } ) result.total_needles = 1 # Store original and compress original_json = json.dumps(items) config = SmartCrusherConfig(max_items_after_crush=15) start = time.perf_counter() compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) # Store in CCR cache hash_key = store.store( original=original_json, compressed=compressed_json, original_item_count=1000, compressed_item_count=15, tool_name="transaction_search", ) # Search for the specific UUID search_results = _ccr_retrieve_items(store, hash_key) result.latency_ms = (time.perf_counter() - start) * 1000 # Check if target UUID was found found_target = any(item.get("transaction_id") == target_uuid for item in search_results) result.needles_retained = 1 if found_target else 0 result.retention_rate = result.needles_retained / result.total_needles result.items_retrieved = len(search_results) result.retrieval_accuracy = 1.0 if found_target else 0.0 result.passed = found_target if not result.passed: result.failures.append( f"Could not retrieve target UUID {target_uuid[:8]}... via CCR search" ) result.details = { "target_uuid": target_uuid, "search_results_count": len(search_results), "found_target": found_target, "hash_key": hash_key, } return result # ============================================================================= # TEST 3: Anomaly Detection # ============================================================================= def test_anomaly_retention() -> RegressionResult: """ Test that statistical anomalies are preserved during compression. Scenario: 1000 metrics mostly at ~50, but with 5 spikes at 500+. Those spikes MUST survive compression. """ result = RegressionResult( name="Anomaly Retention", description="Verify statistical outliers survive compression" ) # Generate metrics with anomalies import random random.seed(42) # Reproducible items = [] anomaly_indices = [10, 200, 450, 700, 990] # 5 spikes for i in range(1000): if i in anomaly_indices: # Anomaly: 10x normal value value = 500 + random.randint(0, 100) else: # Normal: around 50 value = 50 + random.randint(-10, 10) items.append( { "timestamp": f"2025-01-07T{(i // 60):02d}:{(i % 60):02d}:00Z", "cpu_percent": value, "host": "prod-server-1", } ) result.total_needles = len(anomaly_indices) # Compress config = SmartCrusherConfig( max_items_after_crush=20, preserve_change_points=True, ) original_json = json.dumps(items) start = time.perf_counter() compressed_json, was_modified, _ = smart_crush_tool_output(original_json, config) result.latency_ms = (time.perf_counter() - start) * 1000 # Count anomalies (cpu > 200) in compressed output compressed = json.loads(compressed_json) anomalies_found = [ item for item in compressed if isinstance(item.get("cpu_percent"), (int, float)) and item["cpu_percent"] > 200 ] result.needles_retained = len(anomalies_found) result.retention_rate = result.needles_retained / result.total_needles result.items_compressed = len(compressed) # Pass if at least 80% of anomalies retained (some might be in change point windows) result.passed = result.retention_rate >= 0.8 if not result.passed: result.failures.append( f"Lost too many anomalies: {result.needles_retained}/{result.total_needles} retained" ) result.details = { "original_items": 1000, "compressed_items": len(compressed), "anomaly_positions": anomaly_indices, "anomalies_retained": result.needles_retained, } return result # ============================================================================= # TEST 4: Full Retrieval Accuracy # ============================================================================= def test_full_retrieval() -> RegressionResult: """ Test that full retrieval returns EXACTLY the original content. """ result = RegressionResult( name="Full Retrieval Accuracy", description="Verify full retrieval returns exact original content", ) reset_compression_store() store = get_compression_store() # Generate test data items = [{"id": i, "name": f"item_{i}", "value": i * 10} for i in range(100)] original_json = json.dumps(items) compressed_json = json.dumps(items[:10]) # Simulate compression # Store hash_key = store.store( original=original_json, compressed=compressed_json, original_item_count=100, compressed_item_count=10, tool_name="test_tool", ) start = time.perf_counter() # Retrieve entry = store.retrieve(hash_key) result.latency_ms = (time.perf_counter() - start) * 1000 # Verify content matches exactly if entry is None: result.passed = False result.failures.append("Retrieval returned None") else: retrieved_items = json.loads(entry.original_content) result.passed = retrieved_items == items result.items_retrieved = len(retrieved_items) result.retrieval_accuracy = 1.0 if result.passed else 0.0 if not result.passed: result.failures.append("Retrieved content does not match original") result.total_needles = 100 result.needles_retained = result.items_retrieved result.retention_rate = 1.0 if result.passed else 0.0 result.details = { "original_items": 100, "retrieved_items": result.items_retrieved, "hash_key": hash_key, } return result # ============================================================================= # TEST 5: Feedback Learning # ============================================================================= def test_feedback_learning() -> RegressionResult: """ Test that the feedback system learns from retrieval patterns. Scenario: Simulate high retrieval rate, verify system recommends less aggressive compression. """ result = RegressionResult( name="Feedback Learning", description="Verify feedback loop adjusts compression based on patterns", ) reset_compression_feedback() feedback = get_compression_feedback() tool_name = "high_retrieval_tool" start = time.perf_counter() # Simulate 10 compressions for _ in range(10): feedback.record_compression(tool_name, 1000, 20) # Simulate 6 retrievals (60% rate - HIGH) from headroom.cache.compression_store import RetrievalEvent for i in range(6): event = RetrievalEvent( hash=f"hash{i:012d}", query="find errors", items_retrieved=100, total_items=1000, tool_name=tool_name, timestamp=time.time(), retrieval_type="search", ) feedback.record_retrieval(event) # Get hints hints = feedback.get_compression_hints(tool_name) result.latency_ms = (time.perf_counter() - start) * 1000 # Verify hints recommend less aggressive compression pattern = feedback.get_all_patterns().get(tool_name) checks_passed = 0 total_checks = 3 # Check 1: Retrieval rate is tracked correctly if pattern and abs(pattern.retrieval_rate - 0.6) < 0.01: checks_passed += 1 else: result.failures.append( f"Retrieval rate incorrect: {pattern.retrieval_rate if pattern else 'N/A'}" ) # Check 2: Hints suggest more items (>15 default) if hints.max_items > 15: checks_passed += 1 else: result.failures.append(f"max_items not increased: {hints.max_items}") # Check 3: Aggressiveness reduced (<0.7 default) if hints.aggressiveness < 0.7: checks_passed += 1 else: result.failures.append(f"Aggressiveness not reduced: {hints.aggressiveness}") result.passed = checks_passed == total_checks result.retrieval_accuracy = checks_passed / total_checks result.details = { "compressions_recorded": 10, "retrievals_recorded": 6, "calculated_retrieval_rate": pattern.retrieval_rate if pattern else 0, "recommended_max_items": hints.max_items, "recommended_aggressiveness": hints.aggressiveness, "reason": hints.reason, } return result # ============================================================================= # TEST 6: Search Within Cached Content # ============================== - benchmarks/ccr_retrieval_history_benchmark.py (2937 bytes)
#!/usr/bin/env python3 """Measure CCR retrieval-history maintenance once the event cap is reached. Past 1,000 events the history has to discard the oldest event on every single retrieval. Bounding the container itself makes that O(1); the previous append-then-reslice rebuilt a 1,000-pointer list each time. This times ``_log_retrieval`` directly — the store's public ``retrieve()`` also does backend I/O and a feedback drain, which would bury the difference. Read the result as "cost of one history append at steady state", not as a proxy for end-to-end retrieval latency. Usage: python benchmarks/ccr_retrieval_history_benchmark.py python benchmarks/ccr_retrieval_history_benchmark.py --events 5000 --runs 7 """ from __future__ import annotations import argparse import statistics import sys import time from pathlib import Path # Run as a script, sys.path[0] is benchmarks/, so an editable install of another # checkout would win for `import headroom` and this would silently measure the # wrong tree. sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) from headroom.cache.compression_store import CompressionStore # noqa: E402 def _fill_to_cap(store: CompressionStore) -> None: """Bring the history to its cap so every later append must evict.""" for i in range(store._max_events): store._log_retrieval( hash_key="h", query=f"warm{i}", items_retrieved=1, total_items=1, tool_name="bench", retrieval_type="full", ) store._pending_feedback_events.clear() def measure(events: int) -> float: """Return microseconds per history append at steady state.""" store = CompressionStore(enable_feedback=True) _fill_to_cap(store) start = time.perf_counter() for i in range(events): store._log_retrieval( hash_key="h", query=f"q{i}", items_retrieved=1, total_items=1, tool_name="bench", retrieval_type="full", ) elapsed = time.perf_counter() - start # Each run uses a fresh store, so the unbounded feedback queue this also # filled goes away with it. return elapsed / events * 1e6 def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--events", type=int, default=20_000) parser.add_argument("--runs", type=int, default=7) args = parser.parse_args() store = CompressionStore() print(f"container: {type(store._retrieval_events).__name__}") print(f"cap: {store._max_events} events") samples = [measure(args.events) for _ in range(args.runs)] print(f"appends per run: {args.events}, runs: {args.runs}") print(f"median: {statistics.median(samples):.3f} us/append") print(f"min: {min(samples):.3f} us, max: {max(samples):.3f} us") return 0 if __name__ == "__main__": raise SystemExit(main()) - benchmarks/ccr_sqlite_expiry_benchmark.py (2810 bytes)
#!/usr/bin/env python3 """Measure SQLite CCR insertion cleanup without expiring live entries.""" from __future__ import annotations import argparse import json import statistics import sys import tempfile import time from pathlib import Path # Run as a script sys.path[0] is benchmarks/, so an editable install of another # checkout would win and we would silently measure the wrong tree. sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) from headroom.cache.backends.sqlite import SQLiteBackend from headroom.cache.compression_store import CompressionEntry, CompressionStore def benchmark(entry_count: int, payload_bytes: int, runs: int) -> dict[str, object]: with tempfile.TemporaryDirectory(prefix="headroom-ccr-expiry-") as tmp: backend = SQLiteBackend(Path(tmp) / "ccr.db") payload = "x" * payload_bytes for index in range(entry_count): key = str(index) backend.set( key, CompressionEntry( hash=key, original_content=payload, compressed_content="compact", original_tokens=payload_bytes // 4, compressed_tokens=2, original_item_count=0, compressed_item_count=0, tool_name=None, tool_call_id=None, query_context=None, created_at=time.time(), ttl=3600, ), ) store = CompressionStore( max_entries=entry_count + runs + 1, backend=backend, enable_feedback=False, ) samples_ms: list[float] = [] for index in range(runs): start = time.perf_counter() key = store.store(f"new original {index}", f"new compact {index}") samples_ms.append((time.perf_counter() - start) * 1000) assert store.retrieve(key) is not None backend._conn.close() return { "preexisting_entries": entry_count, "original_bytes_each": payload_bytes, "runs": runs, "median_insert_ms": statistics.median(samples_ms), } def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--entries", type=int, nargs="+", default=[100, 500, 1000]) parser.add_argument("--payload-bytes", type=int, nargs="+", default=[1024, 16384]) parser.add_argument("--runs", type=int, default=7) args = parser.parse_args() results = [ benchmark(entry_count, payload_bytes, args.runs) for payload_bytes in args.payload_bytes for entry_count in args.entries ] print(json.dumps(results, indent=2)) if __name__ == "__main__": main() - benchmarks/claude_session_branch_compare.py (22840 bytes)
#!/usr/bin/env python3 """Compare Claude session mode simulations across two git refs.""" from __future__ import annotations import argparse import json import os import shutil import subprocess import sys import tempfile from dataclasses import asdict, dataclass from pathlib import Path from typing import Any if __package__ in {None, ""}: sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from benchmarks.claude_session_mode_benchmark import ( IMPACT_DIRECTION, OUTPUT_JSON, PROXY_MODE_CACHE, PROXY_MODE_TOKEN, format_currency, ) DEFAULT_OUTPUT_DIR = Path("benchmark_results") / "branch_compare" @dataclass class BranchResult: ref: str label: str commit: str summary: str dataset: dict[str, Any] observed: dict[str, Any] summaries: dict[str, dict[str, Any]] winners: dict[str, str] output_dir: str def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--left-ref", default="upstream/main") parser.add_argument("--right-ref", default="HEAD") parser.add_argument("--left-label", default="main") parser.add_argument("--right-label", default="pr") parser.add_argument("--root", type=Path, default=Path.home() / ".claude" / "projects") parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT_DIR) parser.add_argument("--max-sessions", type=int, default=None) parser.add_argument("--recent-turns-per-session", type=int, default=None) parser.add_argument("--cache-ttl-minutes", type=int, default=5) parser.add_argument("--cache-write-multiplier", type=float, default=1.25) parser.add_argument("--workers", type=int, default=1) parser.add_argument( "--python", default=sys.executable, help="Python executable to use inside each worktree.", ) parser.add_argument( "--keep-worktrees", action="store_true", help="Do not remove temporary worktrees after the comparison run.", ) return parser.parse_args() def _run_git(args: list[str], cwd: Path) -> str: completed = subprocess.run( ["git", *args], cwd=cwd, check=True, capture_output=True, text=True, ) return completed.stdout.strip() def _ref_slug(ref: str) -> str: return "".join(ch if ch.isalnum() else "-" for ch in ref).strip("-").lower() or "ref" def _branch_output_dir(base: Path, label: str) -> Path: return base / _ref_slug(label) def _comparison_paths(base: Path) -> tuple[Path, Path, Path]: return ( base / "claude_session_branch_compare.md", base / "claude_session_branch_compare.json", base / "claude_session_branch_compare.html", ) def _mode_metric(branch: BranchResult, mode: str, field: str) -> float: summary = branch.summaries[mode] if field == "no_cache_total_cost_usd": if "no_cache_total_cost_usd" in summary: value = summary["no_cache_total_cost_usd"] else: value = ( float(summary["paid_input_cost_usd"]) + (float(summary["cache_read_cost_usd"]) * 10.0) + float(summary["paid_output_cost_usd"]) ) elif field == "prompt_window_with_cache": value = float(summary["forwarded_input_tokens"]) elif field == "prompt_window_without_cache_reads": value = float(summary["forwarded_input_tokens"]) - float(summary["cache_read_tokens"]) else: value = summary[field] if isinstance(value, bool): return float(value) return float(value) def _delta(left: float, right: float) -> float: return right - left def _classify_delta(field: str, delta: float) -> str: direction = IMPACT_DIRECTION.get(field, "same") tolerance = 1e-9 if abs(delta) <= tolerance: return "no_change" if direction == "lower": return "assist" if delta < 0 else "harm" if direction == "higher": return "assist" if delta > 0 else "harm" return "harm" def _build_benchmark_command( python_executable: str, script_path: Path, root: Path, output_dir: Path, max_sessions: int | None, recent_turns_per_session: int | None, cache_ttl_minutes: int, cache_write_multiplier: float, workers: int, ) -> list[str]: command = [ python_executable, str(script_path), "--root", str(root), "--output-dir", str(output_dir), "--cache-ttl-minutes", str(cache_ttl_minutes), "--cache-write-multiplier", str(cache_write_multiplier), "--workers", str(workers), ] if max_sessions is not None: command.extend(["--max-sessions", str(max_sessions)]) if recent_turns_per_session is not None: command.extend(["--recent-turns-per-session", str(recent_turns_per_session)]) return command def _load_branch_result( repo_root: Path, ref: str, label: str, branch_output_dir: Path, ) -> BranchResult: payload = json.loads((branch_output_dir / OUTPUT_JSON).read_text(encoding="utf-8")) commit = _run_git(["rev-parse", ref], repo_root) summary = _run_git(["show", "-s", "--format=%s", ref], repo_root) return BranchResult( ref=ref, label=label, commit=commit, summary=summary, dataset=payload["dataset"], observed=payload["observed"], summaries=payload["summaries"], winners=payload["winners"], output_dir=str(branch_output_dir), ) def _run_branch_benchmark( repo_root: Path, ref: str, label: str, args: argparse.Namespace, worktree_root: Path, ) -> BranchResult: worktree_dir = worktree_root / _ref_slug(label) branch_output_dir = _branch_output_dir(args.output_dir, label) branch_output_dir.mkdir(parents=True, exist_ok=True) if worktree_dir.exists(): shutil.rmtree(worktree_dir) _run_git(["worktree", "add", "--detach", str(worktree_dir), ref], repo_root) try: command = _build_benchmark_command( python_executable=args.python, script_path=worktree_dir / "benchmarks" / "claude_session_mode_benchmark.py", root=args.root, output_dir=branch_output_dir, max_sessions=args.max_sessions, recent_turns_per_session=args.recent_turns_per_session, cache_ttl_minutes=args.cache_ttl_minutes, cache_write_multiplier=args.cache_write_multiplier, workers=args.workers, ) env = os.environ.copy() env["PYTHONPATH"] = os.pathsep.join([str(worktree_dir), env.get("PYTHONPATH", "")]).rstrip( os.pathsep ) subprocess.run(command, cwd=worktree_dir, check=True, env=env) return _load_branch_result(repo_root, ref, label, branch_output_dir) finally: if not args.keep_worktrees: subprocess.run( ["git", "worktree", "remove", "--force", str(worktree_dir)], cwd=repo_root, check=True, ) def _winner_line(metric: str, left: BranchResult, right: BranchResult) -> str: left_winner = left.winners[metric] right_winner = right.winners[metric] if left_winner == right_winner: return f"- {metric}: both pick `{left_winner}`" return ( f"- {metric}: `{left.label}` picks `{left_winner}`, `{right.label}` picks `{right_winner}`" ) def _build_six_way_rows( left: BranchResult, right: BranchResult ) -> list[dict[str, str | float | int]]: rows: list[dict[str, str | float | int]] = [] for branch in (left, right): for mode in ("baseline", PROXY_MODE_TOKEN, PROXY_MODE_CACHE): summary = branch.summaries[mode] cost_delta = _mode_metric(branch, mode, "total_cost_usd") - _mode_metric( branch, "baseline", "total_cost_usd" ) window_delta = int( _mode_metric(branch, mode, "prompt_window_with_cache") - _mode_metric(branch, "baseline", "prompt_window_with_cache") ) read_delta = int( _mode_metric(branch, mode, "cache_read_tokens") - _mode_metric(branch, "baseline", "cache_read_tokens") ) write_delta = int( _mode_metric(branch, mode, "cache_write_tokens") - _mode_metric(branch, "baseline", "cache_write_tokens") ) paid_input_delta = int( _mode_metric(branch, mode, "regular_input_tokens") - _mode_metric(branch, "baseline", "regular_input_tokens") ) rows.append( { "branch": branch.label, "mode": mode, "forwarded_input_tokens": int(summary["forwarded_input_tokens"]), "cache_read_tokens": int(summary["cache_read_tokens"]), "cache_write_tokens": int(summary["cache_write_tokens"]), "regular_input_tokens": int(summary["regular_input_tokens"]), "output_tokens": int(summary["output_tokens"]), "total_cost_usd": float(summary["total_cost_usd"]), "cost_delta_vs_branch_baseline": cost_delta, "window_delta_vs_branch_baseline": window_delta, "cache_read_delta_vs_branch_baseline": read_delta, "cache_write_delta_vs_branch_baseline": write_delta, "paid_input_delta_vs_branch_baseline": paid_input_delta, "is_branch_winner": "yes" if branch.winners["total_cost"] == mode else "no", } ) return rows def build_compare_markdown(left: BranchResult, right: BranchResult) -> str: six_way_rows = _build_six_way_rows(left, right) lines = [ "# Claude Session Branch Comparison", "", "## Branches", "", f"- {left.label}: `{left.ref}` @ `{left.commit[:12]}` - {left.summary}", f"- {right.label}: `{right.ref}` @ `{right.commit[:12]}` - {right.summary}", "", "## Dataset", "", f"- Projects: {right.dataset['projects']}", f"- Sessions: {right.dataset['sessions']}", f"- Requests: {right.dataset['requests']}", f"- Sampled requests: {right.dataset.get('sampled_requests', 0)}", f"- Sampling: {right.dataset.get('sampling_note', 'Full sessions')}", "", "## Winner Comparison", "", _winner_line("total_cost", left, right), _winner_line("no_cache_total_cost", left, right), _winner_line("window_with_cache", left, right), _winner_line("window_without_cache_reads", left, right), "", "## Six-Way Mode Matrix", "", "| Branch | Mode | Forwarded Input | Cache Read | Cache Write | Paid Input | Paid Output | Total Cost | Cost Δ vs Branch Baseline | Window Δ vs Branch Baseline | Winner |", "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |", *[ "| " + " | ".join( [ str(row["branch"]), str(row["mode"]), f"{int(row['forwarded_input_tokens']):,}", f"{int(row['cache_read_tokens']):,}", f"{int(row['cache_write_tokens']):,}", f"{int(row['regular_input_tokens']):,}", f"{int(row['output_tokens']):,}", format_currency(float(row["total_cost_usd"])), format_currency(float(row["cost_delta_vs_branch_baseline"])), f"{int(row['window_delta_vs_branch_baseline']):,}", str(row["is_branch_winner"]), ] ) + " |" for row in six_way_rows ], "", "## Mode Deltas", "", f"| Mode | Metric | {left.label} | {right.label} | Delta ({right.label} - {left.label}) | Classification |", "| --- | --- | ---: | ---: | ---: | --- |", ] metrics = [ ("total_cost_usd", "Total Cost", format_currency), ("no_cache_total_cost_usd", "No-Cache Total Cost", format_currency), ("forwarded_input_tokens", "Forwarded Input Tokens", lambda v: f"{int(v):,}"), ("cache_read_tokens", "Cache Read Tokens", lambda v: f"{int(v):,}"), ("cache_write_tokens", "Cache Write Tokens", lambda v: f"{int(v):,}"), ("cache_bust_turns", "Cache Bust Turns", lambda v: f"{int(v):,}"), ("ttl_expiry_turns", "TTL Expiry Turns", lambda v: f"{int(v):,}"), ("prompt_window_with_cache", "Window With Cache", lambda v: f"{int(v):,}"), ( "prompt_window_without_cache_reads", "Window Without Cache Reads", lambda v: f"{int(v):,}", ), ] for mode in ("baseline", PROXY_MODE_TOKEN, PROXY_MODE_CACHE): for field, label, formatter in metrics: left_value = _mode_metric(left, mode, field) right_value = _mode_metric(right, mode, field) delta = _delta(left_value, right_value) delta_text = format_currency(delta) if "cost" in field else f"{int(delta):,}" classification = _classify_delta(field, delta) lines.append( f"| {mode} | {label} | {formatter(left_value)} | {formatter(right_value)} | {delta_text} | {classification} |" ) return "\n".join(lines) def build_compare_html(left: BranchResult, right: BranchResult) -> str: six_way_rows = [] for row in _build_six_way_rows(left, right): six_way_rows.append( "<tr>" f"<td>{row['branch']}</td>" f"<td><span class='pill'>{row['mode']}</span></td>" f"<td>{int(row['forwarded_input_tokens']):,}</td>" f"<td>{int(row['cache_read_tokens']):,}</td>" f"<td>{int(row['cache_write_tokens']):,}</td>" f"<td>{int(row['regular_input_tokens']):,}</td>" f"<td>{int(row['output_tokens']):,}</td>" f"<td>{format_currency(float(row['total_cost_usd']))}</td>" f"<td>{format_currency(float(row['cost_delta_vs_branch_baseline']))}</td>" f"<td>{int(row['window_delta_vs_branch_baseline']):,}</td>" f"<td>{row['is_branch_winner']}</td>" "</tr>" ) cards = [] for branch in (left, right): cards.append( "<div class='card'>" f"<div class='eyebrow'>{branch.label}</div>" f"<h2>{branch.ref}</h2>" f"<p><code>{branch.commit[:12]}</code></p>" f"<p>{branch.summary}</p>" "<div class='winner-grid'>" f"<div><span>Total Cost</span><strong>{branch.winners['total_cost']}</strong></div>" f"<div><span>No Cache</span><strong>{branch. - benchmarks/claude_session_mode_benchmark.py (82353 bytes)
#!/usr/bin/env python3 """Replay real Claude Code sessions through baseline/token/cache simulations.""" from __future__ import annotations import argparse import concurrent.futures import copy import json import logging import os from collections import Counter from dataclasses import asdict, dataclass, field from datetime import datetime, timedelta, timezone from pathlib import Path from typing import Any from headroom.cache.compression_cache import CompressionCache from headroom.cache.prefix_tracker import PrefixCacheTracker from headroom.pricing.litellm_pricing import get_model_pricing from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin from headroom.proxy.models import ProxyConfig from headroom.proxy.server import HeadroomProxy from headroom.tokenizers import get_tokenizer from headroom.utils import extract_user_query try: from headroom.proxy.modes import PROXY_MODE_CACHE, PROXY_MODE_TOKEN except ImportError: PROXY_MODE_CACHE = "cache" PROXY_MODE_TOKEN = "token" DEFAULT_ROOT = Path.home() / ".claude" / "projects" DEFAULT_OUTPUT_DIR = Path("benchmark_results") DEFAULT_CACHE_TTL_MINUTES = 5 OUTPUT_MD = "claude_session_mode_simulation.md" OUTPUT_JSON = "claude_session_mode_simulation.json" OUTPUT_HTML = "claude_session_mode_simulation.html" CHECKPOINT_DIRNAME = "checkpoints" @dataclass class ReplayTurn: session_id: str project_key: str decoded_project_path: str request_id: str model: str timestamp: datetime input_messages: list[dict[str, Any]] assistant_message: dict[str, Any] output_tokens: int observed_input_tokens: int = 0 observed_cache_read_tokens: int = 0 observed_cache_write_tokens: int = 0 @dataclass class SessionReplay: session_id: str project_key: str decoded_project_path: str turns: list[ReplayTurn] = field(default_factory=list) @dataclass class TurnMetrics: session_id: str request_id: str model: str timestamp: str raw_input_tokens: int forwarded_input_tokens: int cache_read_tokens: int cache_write_tokens: int regular_input_tokens: int output_tokens: int paid_input_cost_usd: float cache_read_cost_usd: float cache_write_cost_usd: float paid_output_cost_usd: float total_cost_usd: float @dataclass class ModeSummary: mode: str sessions: int = 0 requests: int = 0 raw_input_tokens: int = 0 forwarded_input_tokens: int = 0 cache_read_tokens: int = 0 cache_write_tokens: int = 0 regular_input_tokens: int = 0 output_tokens: int = 0 paid_input_cost_usd: float = 0.0 cache_read_cost_usd: float = 0.0 cache_write_cost_usd: float = 0.0 paid_output_cost_usd: float = 0.0 total_cost_usd: float = 0.0 cache_eligible_turns: int = 0 cache_bust_turns: int = 0 ttl_expiry_turns: int = 0 rewrite_turns: int = 0 stable_replay_rewrite_turns: int = 0 busting_rewrite_turns: int = 0 non_cache_eligible_rewrite_turns: int = 0 retroactive_rewrite_turns: int = 0 latest_turn_only_rewrite_turns: int = 0 turns: list[TurnMetrics] = field(default_factory=list) @property def raw_tokens(self) -> int: return self.raw_input_tokens + self.output_tokens @property def cache_tokens(self) -> int: return self.cache_read_tokens + self.cache_write_tokens @property def prompt_window_with_cache(self) -> int: return self.forwarded_input_tokens @property def prompt_window_without_cache_reads(self) -> int: return self.forwarded_input_tokens - self.cache_read_tokens @property def no_cache_total_cost_usd(self) -> float: return ( self.paid_input_cost_usd + (self.cache_read_cost_usd * 10.0) + self.paid_output_cost_usd ) @property def no_cache_paid_input_tokens(self) -> int: return self.forwarded_input_tokens @dataclass class DatasetSummary: projects: int sessions: int requests: int models: dict[str, int] decoded_project_paths: int sampled_requests: int = 0 sampling_note: str = "" IMPACT_DIRECTION = { "forwarded_input_tokens": "lower", "cache_read_tokens": "higher", "cache_write_tokens": "lower", "regular_input_tokens": "lower", "output_tokens": "same", "total_cost_usd": "lower", "no_cache_total_cost_usd": "lower", "prompt_window_with_cache": "lower", "prompt_window_without_cache_reads": "lower", "cache_bust_turns": "lower", "ttl_expiry_turns": "lower", "rewrite_turns": "lower", "stable_replay_rewrite_turns": "lower", "busting_rewrite_turns": "lower", "non_cache_eligible_rewrite_turns": "lower", "retroactive_rewrite_turns": "lower", "latest_turn_only_rewrite_turns": "lower", } @dataclass class ObservedSummary: sessions: int = 0 requests: int = 0 input_tokens: int = 0 cache_read_tokens: int = 0 cache_write_tokens: int = 0 output_tokens: int = 0 total_cost_usd: float = 0.0 cache_read_cost_usd: float = 0.0 cache_write_cost_usd: float = 0.0 paid_input_cost_usd: float = 0.0 paid_output_cost_usd: float = 0.0 healthy_growth_turns: int = 0 broken_prefix_turns: int = 0 resume_like_resets: int = 0 @property def raw_tokens(self) -> int: return ( self.input_tokens + self.cache_read_tokens + self.cache_write_tokens + self.output_tokens ) @property def cache_ratio_pct(self) -> float: total = self.input_tokens + self.cache_read_tokens + self.cache_write_tokens if total <= 0: return 0.0 return self.cache_read_tokens / total * 100.0 def _update_dataset_with_replay( dataset: DatasetSummary | None, replay: SessionReplay ) -> DatasetSummary: if dataset is None: dataset = DatasetSummary( projects=0, sessions=0, requests=0, models={}, decoded_project_paths=0, ) projects = {replay.project_key} project_paths = {replay.decoded_project_path} model_counts = Counter(dataset.models) requests = dataset.requests for turn in replay.turns: model_counts[turn.model] += 1 requests += 1 return DatasetSummary( projects=dataset.projects + len(projects), sessions=dataset.sessions + 1, requests=requests, models=dict(sorted(model_counts.items())), decoded_project_paths=dataset.decoded_project_paths + len(project_paths), ) def _turn_metrics_from_dict(data: dict[str, Any]) -> TurnMetrics: return TurnMetrics(**data) def _mode_summary_from_dict(data: dict[str, Any]) -> ModeSummary: turns = [_turn_metrics_from_dict(turn) for turn in data.get("turns", [])] summary = ModeSummary( mode=data["mode"], sessions=data.get("sessions", 0), requests=data.get("requests", 0), raw_input_tokens=data.get("raw_input_tokens", 0), forwarded_input_tokens=data.get("forwarded_input_tokens", 0), cache_read_tokens=data.get("cache_read_tokens", 0), cache_write_tokens=data.get("cache_write_tokens", 0), regular_input_tokens=data.get("regular_input_tokens", 0), output_tokens=data.get("output_tokens", 0), paid_input_cost_usd=data.get("paid_input_cost_usd", 0.0), cache_read_cost_usd=data.get("cache_read_cost_usd", 0.0), cache_write_cost_usd=data.get("cache_write_cost_usd", 0.0), paid_output_cost_usd=data.get("paid_output_cost_usd", 0.0), total_cost_usd=data.get("total_cost_usd", 0.0), cache_eligible_turns=data.get("cache_eligible_turns", 0), cache_bust_turns=data.get("cache_bust_turns", 0), ttl_expiry_turns=data.get("ttl_expiry_turns", 0), rewrite_turns=data.get("rewrite_turns", 0), stable_replay_rewrite_turns=data.get("stable_replay_rewrite_turns", 0), busting_rewrite_turns=data.get("busting_rewrite_turns", 0), non_cache_eligible_rewrite_turns=data.get("non_cache_eligible_rewrite_turns", 0), retroactive_rewrite_turns=data.get("retroactive_rewrite_turns", 0), latest_turn_only_rewrite_turns=data.get("latest_turn_only_rewrite_turns", 0), turns=turns, ) return summary def decode_project_key(project_key: str) -> str: """Decode Claude's project directory encoding back to a local path-ish string.""" if "--" not in project_key: return project_key.replace("-", "\\") drive, remainder = project_key.split("--", 1) return drive + ":\\" + remainder.replace("-", "\\") def _parse_timestamp(value: str | None) -> datetime: if not value: return datetime.min.replace(tzinfo=timezone.utc) if value.endswith("Z"): value = value[:-1] + "+00:00" return datetime.fromisoformat(value).astimezone(timezone.utc) def _canonical_block_key(block: Any) -> str: return json.dumps(block, sort_keys=True, separators=(",", ":"), ensure_ascii=False) def _assistant_blocks_from_content(content: Any) -> list[dict[str, Any]]: if isinstance(content, str): return [{"type": "text", "text": content}] if content else [] if isinstance(content, list): return [block for block in content if isinstance(block, dict)] return [] def _messages_have_images(messages: list[dict[str, Any]]) -> bool: for message in messages: content = message.get("content") if not isinstance(content, list): continue for block in content: if isinstance(block, dict) and block.get("type") == "image": return True return False def _finalize_group( group: dict[str, Any] | None, pending_messages: list[dict[str, Any]], turns: list[ReplayTurn], *, session_id: str, project_key: str, decoded_project_path: str, ) -> None: if not group: return assistant_message = { "role": "assistant", "content": group["blocks"] if group["blocks"] else "", } turns.append( ReplayTurn( session_id=session_id, project_key=project_key, decoded_project_path=decoded_project_path, request_id=group["request_id"], model=group["model"], timestamp=group["timestamp"], input_messages=copy.deepcopy(pending_messages), assistant_message=assistant_message, output_tokens=group["output_tokens"], observed_input_tokens=group["observed_input_tokens"], observed_cache_read_tokens=group["observed_cache_read_tokens"], observed_cache_write_tokens=group["observed_cache_write_tokens"], ) ) def load_session_replay(session_file: Path) -> SessionReplay | None: """Load a top-level Claude session transcript into replayable request turns.""" project_key = session_file.parent.name decoded_project_path = decode_project_key(project_key) session_id = session_file.stem pending_messages: list[dict[str, Any]] = [] turns: list[ReplayTurn] = [] current_group: dict[str, Any] | None = None try: with session_file.open("r", encoding="utf-8") as handle: for raw_line in handle: line = raw_line.strip() if not line: continue try: event = json.loads(line) except json.JSONDecodeError: continue event_type = event.get("type") message = event.get("message") if ( event_type == "user" and isinstance(message, dict) and message.get("role") == "user" ): _finalize_group( current_group, pending_messages, turns, session_id=session_id, project_key=project_key, decoded_project_path=decoded_project_path, ) current_group = None pending_messages.clear() pending_messages.append(copy.deepcopy(message)) continue if ( event_type == "assistant" and isinstance(message, dict) and message.get("role") == "assistant" and event.get("requestId") ): request_id = str(event["requestId"]) usage = message.get("usage") or {} timestamp = _parse_timestamp(event.get("timestamp")) blocks = _assistant_blocks_from_content(message.get("content")) if current_group is None or current_group["request_id"] != request_id: had_group = current_group is not None _finalize_group( current_group, pending_messages, turns, session_id=session_id, project_key=project_key, decoded_project_path=decoded_project_path, ) if had_group: pending_messages.clear() current_group = { "request_id": request_id, "model": str(message.get("model", "unknown")), "timestamp": timestamp, "blocks": [], "seen": set(), "output_tokens": 0, "observed_input_tokens": 0, "observed_cache_read_tokens": 0, "observed_cache_write_tokens": 0, } for block in blocks: key = _canonical_block_key(block) if key not in current_group["seen"]: current_group["seen"].add(key) current_group["blocks"].append(copy.deepcopy(block)) current_group["output_tokens"] = max( current_group["output_tokens"], int(usage.get("output_tokens", 0) or 0), ) current_group["observed_input_tokens"] = max( current_group["observed_input_tokens"], int(usage.get("input_tokens", 0) or 0), ) current_group["observed_cache_read_tokens"] = max( current_group["observed_cache_read_tokens"], int(usage.get("cache_read_input_tokens", 0) or 0), ) current_group["observed_cache_ - benchmarks/codemode_ab_report.py (3984 bytes)
#!/usr/bin/env python3 """Paired report for the code-mode steering A/B (benchmarks/codemode_ab.py).""" from __future__ import annotations import argparse import json import statistics import sys from collections import defaultdict def paired(rows, key): """Per-(task,rep) steered-minus-control deltas for a numeric field.""" by = defaultdict(dict) for r in rows: if r.get("error") or r.get(key) is None: continue by[(r["task"], r.get("rep"))][r["arm"]] = r[key] return [ (k, v["steered"] - v["control"], v["control"], v["steered"]) for k, v in by.items() if "steered" in v and "control" in v ] def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--results", required=True) args = ap.parse_args() rows = json.loads(open(args.results).read()) # Re-grade stored answers with the current scorer so a grading fix does not # require re-running (and re-paying for) the suite. sys.path.insert(0, str(__import__("pathlib").Path(__file__).resolve().parent)) from codemode_ab import TASKS, TASKS_MULTI, score spec = {t[0]: (t[2], t[3]) for t in list(TASKS) + list(TASKS_MULTI)} for r in rows: if r.get("task") in spec and "answer" in r: truth, kind = spec[r["task"]] r["correct"] = score(kind, truth, r["answer"]) errs = [r for r in rows if r.get("error")] rows = [r for r in rows if not r.get("error")] print(f"runs: {len(rows)} errors: {len(errs)}") for arm in ("control", "steered"): a = [r for r in rows if r["arm"] == arm] if not a: continue ok = sum(1 for r in a if r.get("correct")) print(f"\n[{arm}] n={len(a)} correct={ok}/{len(a)} ({100 * ok / len(a):.0f}%)") for f, label, unit in ( ("cost_usd", "cost", "$"), ("num_turns", "turns", ""), ("tool_calls", "tool calls", ""), ("bash_calls", "bash calls", ""), ("compound_bash", "compound bash", ""), ("fetched_chars", "bytes fetched", ""), ("wall_s", "wall", "s"), ): vals = [r.get(f) or 0 for r in a] print( f" {label:16s} mean {unit}{statistics.mean(vals):>10.4f} " f"median {unit}{statistics.median(vals):>10.4f} total {unit}{sum(vals):>12.2f}" ) print("\n=== PAIRED (steered - control), per task ===") for f, label in ( ("cost_usd", "cost $"), ("num_turns", "turns"), ("tool_calls", "tool calls"), ("fetched_chars", "bytes fetched"), ("compound_bash", "compound bash"), ): d = paired(rows, f) if not d: continue deltas = [x[1] for x in d] wins = sum(1 for x in deltas if x < 0) losses = sum(1 for x in deltas if x > 0) mean = statistics.mean(deltas) ctrl_tot = sum(x[2] for x in d) pct = 100 * sum(deltas) / ctrl_tot if ctrl_tot else 0 line = ( f" {label:16s} mean Δ {mean:+12.4f} total Δ {sum(deltas):+12.4f} " f"({pct:+.1f}%) steered better/worse/tie: {wins}/{losses}/{len(deltas) - wins - losses}" ) if len(deltas) > 1: sd = statistics.stdev(deltas) se = sd / (len(deltas) ** 0.5) line += f" 95%CI [{mean - 1.96 * se:+.4f}, {mean + 1.96 * se:+.4f}]" print(line) print("\n=== per-task cost detail ===") d = paired(rows, "cost_usd") print(f" {'task':24s} {'control':>10s} {'steered':>10s} {'delta':>10s} {'Δ%':>7s}") for (task, _rep), delta, c, s in sorted(d): print(f" {task:24s} {c:10.4f} {s:10.4f} {delta:+10.4f} {100 * delta / c:+6.1f}%") d = paired(rows, "correct") regress = [k for k, dd, c, s in d if c and not s] if regress: print(f"\n !! correctness REGRESSED on: {regress}") return 0 if __name__ == "__main__": sys.exit(main()) - benchmarks/codemode_ab.py (12630 bytes)
#!/usr/bin/env python3 """Live A/B for the headroom-codemode probe-batching directive. Runs a fixed exploration-task suite through `claude -p` twice per task -- once with the steering plugin active (HEADROOM_CODEMODE=1) and once without -- in a clean detached worktree, then scores cost, turns, burst structure, bytes fetched and answer correctness from the resulting session transcripts. Replay benchmarks cannot measure this: steering changes which calls the model emits, so the counterfactual is not present in recorded sessions. """ from __future__ import annotations import argparse import json import os import re import subprocess import sys import time from pathlib import Path PROJECTS = Path.home() / ".claude" / "projects" # (id, prompt, ground truth, kind) TASKS = [ ( "ccr_py_count", "How many .py files are directly in headroom/ccr/? Answer with just the number.", "10", "num", ), ( "max_rounds", "What is the default value of max_retrieval_rounds in ResponseHandlerConfig? Answer with just the number.", "3", "num", ), ( "plugin_dirs", "How many directories are directly under plugins/? Answer with just the number.", "5", "num", ), ( "logcompressor_lines", "How many lines are in crates/headroom-core/src/transforms/log_compressor.rs? Answer with just the number.", "1795", "num", ), ( "openclaw_version", "What version is declared in plugins/openclaw/package.json? Answer with just the version string.", "0.37.0", "exact", ), ( "test_ccr_files", "How many files directly in tests/ (not in subdirectories) have names starting with test_ccr? Answer with just the number.", "24", "num", ), ( "crate_names", "List the directory names under crates/ as a comma-separated sorted list, nothing else.", "headroom-core,headroom-parity,headroom-proxy,headroom-py,headroom-simulators", "set", ), ( "agenthooks_events", "How many distinct hook event names are defined in plugins/headroom-agent-hooks/hooks/hooks.json? Answer with just the number.", "2", "num", ), ] # Multi-probe suite: each task needs several small independent facts, i.e. the # exact shape the directive targets. This measures the CEILING for steering, # not a field average -- real observed bursts average 2.6 calls. TASKS_MULTI = [ ( "crate_rs_counts", "For each directory under crates/, report how many .rs files it contains recursively. " "Answer as name=count pairs, comma-separated, sorted by name, nothing else.", "headroom-core=99,headroom-parity=3,headroom-proxy=86,headroom-py=2,headroom-simulators=7", "seq", ), ( "file_existence", "Do these paths exist? pyproject.toml, uv.lock, Cargo.toml, mkdocs.yml, tsconfig.json. " "Answer as five yes/no values, comma-separated, in that order, nothing else.", "yes,yes,yes,no,no", "seq", ), ( "dir_counts", "Report three numbers: how many .py files are directly in benchmarks/, how many .md files " "are directly in wiki/, and how many directories are directly under plugins/. " "Answer as three comma-separated numbers in that order, nothing else.", "25,35,5", "seq", ), ( "ccr_symbols", "In headroom/ccr/, report how many .py files contain the string 'CCR_TOOL_NAME', how many " "contain 'async def', and how many contain 'class '. Answer as three comma-separated numbers " "in that order, nothing else.", "6,5,8", "seq", ), ( "plugin_versions", "Report the version field from each of: plugins/openclaw/package.json, " "plugins/headroom-agent-hooks/.claude-plugin/plugin.json. " "Answer as two comma-separated version strings in that order, nothing else.", "0.37.0,0.37.0", "seq", ), ( "line_counts", "Report the line count of each of these files: headroom/ccr/response_handler.py, " "headroom/ccr/tool_injection.py, headroom/ccr/mcp_server.py. " "Answer as three comma-separated numbers in that order, nothing else.", "1095,615,1199", "seq", ), ] def score(kind: str, truth: str, answer: str) -> bool: """Grade an answer. Answers often carry a trailing citation ("3\n\n(file.py:72)"), so numeric grading reads the FIRST number, not the last -- otherwise a line number in the citation is graded as the answer. """ a = (answer or "").strip().replace("`", "") if kind == "num": nums = re.findall(r"\d[\d,]*", a) return bool(nums) and nums[0].replace(",", "") == truth if kind == "exact": return truth in a if kind == "seq": want = [t.strip().lower() for t in truth.split(",")] # take the first len(want) comma/whitespace-separated fields of the # first line that has at least that many -- ignores trailing prose for line in [a] + a.splitlines(): got = [t.strip().lower() for t in re.split(r"[,\s]+", line) if t.strip()] if len(got) >= len(want) and got[: len(want)] == want: return True return False if kind == "set": got = {t.strip().lower() for t in re.split(r"[,\s]+", a) if t.strip()} want = {t.strip().lower() for t in truth.split(",")} return want.issubset(got) return False def trace(session_id: str) -> dict: """Pull burst structure and bytes fetched out of the session transcript.""" hits = list(PROJECTS.glob(f"**/{session_id}.jsonl")) if not hits: return {"trace_found": False} rows = [] for line in hits[0].open(errors="replace"): try: rows.append(json.loads(line)) except Exception: pass calls = bash_calls = 0 bursts: list[int] = [] run = 0 fetched = 0 pend: set[str] = set() batched = 0 # compound shell commands (a proxy for directive compliance) def text_of(x): if isinstance(x, str): return x if isinstance(x, list): return "\n".join( b.get("text", "") if isinstance(b, dict) and b.get("type") == "text" else json.dumps(b) for b in x ) return json.dumps(x) if x is not None else "" for d in rows: t = d.get("type") if t == "assistant": tus = [ b for b in (d.get("message") or {}).get("content") or [] if isinstance(b, dict) and b.get("type") == "tool_use" ] if tus: run += 1 for b in tus: calls += 1 pend.add(b.get("id")) if b.get("name") == "Bash": bash_calls += 1 cmd = (b.get("input") or {}).get("command") or "" if re.search(r";|&&", cmd): batched += 1 else: if run: bursts.append(run) run = 0 elif t in ("user", "attachment"): for b in (d.get("message") or {}).get("content") or []: if ( isinstance(b, dict) and b.get("type") == "tool_result" and b.get("tool_use_id") in pend ): pend.discard(b.get("tool_use_id")) fetched += len(text_of(b.get("content"))) if run: bursts.append(run) return { "trace_found": True, "tool_calls": calls, "bash_calls": bash_calls, "compound_bash": batched, "bursts": len(bursts), "max_burst": max(bursts) if bursts else 0, "mean_burst": round(sum(bursts) / len(bursts), 2) if bursts else 0, "fetched_chars": fetched, } def run_one(task, arm: str, wt: Path, plugin_dir: Path, model: str | None) -> dict: tid, prompt, truth, kind = task env = dict(os.environ) env["HEADROOM_CODEMODE"] = "1" if arm == "steered" else "0" cmd = [ "claude", "-p", prompt, "--output-format", "json", "--permission-mode", "auto", "--plugin-dir", str(plugin_dir), ] if model: cmd += ["--model", model] t0 = time.time() proc = subprocess.run(cmd, cwd=wt, env=env, capture_output=True, text=True, timeout=600) wall = time.time() - t0 try: d = json.loads(proc.stdout) except Exception: return { "task": tid, "arm": arm, "error": (proc.stdout or proc.stderr)[:300], "wall_s": wall, } u = d.get("usage") or {} rec = { "task": tid, "arm": arm, "session_id": d.get("session_id"), "cost_usd": d.get("total_cost_usd"), "num_turns": d.get("num_turns"), "wall_s": round(wall, 1), "input_tokens": u.get("input_tokens", 0), "cache_creation": u.get("cache_creation_input_tokens", 0), "cache_read": u.get("cache_read_input_tokens", 0), "output_tokens": u.get("output_tokens", 0), "answer": str(d.get("result", ""))[:200], } rec["correct"] = score(kind, truth, rec["answer"]) rec.update(trace(rec["session_id"] or "")) return rec def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--worktree", required=True) ap.add_argument("--plugin-dir", required=True) ap.add_argument("--reps", type=int, default=1) ap.add_argument("--model", default=None) ap.add_argument("--out", default="benchmark_results/codemode_ab.json") ap.add_argument("--suite", choices=["simple", "multi"], default="simple") args = ap.parse_args() wt = Path(args.worktree).resolve() # The per-run cleanup below is `git reset --hard` + `git clean -fd`: refuse a # worktree that is not a clean git checkout so it cannot destroy user work. # --untracked-files=all: `status.showUntrackedFiles=no` would hide the untracked # files that `git clean -fd` deletes. git = subprocess.run( ["git", "status", "--porcelain", "--untracked-files=all"], cwd=wt, capture_output=True, text=True, ) if git.returncode != 0: print(f"error: --worktree {wt} is not inside a git work tree", file=sys.stderr) return 2 if git.stdout.strip(): print( f"error: --worktree {wt} has uncommitted changes; the script runs " "`git reset --hard` and `git clean -fd` there after each run. " "Commit or stash first, or use a clean detached worktree.", file=sys.stderr, ) return 2 pd = Path(args.plugin_dir).resolve() out = Path(args.out) out.parent.mkdir(parents=True, exist_ok=True) results = json.loads(out.read_text()) if out.exists() else [] done = {(r["task"], r["arm"], r.get("rep")) for r in results} suite = TASKS_MULTI if args.suite == "multi" else TASKS for rep in range(args.reps): for task in suite: # interleave arms so prompt-cache warmth cannot favour one side for arm in ("steered", "control") if rep % 2 == 0 else ("control", "steered"): if (task[0], arm, rep) in done: continue rec = run_one(task, arm, wt, pd, args.model) rec["rep"] = rep results.append(rec) out.write_text(json.dumps(results, indent=2)) print( f" {task[0]:22s} {arm:8s} rep{rep} " f"${rec.get('cost_usd', 0) or 0:.4f} turns={rec.get('num_turns')} " f"calls={rec.get('tool_calls')} ok={rec.get('correct')}", flush=True, ) dirty = subprocess.run( ["git", "status", "--porcelain"], cwd=wt, capture_output=True, text=True ).stdout.strip() if dirty: print(f" !! worktree dirtied, resetting: {dirty[:120]}", flush=True) subprocess.run(["git", "reset", "--hard", "-q"], cwd=wt) subprocess.run(["git", "clean", "-qfd"], cwd=wt) print(f"\nwrote {out}") return 0 if __name__ == "__main__": sys.exit(main()) - benchmarks/comprehensive_eval.py (26940 bytes)
#!/usr/bin/env python3 """ Comprehensive Headroom Evaluation: Real Data, Real Accuracy This benchmark uses REAL data from established sources: 1. Berkeley Function Calling Leaderboard (BFCL) - Real API schemas and ground truth 2. HotpotQA - Real Wikipedia passages with verified answers 3. Cached OSS data - Real GitHub issues, code, and logs from popular projects We measure BOTH: - Compression ratio (token savings) - Accuracy preservation (ground truth comparison) Usage: pip install datasets # For HuggingFace datasets export ANTHROPIC_API_KEY=sk-ant-... python benchmarks/comprehensive_eval.py """ import json import os import time from dataclasses import dataclass from pathlib import Path from typing import Any # ============================================================================= # DATA LOADERS - Real data from established sources # ============================================================================= def load_bfcl_samples(n: int = 20) -> list[dict]: """ Load real function calling examples from Berkeley Function Calling Leaderboard. These are REAL API schemas with ground truth function calls. """ try: from datasets import load_dataset ds = load_dataset( "gorilla-llm/Berkeley-Function-Calling-Leaderboard", "BFCL_v3_live_simple", split="train", # BFCL is plain parquet; never execute a dataset repository's own # loading script, here or anywhere else in the tree. trust_remote_code=False, ) samples = [] for i, item in enumerate(ds): if i >= n: break samples.append( { "id": f"bfcl_{i}", "type": "function_calling", "question": item.get("question", [[]])[0][0]["content"] if item.get("question") else "", "functions": item.get("function", []), "ground_truth": item.get("ground_truth", []), "source": "BFCL_v3", } ) return samples except Exception as e: print(f"Warning: Could not load BFCL dataset: {e}") return [] def load_hotpotqa_samples(n: int = 20) -> list[dict]: """ Load real multi-hop QA examples from HotpotQA. These are REAL Wikipedia passages with verified answers. """ try: from datasets import load_dataset ds = load_dataset("hotpotqa/hotpot_qa", "fullwiki", split="validation") samples = [] for i, item in enumerate(ds): if i >= n: break # Build context from supporting facts context_parts = [] for title, sentences in zip(item["context"]["title"], item["context"]["sentences"]): context_parts.append(f"## {title}\n" + "\n".join(sentences)) samples.append( { "id": f"hotpot_{i}", "type": "multi_hop_qa", "question": item["question"], "context": "\n\n".join(context_parts), "ground_truth": item["answer"], "supporting_facts": item["supporting_facts"], "source": "HotpotQA", } ) return samples except Exception as e: print(f"Warning: Could not load HotpotQA dataset: {e}") return [] def load_real_github_data() -> dict: """ Load cached real GitHub data from popular OSS projects. This includes actual issues, PRs, and code from kubernetes, pytorch, etc. """ # Cache file for reproducibility cache_file = Path(__file__).parent / "data" / "github_cache.json" if cache_file.exists(): with open(cache_file) as f: return json.load(f) # If no cache, return sample structure (would fetch from GitHub API in production) return { "issues": [], "code_snippets": [], "pull_requests": [], "error_logs": [], } def load_real_logs() -> list[dict]: """ Load real production log samples. These are actual log formats from various systems. """ # Real log formats from different systems return [ # Java Spring Boot logs { "type": "java_spring", "content": """2024-01-15 14:23:45.123 ERROR [http-nio-8080-exec-7] c.e.api.UserController - Failed to process request org.springframework.dao.DataAccessException: Unable to acquire connection from pool at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:376) at com.example.api.UserController.getUser(UserController.java:45) Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1421) ... 42 more""", }, # Kubernetes events { "type": "kubernetes", "content": """NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE default 2m Warning FailedScheduling pod/nginx-deployment-5d8b9c7f4-x2k9j 0/3 nodes are available: 3 Insufficient memory default 5m Normal Scheduled pod/redis-master-0 Successfully assigned default/redis-master-0 to node-2 kube-system 1h Warning NodeNotReady node/node-3 Node node-3 status is now: NodeNotReady default 30s Normal Pulled pod/api-server-7f8d9c8b5-m4n2p Container image "api-server:v2.1.0" already present on machine""", }, # Python traceback { "type": "python_traceback", "content": """Traceback (most recent call last): File "/app/services/payment.py", line 127, in process_payment result = stripe.PaymentIntent.create( File "/usr/local/lib/python3.11/site-packages/stripe/api_resources/payment_intent.py", line 87, in create return cls._static_request("post", url, params=params) File "/usr/local/lib/python3.11/site-packages/stripe/api_requestor.py", line 298, in request raise error.CardError(error_data.get("message"), error_data.get("param"), error_data.get("code")) stripe.error.CardError: Your card was declined. This transaction requires authentication. Request ID: req_a1b2c3d4e5f6g7h8 Error Code: card_declined Decline Code: authentication_required""", }, # nginx access logs { "type": "nginx_access", "content": """192.168.1.100 - - [15/Jan/2024:14:30:45 +0000] "GET /api/v2/users/12345 HTTP/1.1" 200 1543 "https://app.example.com/dashboard" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" 192.168.1.101 - - [15/Jan/2024:14:30:46 +0000] "POST /api/v2/orders HTTP/1.1" 201 892 "https://app.example.com/checkout" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" 192.168.1.102 - admin [15/Jan/2024:14:30:47 +0000] "DELETE /api/v2/users/67890 HTTP/1.1" 403 124 "-" "curl/7.81.0" 10.0.0.50 - - [15/Jan/2024:14:30:48 +0000] "GET /health HTTP/1.1" 200 15 "-" "kube-probe/1.25" """, }, ] def load_real_code_samples() -> list[dict]: """ Load real code samples from OSS projects. These are actual implementations, not synthetic examples. """ return [ # Real Python - FastAPI auth middleware pattern { "language": "python", "file": "auth/middleware.py", "source": "FastAPI patterns", "content": '''"""Authentication middleware for FastAPI applications.""" from datetime import datetime, timedelta from typing import Optional import jwt from fastapi import HTTPException, Security, Depends from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials from pydantic import BaseModel class TokenPayload(BaseModel): sub: str exp: datetime iat: datetime scopes: list[str] = [] class JWTBearer(HTTPBearer): def __init__(self, auto_error: bool = True): super().__init__(auto_error=auto_error) async def __call__(self, credentials: HTTPAuthorizationCredentials = Security(HTTPBearer())): if not credentials: raise HTTPException(status_code=403, detail="Invalid authorization code") if credentials.scheme != "Bearer": raise HTTPException(status_code=403, detail="Invalid authentication scheme") return self.verify_jwt(credentials.credentials) def verify_jwt(self, token: str) -> TokenPayload: try: payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM]) return TokenPayload(**payload) except jwt.ExpiredSignatureError: raise HTTPException(status_code=401, detail="Token has expired") except jwt.JWTError: raise HTTPException(status_code=403, detail="Could not validate credentials") def create_access_token(subject: str, scopes: list[str] = [], expires_delta: Optional[timedelta] = None): expire = datetime.utcnow() + (expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)) to_encode = {"sub": subject, "exp": expire, "iat": datetime.utcnow(), "scopes": scopes} return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM) async def get_current_user(token: TokenPayload = Depends(JWTBearer())) -> dict: user = await user_service.get_by_id(token.sub) if not user: raise HTTPException(status_code=404, detail="User not found") return user ''', }, # Real TypeScript - React hook pattern { "language": "typescript", "file": "hooks/useAsync.ts", "source": "React patterns", "content": """import { useState, useCallback, useEffect, useRef } from 'react'; interface AsyncState<T> { data: T | null; error: Error | null; loading: boolean; } interface UseAsyncOptions { immediate?: boolean; onSuccess?: (data: any) => void; onError?: (error: Error) => void; } export function useAsync<T>( asyncFunction: (...args: any[]) => Promise<T>, options: UseAsyncOptions = {} ) { const { immediate = false, onSuccess, onError } = options; const [state, setState] = useState<AsyncState<T>>({ data: null, error: null, loading: immediate, }); const mountedRef = useRef(true); const lastCallId = useRef(0); const execute = useCallback( async (...args: any[]) => { const callId = ++lastCallId.current; setState(prev => ({ ...prev, loading: true, error: null })); try { const result = await asyncFunction(...args); if (mountedRef.current && callId === lastCallId.current) { setState({ data: result, error: null, loading: false }); onSuccess?.(result); } return result; } catch (error) { if (mountedRef.current && callId === lastCallId.current) { const err = error instanceof Error ? error : new Error(String(error)); setState({ data: null, error: err, loading: false }); onError?.(err); } throw error; } }, [asyncFunction, onSuccess, onError] ); useEffect(() => { if (immediate) execute(); return () => { mountedRef.current = false; }; }, []); return { ...state, execute, reset: () => setState({ data: null, error: null, loading: false }) }; } """, }, # Real Go - HTTP middleware pattern { "language": "go", "file": "middleware/ratelimit.go", "source": "Go patterns", "content": """package middleware import ( "net/http" "sync" "time" "golang.org/x/time/rate" ) type visitor struct { limiter *rate.Limiter lastSeen time.Time } type RateLimiter struct { visitors map[string]*visitor mu sync.RWMutex rate rate.Limit burst int cleanup time.Duration } func NewRateLimiter(r rate.Limit, b int) *RateLimiter { rl := &RateLimiter{ visitors: make(map[string]*visitor), rate: r, burst: b, cleanup: time.Minute * 3, } go rl.cleanupVisitors() return rl } func (rl *RateLimiter) getVisitor(ip string) *rate.Limiter { rl.mu.Lock() defer rl.mu.Unlock() v, exists := rl.visitors[ip] if !exists { limiter := rate.NewLimiter(rl.rate, rl.burst) rl.visitors[ip] = &visitor{limiter: limiter, lastSeen: time.Now()} return limiter } v.lastSeen = time.Now() return v.limiter } func (rl *RateLimiter) cleanupVisitors() { for { time.Sleep(rl.cleanup) rl.mu.Lock() for ip, v := range rl.visitors { if time.Since(v.lastSeen) > rl.cleanup { delete(rl.visitors, ip) } } rl.mu.Unlock() } } func (rl *RateLimiter) Limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ip := r.RemoteAddr limiter := rl.getVisitor(ip) if !limiter.Allow() { http.Error(w, "Rate limit exceeded", http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } """, }, ] # ============================================================================= # EVALUATION METRICS # ============================================================================= @dataclass class AccuracyResult: """Ground truth accuracy measurement.""" exact_match: bool f1_score: float contains_answer: bool def compute_f1(prediction: str, ground_truth: str) -> float: """Compute token-level F1 score.""" pred_tokens = set(prediction.lower().split()) truth_tokens = set(ground_truth.lower().split()) if not pred_tokens or not truth_tokens: return 0.0 common = pred_tokens & truth_tokens if not common: return 0.0 precision = len(common) / len(pred_tokens) recall = len(common) / len(truth_tokens) return 2 * precision * recall / (precision + recall) def evaluate_answer(prediction: str, ground_truth: str) -> AccuracyResult: """Evaluate prediction against ground truth.""" pred_lower = prediction.lower().strip() truth_lower = ground_truth.lower().strip() return AccuracyResult( exact_match=pred_lower == truth_lower, f1_score=compute_f1(prediction, ground_truth), contains_answer=truth_lower in pred_lower, ) # ============================================================================= # MIXED CONTENT SCENARIOS # ============================================================================= @dataclass class Scenario: """A test scenario with mixed content types.""" name: str description: str tool_outputs: list[dict] # Simulated tool outputs question: str ground_truth: str | None = None validation_fn: Any = None # Custom validation function def create_sre_scenario() -> Scenario: """ Real SRE incident scenari - benchmarks/compression_benchmark.py (33040 bytes)
""" Truncation vs Summarization vs Headroom: A Fair Benchmark This benchmark compares three approaches to context compression: 1. Truncation - Keep first N items (industry standard) 2. Summarization - Use LLM to summarize (common alternative) 3. Headroom - Statistical compression with retrieval FAIRNESS PRINCIPLES: - Include scenarios where each approach could win - Use realistic data patterns - Measure both compression AND answer quality - Report failures honestly Metrics: - Tokens saved (compression ratio) - Answer accuracy (can LLM still answer correctly?) - Cost (including summarization LLM calls) - Latency """ import hashlib import json import random import time from dataclasses import dataclass from typing import Literal # We'll use OpenAI for the actual LLM calls try: from openai import OpenAI OPENAI_AVAILABLE = True except ImportError: OPENAI_AVAILABLE = False # Headroom imports try: from headroom.config import SmartCrusherConfig from headroom.tokenizers import TiktokenCounter from headroom.transforms.smart_crusher import SmartCrusher HEADROOM_AVAILABLE = True except ImportError: HEADROOM_AVAILABLE = False # Kompress imports (ML baseline) try: from headroom.transforms.kompress_compressor import KompressCompressor, is_kompress_available KOMPRESS_AVAILABLE = is_kompress_available() except ImportError: KOMPRESS_AVAILABLE = False @dataclass class Question: """A question about the data with ground truth answer.""" text: str ground_truth: str answer_location: Literal["early", "middle", "late", "scattered", "semantic"] difficulty: Literal["easy", "medium", "hard"] @dataclass class Scenario: """A benchmark scenario with data and questions.""" name: str description: str data: list[dict] questions: list[Question] expected_winner: str # Which approach should theoretically win @dataclass class ApproachResult: """Result of running one approach on one scenario.""" approach: str scenario: str tokens_original: int tokens_after: int compression_ratio: float compression_latency_ms: float llm_cost_usd: float # Cost of summarization if applicable answers: list[dict] # {question, expected, actual, correct} accuracy: float total_cost_usd: float # Compression cost + query cost # ============================================================================= # DATA GENERATORS - Realistic synthetic data # ============================================================================= def generate_log_data( n_entries: int = 500, error_positions: list[int] = None ) -> tuple[list[dict], list[Question]]: """ Generate realistic server logs. 95% routine logs, 5% interesting events (errors, warnings). Errors placed at specified positions to test different approaches. """ if error_positions is None: # Default: errors at beginning, middle, and end error_positions = [3, n_entries // 2, n_entries - 5] log_templates = [ {"level": "INFO", "message": "Health check passed", "service": "api-gateway"}, {"level": "INFO", "message": "Request processed successfully", "service": "api-gateway"}, {"level": "INFO", "message": "Cache hit for user session", "service": "redis"}, {"level": "INFO", "message": "Database query completed", "service": "postgres"}, {"level": "INFO", "message": "Authentication successful", "service": "auth"}, { "level": "DEBUG", "message": "Connection pool stats: active=5, idle=15", "service": "postgres", }, ] error_templates = [ { "level": "ERROR", "message": "Connection refused to payment-service:8080 - ECONNREFUSED", "service": "payment-processor", "error_code": "PAYMENT_SERVICE_DOWN", "trace_id": "abc123", }, { "level": "ERROR", "message": "Timeout waiting for response from inventory-service after 30000ms", "service": "order-processor", "error_code": "INVENTORY_TIMEOUT", "trace_id": "def456", }, { "level": "CRITICAL", "message": "Out of memory: Java heap space - killing process", "service": "recommendation-engine", "error_code": "OOM_KILLED", "trace_id": "ghi789", }, ] logs = [] base_time = 1705320000 # Some Unix timestamp error_idx = 0 for i in range(n_entries): base_time + i * 60 # 1 minute apart if i in error_positions and error_idx < len(error_templates): entry = error_templates[error_idx].copy() error_idx += 1 else: entry = random.choice(log_templates).copy() entry["timestamp"] = f"2024-01-15T{10 + (i // 60):02d}:{i % 60:02d}:00Z" entry["request_id"] = f"req-{hashlib.md5(str(i).encode()).hexdigest()[:8]}" # nosec B324 logs.append(entry) # Questions designed to test different approaches questions = [ Question( text="What error code was returned by the payment service?", ground_truth="PAYMENT_SERVICE_DOWN", answer_location="early", # Position 3 difficulty="easy", ), Question( text="Which service experienced a timeout and what was the trace ID?", ground_truth="order-processor service had timeout with trace_id def456", answer_location="middle", difficulty="medium", ), Question( text="What critical error occurred and which service was affected?", ground_truth="Out of memory (OOM_KILLED) in recommendation-engine", answer_location="late", # Near end difficulty="medium", ), Question( text="How many distinct error types are in the logs?", ground_truth="3", answer_location="scattered", difficulty="hard", ), ] return logs, questions def generate_file_search_data(n_files: int = 1000) -> tuple[list[dict], list[Question]]: """ Generate realistic code search results. Simulates searching a codebase - lots of files with similar metadata, specific files of interest scattered throughout. """ # Common directories and file patterns dirs = [ "src/api", "src/services", "src/utils", "src/models", "src/controllers", "src/middleware", "tests/unit", "tests/integration", "lib/core", "lib/helpers", "config", "scripts", ] extensions = [".py", ".py", ".py", ".ts", ".js", ".json", ".yaml"] # Weighted toward .py # Files of interest (scattered at specific positions) special_files = { 50: { "path": "src/auth/jwt_handler.py", "size": 2341, "description": "JWT token validation and refresh", }, 250: { "path": "src/services/payment_processor.py", "size": 5672, "description": "Stripe payment integration", }, 500: { "path": "src/middleware/rate_limiter.py", "size": 1823, "description": "Redis-based rate limiting", }, 750: { "path": "config/database.py", "size": 892, "description": "PostgreSQL connection settings", }, 999: { "path": "src/api/health_check.py", "size": 456, "description": "Kubernetes health endpoints", }, } files = [] for i in range(n_files): if i in special_files: f = special_files[i].copy() f["type"] = "file" f["language"] = "python" f["modified"] = "2024-01-15" else: dir_path = random.choice(dirs) ext = random.choice(extensions) f = { "type": "file", "path": f"{dir_path}/module_{i}{ext}", "size": random.randint(200, 5000), "language": "python" if ext == ".py" else "typescript" if ext == ".ts" else "javascript", "modified": f"2024-01-{random.randint(1, 15):02d}", } files.append(f) questions = [ Question( text="Which file handles JWT token operations?", ground_truth="src/auth/jwt_handler.py", answer_location="early", # Position 50 difficulty="easy", ), Question( text="What file contains the Stripe payment integration and how large is it?", ground_truth="src/services/payment_processor.py, 5672 bytes", answer_location="middle", # Position 250 difficulty="medium", ), Question( text="Which file implements rate limiting and what technology does it use?", ground_truth="src/middleware/rate_limiter.py uses Redis", answer_location="middle", # Position 500 difficulty="medium", ), Question( text="What is the last Python file in the results and what does it do?", ground_truth="src/api/health_check.py - Kubernetes health endpoints", answer_location="late", # Position 999 difficulty="hard", ), ] return files, questions def generate_metrics_data(n_points: int = 500) -> tuple[list[dict], list[Question]]: """ Generate realistic time series metrics. Baseline values with anomalies (spikes) at specific positions. This is where Headroom should excel - detecting statistical outliers. """ base_cpu = 45.0 base_memory = 62.0 base_requests = 1000 # Anomaly positions anomalies = { 50: {"cpu": 95.0, "memory": 88.0, "requests": 5000, "event": "traffic_spike"}, 200: {"cpu": 98.0, "memory": 95.0, "requests": 150, "event": "service_degradation"}, 450: {"cpu": 15.0, "memory": 30.0, "requests": 50, "event": "service_restart"}, } metrics = [] base_time = 1705320000 for i in range(n_points): base_time + i * 60 if i in anomalies: point = { "timestamp": f"2024-01-15T{10 + (i // 60):02d}:{i % 60:02d}:00Z", "cpu_percent": anomalies[i]["cpu"], "memory_percent": anomalies[i]["memory"], "requests_per_min": anomalies[i]["requests"], "status": "degraded" if anomalies[i]["event"] != "traffic_spike" else "ok", "event": anomalies[i]["event"], } else: point = { "timestamp": f"2024-01-15T{10 + (i // 60):02d}:{i % 60:02d}:00Z", "cpu_percent": round(base_cpu + random.uniform(-5, 5), 1), "memory_percent": round(base_memory + random.uniform(-3, 3), 1), "requests_per_min": base_requests + random.randint(-100, 100), "status": "ok", } metrics.append(point) questions = [ Question( text="When did the traffic spike occur and what was the requests_per_min?", ground_truth="Around 10:50, requests_per_min was 5000", answer_location="early", difficulty="easy", ), Question( text="What event caused service degradation and what were the CPU/memory values?", ground_truth="service_degradation event, CPU 98%, memory 95%", answer_location="middle", difficulty="medium", ), Question( text="When did the service restart and how can you tell from the metrics?", ground_truth="Around 17:30, CPU dropped to 15%, memory to 30%, requests to 50", answer_location="late", difficulty="hard", ), Question( text="How many anomalous events occurred in total?", ground_truth="3", answer_location="scattered", difficulty="hard", ), ] return metrics, questions # ============================================================================= # COMPRESSION APPROACHES # ============================================================================= def truncate_data(data: list[dict], max_items: int = 20) -> list[dict]: """Simple truncation - keep first N items.""" return data[:max_items] def summarize_data( data: list[dict], client: "OpenAI", model: str = "gpt-4o-mini" ) -> tuple[str, float]: """ Use LLM to summarize the data. Returns (summary_text, cost_usd). """ data_str = json.dumps(data, indent=2) # Truncate if too long for summarization call if len(data_str) > 100000: data_str = data_str[:100000] + "\n... [truncated for summarization]" prompt = f"""Summarize this data concisely, preserving all important information including: - Any errors, warnings, or anomalies - Key identifiers (IDs, names, paths) - Statistical outliers - Important events Data: {data_str} Provide a structured summary that retains all critical details.""" start = time.time() response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], max_tokens=2000, ) latency = (time.time() - start) * 1000 summary = response.choices[0].message.content # Estimate cost (gpt-4o-mini pricing) input_tokens = response.usage.prompt_tokens output_tokens = response.usage.completion_tokens cost = (input_tokens * 0.00015 + output_tokens * 0.0006) / 1000 # Per token pricing return summary, cost, latency def kompress_compress(data: list[dict]) -> tuple[str, dict]: """ Use Kompress (ModernBERT) for ML-based compression. Returns (compressed_text, metadata). """ if not KOMPRESS_AVAILABLE: raise RuntimeError("Kompress not available. Install with: pip install headroom-ai[ml]") compressor = KompressCompressor() # Convert data to string for Kompress (it works on text, not structured data) data_str = json.dumps(data, indent=2) start = time.time() result = compressor.compress(data_str) latency = (time.time() - start) * 1000 metadata = { "latency_ms": latency, "original_tokens": result.original_tokens, "compressed_tokens": result.compressed_tokens, "compression_ratio": result.compression_ratio, } return result.compressed, metadata def headroom_compress(data: list[dict], query_context: str = "") -> tuple[list[dict], dict]: """ Use Headroom's SmartCrusher for statistical compression. Returns (compressed_data, metadata). """ if not HEADROOM_AVAILABLE: raise RuntimeErro - benchmarks/conftest.py (10947 bytes)
"""Pytest fixtures for Headroom benchmarks. This module provides shared fixtures for benchmark tests including: - Generated data arrays of various sizes - Conversation fixtures with tool calls - System prompts with/without dynamic dates - Mock tokenizers for consistent measurement All fixtures are designed to produce deterministic data for reliable benchmark comparisons across runs. """ from __future__ import annotations import json import random from typing import Any import pytest from benchmarks.scenarios.conversations import ( generate_agentic_conversation, generate_rag_conversation, ) from benchmarks.scenarios.tool_outputs import ( generate_api_responses, generate_database_rows, generate_log_entries, generate_search_results, ) # Set seed for reproducible benchmarks random.seed(42) # ============================================================================= # Mock Tokenizer # ============================================================================= class MockTokenCounter: """Mock token counter for benchmarks. Uses simple character-based estimation (4 chars = 1 token) for fast, consistent token counting without model dependencies. """ def count_text(self, text: str) -> int: """Estimate tokens in text (4 chars = 1 token).""" return max(1, len(text) // 4) def count_message(self, message: dict[str, Any]) -> int: """Estimate tokens in a message.""" content = message.get("content", "") if isinstance(content, str): return self.count_text(content) + 4 # Overhead for role elif isinstance(content, list): total = 0 for block in content: if isinstance(block, dict): if block.get("type") == "text": total += self.count_text(block.get("text", "")) elif block.get("type") == "tool_result": total += self.count_text(str(block.get("content", ""))) elif block.get("type") == "tool_use": total += self.count_text(json.dumps(block.get("input", {}))) return total + 4 else: return 10 # Default estimate def count_messages(self, messages: list[dict[str, Any]]) -> int: """Estimate tokens in message list.""" return sum(self.count_message(m) for m in messages) @pytest.fixture def mock_token_counter() -> MockTokenCounter: """Provide mock token counter for benchmarks.""" return MockTokenCounter() @pytest.fixture def mock_tokenizer(mock_token_counter: MockTokenCounter): """Provide mock Tokenizer wrapper.""" from headroom.tokenizer import Tokenizer return Tokenizer(token_counter=mock_token_counter, model="benchmark-model") # ============================================================================= # Data Array Fixtures (various sizes) # ============================================================================= @pytest.fixture def items_100() -> list[dict[str, Any]]: """Generate 100 search result items.""" random.seed(42) return generate_search_results(100) @pytest.fixture def items_1000() -> list[dict[str, Any]]: """Generate 1000 search result items.""" random.seed(42) return generate_search_results(1000) @pytest.fixture def items_10000() -> list[dict[str, Any]]: """Generate 10000 search result items.""" random.seed(42) return generate_search_results(10000) @pytest.fixture def log_entries_100() -> list[dict[str, Any]]: """Generate 100 log entries.""" random.seed(42) return generate_log_entries(100) @pytest.fixture def log_entries_1000() -> list[dict[str, Any]]: """Generate 1000 log entries.""" random.seed(42) return generate_log_entries(1000) @pytest.fixture def database_rows_100() -> list[dict[str, Any]]: """Generate 100 database rows with metrics (for anomaly detection).""" random.seed(42) return generate_database_rows(100, table_type="metrics") @pytest.fixture def database_rows_1000() -> list[dict[str, Any]]: """Generate 1000 database rows with metrics.""" random.seed(42) return generate_database_rows(1000, table_type="metrics") @pytest.fixture def api_responses_100() -> list[dict[str, Any]]: """Generate 100 API response items.""" random.seed(42) return generate_api_responses(100) # ============================================================================= # Conversation Fixtures # ============================================================================= @pytest.fixture def conversation_10_turns() -> list[dict[str, Any]]: """Generate 10-turn agentic conversation with tool calls.""" random.seed(42) return generate_agentic_conversation( turns=10, tool_calls_per_turn=1, items_per_tool_response=50 ) @pytest.fixture def conversation_50_turns() -> list[dict[str, Any]]: """Generate 50-turn agentic conversation with tool calls.""" random.seed(42) return generate_agentic_conversation( turns=50, tool_calls_per_turn=2, items_per_tool_response=50 ) @pytest.fixture def conversation_200_turns() -> list[dict[str, Any]]: """Generate 200-turn agentic conversation (stress test).""" random.seed(42) return generate_agentic_conversation( turns=200, tool_calls_per_turn=1, items_per_tool_response=30 ) @pytest.fixture def rag_conversation_5k() -> list[dict[str, Any]]: """Generate RAG conversation with ~5K context tokens.""" random.seed(42) return generate_rag_conversation(context_tokens=5000, num_queries=3) @pytest.fixture def rag_conversation_20k() -> list[dict[str, Any]]: """Generate RAG conversation with ~20K context tokens.""" random.seed(42) return generate_rag_conversation(context_tokens=20000, num_queries=5) @pytest.fixture def rag_conversation_50k() -> list[dict[str, Any]]: """Generate RAG conversation with ~50K context tokens.""" random.seed(42) return generate_rag_conversation(context_tokens=50000, num_queries=5) # ============================================================================= # System Prompt Fixtures # ============================================================================= @pytest.fixture def system_prompt_with_date() -> str: """System prompt containing dynamic date.""" return """You are a helpful AI assistant. Current date: 2025-01-06 Today is Monday, January 6th, 2025. You have access to various tools for searching and querying data. Always provide accurate and helpful responses.""" @pytest.fixture def system_prompt_without_date() -> str: """System prompt without dynamic date (stable).""" return """You are a helpful AI assistant. You have access to various tools for searching and querying data. Always provide accurate and helpful responses. Guidelines: 1. Be concise and accurate 2. Use tools when appropriate 3. Cite sources when available""" @pytest.fixture def system_prompt_long() -> str: """Long system prompt for cache alignment testing.""" sections = [ "You are an expert AI assistant with deep knowledge in software engineering.", "\n\n## Capabilities\n- Code analysis and review\n- Debugging and troubleshooting\n- Architecture recommendations\n- Performance optimization", "\n\n## Guidelines\n1. Always explain your reasoning\n2. Provide code examples when helpful\n3. Consider edge cases\n4. Suggest best practices", "\n\n## Tools Available\n- search_code: Search code repositories\n- query_database: Query application databases\n- get_logs: Retrieve service logs\n- run_tests: Execute test suites", "\n\n## Response Format\n- Use markdown for formatting\n- Include code blocks with syntax highlighting\n- Organize long responses with headers\n- Summarize key points at the end", ] return "".join(sections) @pytest.fixture def messages_with_tool_output(items_100) -> list[dict[str, Any]]: """Messages containing a tool output for crushing.""" return [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Search for recent users"}, { "role": "assistant", "content": None, "tool_calls": [ { "id": "call_123", "type": "function", "function": {"name": "search_users", "arguments": '{"limit": 100}'}, } ], }, { "role": "tool", "tool_call_id": "call_123", "content": json.dumps(items_100), }, ] @pytest.fixture def messages_with_system_date(system_prompt_with_date) -> list[dict[str, Any]]: """Messages with system prompt containing date.""" return [ {"role": "system", "content": system_prompt_with_date}, {"role": "user", "content": "What's the current date?"}, {"role": "assistant", "content": "Today is January 6th, 2025."}, ] # ============================================================================= # Transform Configuration Fixtures # ============================================================================= @pytest.fixture def smart_crusher_config(): """SmartCrusher config optimized for benchmarks.""" from headroom.config import SmartCrusherConfig return SmartCrusherConfig( enabled=True, min_items_to_analyze=5, min_tokens_to_crush=0, # Always crush max_items_after_crush=15, variance_threshold=2.0, ) @pytest.fixture def cache_aligner_config(): """CacheAligner config for benchmarks.""" from headroom.config import CacheAlignerConfig return CacheAlignerConfig( enabled=True, normalize_whitespace=True, collapse_blank_lines=True, ) # ============================================================================= # JSON String Fixtures (for relevance benchmarks) # ============================================================================= @pytest.fixture def json_items_100(items_100) -> list[str]: """100 items as JSON strings.""" return [json.dumps(item) for item in items_100] @pytest.fixture def json_items_1000(items_1000) -> list[str]: """1000 items as JSON strings.""" return [json.dumps(item) for item in items_1000] @pytest.fixture def query_context_uuid() -> str: """Query context containing a UUID (for BM25 testing).""" return "Find the record with UUID 550e8400-e29b-41d4-a716-446655440000" @pytest.fixture def query_context_semantic() -> str: """Query context requiring semantic understanding.""" return "Show me all the failed requests and errors" @pytest.fixture def query_context_mixed() -> str: """Query context with both exact match and semantic terms.""" return "Find user 12345 and show any associated errors" - benchmarks/dynamic_detector_benchmark.py (11430 bytes)
#!/usr/bin/env python3 """ Real-world benchmark for DynamicContentDetector. Tests the detector against realistic system prompts from AI coding agents, chatbots, and enterprise applications. """ import statistics import time from dataclasses import dataclass from typing import Any from headroom.cache.dynamic_detector import ( DetectorConfig, DynamicContentDetector, ) @dataclass class BenchmarkResult: """Result of a single benchmark run.""" name: str content_length: int spans_found: int categories: list[str] static_length: int dynamic_length: int latency_ms: float tiers_used: list[str] warnings: list[str] # Real-world system prompts REAL_WORLD_PROMPTS = { "claude_code_style": """You are Claude, an AI assistant created by Anthropic to be helpful, harmless, and honest. Today is Tuesday, January 7, 2026. Current time: 10:30:45 AM PST. You are operating in a software development environment with access to: - File system operations - Terminal commands - Web search Session ID: sess_abc123def456ghi789jkl012 Request ID: req_xyz789abc123def456ghi789 User: tchopra Workspace: /Users/tchopra/claude-projects/headroom Be concise, accurate, and helpful. Follow the user's instructions carefully.""", "enterprise_assistant": """You are an enterprise AI assistant for Acme Corporation. Current Date: 2026-01-07T10:30:00Z Last Updated: 2026-01-07T09:00:00Z User Profile: - Name: John Smith - Employee ID: EMP-2024-00542 - Department: Engineering - Manager: Sarah Johnson - Location: San Francisco, CA - Hire Date: March 15, 2023 System Status: - API Version: v2.3.1-beta - Server Load: 45% - Active Users: 1,247 - Queue Length: 23 Budget Information: - Monthly Allowance: $5,000.00 - Used This Month: $2,341.67 - Remaining: $2,658.33 Help the user with their work tasks while following company policies.""", "coding_agent": """You are an autonomous coding agent with access to tools. Environment: - OS: macOS Darwin 25.1.0 - Working Directory: /Users/developer/projects/myapp - Git Branch: feature/JIRA-1234-add-auth - Last Commit: a1b2c3d4e5f6 (2 hours ago) - Node Version: v20.10.0 - Python Version: 3.11.7 Current Task Context: - Task ID: 550e8400-e29b-41d4-a716-446655440000 - Created: 2026-01-07T08:15:30Z - Priority: High - Estimated Time: 2 hours API Keys Available: - OPENAI_API_KEY: sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxxxx - ANTHROPIC_API_KEY: sk-ant-xxxxxxxxxxxxxxxxxxxxxxxxxxxx - DATABASE_URL: postgresql://user:pass@localhost:5432/mydb Execute tasks step by step, verify each action, and report progress.""", "customer_support": """You are a customer support agent for TechStore Inc. Current Time: January 7, 2026, 3:45 PM EST Support Ticket: #TKT-2026-0107-4521 Customer Information: - Name: Alice Chen - Email: [email protected] - Phone: (555) 123-4567 - Customer Since: August 2021 - Loyalty Tier: Gold - Total Purchases: $12,456.78 Recent Orders: - Order #ORD-2026-0105-7823 - iPhone 15 Pro - $1,199.00 - Delivered - Order #ORD-2025-1220-3456 - AirPods Pro - $249.00 - Delivered - Order #ORD-2025-1115-9012 - MacBook Air - $1,299.00 - Returned Active Issues: - Case #CS-2026-0107-001 - Battery drain issue - Open since today Provide helpful, empathetic support while following company guidelines.""", "data_analysis": """You are a data analysis assistant. Report Generated: 2026-01-07 10:30:00 UTC Report ID: RPT-550e8400-e29b-41d4-a716-446655440000 Data Range: 2025-12-01 to 2025-12-31 Summary Statistics: - Total Revenue: $1,234,567.89 - Total Orders: 45,678 - Average Order Value: $27.03 - Top Product: Widget Pro ($234,567.00) - Top Region: California (23.4%) Key Metrics: - DAU: 125,000 - MAU: 890,000 - Churn Rate: 2.3% - NPS Score: 67 Anomalies Detected: - Spike on Dec 15: 3.2x normal traffic - Drop on Dec 25: 0.4x normal (expected - holiday) Help analyze the data and provide insights.""", "minimal_static": """You are a helpful AI assistant. Your role is to: 1. Answer questions accurately 2. Be concise and clear 3. Follow instructions carefully 4. Admit when you don't know something Always be helpful, harmless, and honest.""", "heavy_dynamic": """Session started at 2026-01-07T10:30:45.123Z Request ID: req_abc123def456ghi789jkl012mno345pqr678 Trace ID: 550e8400-e29b-41d4-a716-446655440000 Parent Span: span_xyz789abc123 User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) IP Address: 192.168.1.100 Geo: San Francisco, CA, USA (37.7749, -122.4194) Auth Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... Token Expires: 2026-01-07T11:30:45Z Refresh Token: rt_abc123def456 Last Login: 2026-01-06T18:45:30Z Login Count: 1,247 Account Balance: $5,432.10 Credit Limit: $10,000.00 Real-time Stock Prices (as of 10:30 AM): - AAPL: $185.42 (+1.2%) - GOOGL: $142.89 (-0.5%) - MSFT: $378.23 (+0.8%) - AMZN: $156.78 (+2.1%) Process this request.""", } def run_benchmark( prompts: dict[str, str], tiers: list[str], iterations: int = 10, ) -> dict[str, Any]: """Run benchmark on prompts with specified tiers.""" config = DetectorConfig(tiers=tiers) # type: ignore detector = DynamicContentDetector(config) results: dict[str, list[BenchmarkResult]] = {} for name, content in prompts.items(): results[name] = [] for _ in range(iterations): start = time.perf_counter() result = detector.detect(content) elapsed = (time.perf_counter() - start) * 1000 categories = list({s.category.value for s in result.spans}) results[name].append( BenchmarkResult( name=name, content_length=len(content), spans_found=len(result.spans), categories=categories, static_length=len(result.static_content), dynamic_length=len(result.dynamic_content), latency_ms=elapsed, tiers_used=result.tiers_used, warnings=result.warnings, ) ) return results def print_results( results: dict[str, list[BenchmarkResult]], tier_name: str, ): """Print benchmark results.""" print(f"\n{'=' * 80}") print(f"BENCHMARK RESULTS: {tier_name}") print(f"{'=' * 80}") for name, runs in results.items(): latencies = [r.latency_ms for r in runs] avg_latency = statistics.mean(latencies) std_latency = statistics.stdev(latencies) if len(latencies) > 1 else 0 # Use first run for span info (consistent across runs) first = runs[0] compression = ( (1 - first.static_length / first.content_length) * 100 if first.content_length > 0 else 0 ) print(f"\n📄 {name}") print(f" Content: {first.content_length:,} chars") print(f" Spans found: {first.spans_found}") print(f" Categories: {', '.join(first.categories) if first.categories else 'none'}") print(f" Static: {first.static_length:,} chars | Dynamic: {first.dynamic_length:,} chars") print(f" Compression: {compression:.1f}% removed") print(f" Latency: {avg_latency:.2f}ms ± {std_latency:.2f}ms") print(f" Tiers used: {', '.join(first.tiers_used)}") if first.warnings: print(f" ⚠️ Warnings: {len(first.warnings)}") def print_comparison(all_results: dict[str, dict[str, list[BenchmarkResult]]]): """Print comparison across tiers.""" print(f"\n{'=' * 80}") print("TIER COMPARISON") print(f"{'=' * 80}") prompts = list(REAL_WORLD_PROMPTS.keys()) tiers = list(all_results.keys()) # Header header = f"{'Prompt':<25}" for tier in tiers: header += f" | {tier:>12} spans | {'latency':>8}" print(header) print("-" * len(header)) for prompt in prompts: row = f"{prompt:<25}" for tier in tiers: if prompt in all_results[tier]: runs = all_results[tier][prompt] spans = runs[0].spans_found latency = statistics.mean([r.latency_ms for r in runs]) row += f" | {spans:>12} | {latency:>7.2f}ms" else: row += f" | {'N/A':>12} | {'N/A':>8}" print(row) # Summary print(f"\n{'=' * 80}") print("SUMMARY") print(f"{'=' * 80}") for tier in tiers: all_latencies = [] total_spans = 0 for runs in all_results[tier].values(): all_latencies.extend([r.latency_ms for r in runs]) total_spans += runs[0].spans_found avg = statistics.mean(all_latencies) p50 = statistics.median(all_latencies) p99 = ( sorted(all_latencies)[int(len(all_latencies) * 0.99)] if len(all_latencies) > 1 else avg ) print(f"\n{tier}:") print(f" Total spans detected: {total_spans}") print(f" Avg latency: {avg:.2f}ms") print(f" P50 latency: {p50:.2f}ms") print(f" P99 latency: {p99:.2f}ms") def show_detection_details(prompt_name: str, content: str): """Show detailed detection for a specific prompt.""" print(f"\n{'=' * 80}") print(f"DETECTION DETAILS: {prompt_name}") print(f"{'=' * 80}") config = DetectorConfig(tiers=["regex"]) detector = DynamicContentDetector(config) result = detector.detect(content) print(f"\nOriginal content ({len(content)} chars):") print("-" * 40) print(content[:500] + "..." if len(content) > 500 else content) print(f"\n\nDetected spans ({len(result.spans)}):") print("-" * 40) for span in result.spans: print( f" [{span.category.value:12}] '{span.text[:50]}{'...' if len(span.text) > 50 else ''}'" ) print(f"\n\nStatic content ({len(result.static_content)} chars):") print("-" * 40) print( result.static_content[:500] + "..." if len(result.static_content) > 500 else result.static_content ) print(f"\n\nDynamic content ({len(result.dynamic_content)} chars):") print("-" * 40) print(result.dynamic_content if result.dynamic_content else "(none)") def main(): """Run the benchmark.""" print("🚀 Dynamic Content Detector - Real World Benchmark") print("=" * 80) iterations = 20 # Test each tier configuration tier_configs = { "regex_only": ["regex"], # "regex+ner": ["regex", "ner"], # Uncomment if spacy installed # "all_tiers": ["regex", "ner", "semantic"], # Uncomment if all deps installed } all_results: dict[str, dict[str, list[BenchmarkResult]]] = {} for tier_name, tiers in tier_configs.items(): print(f"\n⏱️ Running {tier_name} ({iterations} iterations per prompt)...") results = run_benchmark(REAL_WORLD_PROMPTS, tiers, iterations) all_results[tier_name] = results print_results(results, tier_name) # Print comparison if multiple tiers tested if len(all_results) > 1: print_comparison(all_results) # Show detailed detection for a few prompts print("\n" + "=" * 80) print("DETAILED DETECTION EXAMPLES") print("=" * 80) for name in ["claude_code_style", "enterprise_assistant", "heavy_dynamic"]: show_detection_details(name, REAL_WORLD_PROMPTS[name]) if __name__ == "__main__": main() - benchmarks/headroom_adversarial_benchmark.py (22318 bytes)
""" Headroom ADVERSARIAL Benchmark: True Worst Cases The previous "worst case" scenarios still had JSON structure. This benchmark tests TRUE adversarial cases: 1. Dense prose - research papers, no structure 2. Code diffs - every line matters, minimal redundancy 3. Encrypted/random data - no patterns possible 4. Tiny datasets - not enough data for statistics 5. High-entropy unique content - no repeated patterns """ import hashlib import json import os import random import string from dataclasses import dataclass try: from openai import OpenAI # noqa: F401 OPENAI_AVAILABLE = True except ImportError: OPENAI_AVAILABLE = False try: from headroom import HeadroomClient, OpenAIProvider HEADROOM_AVAILABLE = True except ImportError: HEADROOM_AVAILABLE = False # ============================================================================= # ADVERSARIAL DATA GENERATORS # ============================================================================= def generate_research_paper_excerpts(num_papers: int = 10) -> dict: """ Dense academic text - every word carries meaning. No JSON structure, no repetition, pure prose. """ # Simulated research paper abstracts - dense, unique content papers = [] topics = [ ("quantum computing", "qubit coherence", "error correction", "topological"), ("machine learning", "transformer architecture", "attention mechanism", "gradient"), ("climate science", "carbon sequestration", "permafrost", "albedo effect"), ("neuroscience", "synaptic plasticity", "hippocampal", "neurogenesis"), ("economics", "monetary policy", "inflation targeting", "yield curve"), ("genetics", "CRISPR-Cas9", "gene expression", "epigenetic"), ("astrophysics", "gravitational waves", "neutron star", "black hole merger"), ("materials science", "graphene", "superconductivity", "metamaterial"), ("cryptography", "post-quantum", "lattice-based", "homomorphic encryption"), ("pharmacology", "receptor binding", "pharmacokinetics", "bioavailability"), ] for i in range(num_papers): topic = topics[i % len(topics)] # Generate unique, dense academic prose abstract = f""" This paper presents novel findings in {topic[0]} research, specifically addressing the challenge of {topic[1]} optimization. Our methodology employs a combination of {topic[2]} analysis and {topic[3]} modeling approaches that have not been previously explored in the literature. Through rigorous experimentation with {random.randint(50, 500)} samples across {random.randint(3, 12)} controlled conditions, we demonstrate a {random.randint(15, 45)}% improvement over baseline methods (p < 0.{random.randint(1, 5):02d}). The theoretical framework builds upon the seminal work of {random.choice(["Smith et al.", "Johnson & Lee", "Chen group", "Williams lab"])} (20{random.randint(15, 23)}), extending their {random.choice(["analytical", "computational", "experimental", "theoretical"])} approach to address {random.choice(["scalability concerns", "edge cases", "real-world constraints", "noise sensitivity"])}. Our key contribution is the development of a {random.choice(["novel algorithm", "unified framework", "hybrid methodology", "robust protocol"])} that achieves {random.choice(["state-of-the-art", "competitive", "superior", "breakthrough"])} performance while maintaining {random.choice(["computational efficiency", "interpretability", "generalizability", "reproducibility"])}. Implications of this work extend to {random.choice(["industrial applications", "clinical settings", "policy decisions", "fundamental understanding"])} in the domain of {topic[0]}. We identify {random.randint(3, 7)} key factors that influence {topic[1]} behavior, with {random.choice(["temperature", "pressure", "concentration", "frequency", "duration"])} being the most significant (correlation coefficient r = 0.{random.randint(70, 95)}). Future work will focus on {random.choice(["scaling", "optimizing", "validating", "extending"])} these findings to {random.choice(["larger systems", "different domains", "real-world deployment", "clinical trials"])}. """.strip() papers.append( { "paper_id": f"arxiv:{random.randint(2000, 2400)}.{random.randint(10000, 99999)}", "title": f"Advances in {topic[0].title()}: A {random.choice(['Novel', 'Comprehensive', 'Systematic', 'Rigorous'])} Approach to {topic[1].title()}", "authors": [f"Author{j}" for j in range(random.randint(2, 6))], "abstract": abstract, "year": random.randint(2022, 2024), "citations": random.randint(0, 150), } ) # Return as plain text, not JSON structure output = "RESEARCH PAPER SEARCH RESULTS\n" + "=" * 50 + "\n\n" for p in papers: output += f"[{p['paper_id']}] {p['title']}\n" output += f"Authors: {', '.join(p['authors'])} ({p['year']})\n" output += f"Citations: {p['citations']}\n\n" output += p["abstract"] + "\n\n" output += "-" * 50 + "\n\n" return { "tool": "research_search", "result": output, # Plain text, not JSON! } def generate_code_diff(num_files: int = 15, changes_per_file: int = 20) -> dict: """ Git diff output - every line is unique and important. Can't summarize code changes - need exact lines. """ languages = { "py": ( "def ", "class ", "import ", "return ", "if ", "for ", "while ", "try:", "except:", "with ", ), "ts": ( "function ", "const ", "interface ", "import ", "export ", "return ", "if ", "for ", "async ", "await ", ), "go": ( "func ", "type ", "import ", "return ", "if ", "for ", "defer ", "go ", "chan ", "struct ", ), "rs": ( "fn ", "struct ", "impl ", "use ", "let ", "match ", "if ", "for ", "pub ", "async ", ), } diff_output = "" for file_idx in range(num_files): ext = random.choice(list(languages.keys())) keywords = languages[ext] filename = f"src/module_{file_idx}/handler.{ext}" diff_output += f"diff --git a/{filename} b/{filename}\n" diff_output += f"index {hashlib.md5(f'{file_idx}a'.encode()).hexdigest()[:7]}..{hashlib.md5(f'{file_idx}b'.encode()).hexdigest()[:7]} 100644\n" # nosec B324 diff_output += f"--- a/{filename}\n" diff_output += f"+++ b/{filename}\n" line_num = random.randint(10, 50) for change_idx in range(changes_per_file): # Generate realistic code changes keyword = random.choice(keywords) var_name = f"{''.join(random.choices(string.ascii_lowercase, k=random.randint(4, 10)))}" value = random.randint(1, 1000) diff_output += ( f"@@ -{line_num},{random.randint(3, 7)} +{line_num},{random.randint(3, 7)} @@\n" ) # Context line diff_output += f" {random.choice(keywords)}{var_name}_{change_idx}()\n" # Removed line old_impl = f"{keyword}{var_name} = {value}" diff_output += f"- {old_impl}\n" # Added line (different) new_impl = f"{keyword}{var_name} = {value + random.randint(1, 100)}" diff_output += f"+ {new_impl}\n" # More context diff_output += f" {random.choice(keywords)}{var_name}_next()\n" line_num += random.randint(10, 30) diff_output += "\n" return { "tool": "git_diff", "result": diff_output, # Plain text diff } def generate_encrypted_data(size_kb: int = 20) -> dict: """ Base64 encoded / encrypted content - NO patterns possible. This is the ultimate adversarial case for compression. """ # Generate random bytes and base64 encode random_bytes = bytes([random.randint(0, 255) for _ in range(size_kb * 1024)]) import base64 encoded = base64.b64encode(random_bytes).decode("ascii") return { "tool": "encrypted_blob", "result": { "blob_id": f"enc_{hashlib.md5(encoded[:100].encode()).hexdigest()[:16]}", # nosec B324 "encryption": "AES-256-GCM", "content": encoded, "size_bytes": len(random_bytes), }, } def generate_tiny_dataset(num_items: int = 5) -> dict: """ Very small dataset - not enough data for statistical patterns. """ items = [] for i in range(num_items): items.append( { "id": i + 1, "name": f"Item {chr(65 + i)}", "value": random.randint(100, 999), "note": f"Unique note for item {i + 1}: {hashlib.md5(str(i).encode()).hexdigest()[:20]}", # nosec B324 } ) return {"tool": "tiny_query", "result": {"count": num_items, "items": items}} def generate_conversation_history(num_messages: int = 50) -> dict: """ Chat conversation - context and flow matter, not just content. Each message builds on previous, can't remove context. """ participants = ["Alice", "Bob", "Charlie", "Diana"] messages = [] topics = [ "the quarterly review", "the product launch", "the customer feedback", "the technical debt", "the team restructuring", ] current_topic = random.choice(topics) for i in range(num_messages): sender = participants[i % len(participants)] # Change topic occasionally if random.random() < 0.1: current_topic = random.choice(topics) # Generate contextual message message_templates = [ f"I think we need to reconsider {current_topic}. The data shows {random.choice(['promising', 'concerning', 'mixed'])} results.", f"Building on what {participants[(i - 1) % len(participants)]} said, I'd add that {random.choice(['timing', 'resources', 'alignment'])} is crucial here.", f"Let me share some context: when we discussed {current_topic} last month, we agreed on {random.choice(['three priorities', 'a phased approach', 'immediate action'])}.", f"I disagree with the previous point. {current_topic.title()} requires {random.choice(['more analysis', 'quick action', 'stakeholder buy-in'])} first.", f"To summarize so far: we've covered {random.choice(['the risks', 'the opportunities', 'the constraints'])} of {current_topic}. Next steps?", f"Quick question about {current_topic}: have we considered {random.choice(['the budget impact', 'customer perception', 'timeline feasibility'])}?", f"I can take the action item on {current_topic}. Will need input from {random.choice(participants)} by {random.choice(['EOD', 'tomorrow', 'Friday'])}.", ] messages.append( { "timestamp": f"2024-01-17T{10 + (i // 10):02d}:{(i * 2) % 60:02d}:00Z", "sender": sender, "message": random.choice(message_templates), } ) # Format as conversation transcript transcript = "MEETING TRANSCRIPT\n" + "=" * 50 + "\n\n" for msg in messages: transcript += f"[{msg['timestamp']}] {msg['sender']}:\n" transcript += f" {msg['message']}\n\n" return {"tool": "meeting_transcript", "result": transcript} # ============================================================================= # ADVERSARIAL SCENARIOS # ============================================================================= @dataclass class AdversarialScenario: name: str description: str why_adversarial: str system_prompt: str user_query: str tools: list[dict] expected_behavior: str # What we expect to happen def create_research_synthesis_scenario() -> AdversarialScenario: return AdversarialScenario( name="Research Paper Synthesis", description="Synthesize findings from 10 research papers", why_adversarial="Dense academic prose with no structural repetition. Every sentence carries unique meaning. No JSON overhead to compress.", system_prompt="""You are a research assistant synthesizing academic papers. Each paper's findings are important. Don't skip any paper. Focus on methodology differences and key findings.""", user_query="Synthesize these research papers. For each paper, summarize the key methodology and findings. Then identify common themes and contradictions across papers.", tools=[generate_research_paper_excerpts(num_papers=10)], expected_behavior="Headroom should have minimal compression - prose has no structural redundancy", ) def create_code_review_scenario() -> AdversarialScenario: return AdversarialScenario( name="Code Diff Review", description="Review a large code diff across 15 files", why_adversarial="Git diffs have minimal redundancy. Each +/- line is unique code. Can't summarize - reviewer needs exact changes.", system_prompt="""You are a senior engineer reviewing a pull request. Every changed line matters. Look for bugs, style issues, and potential problems. Don't skip any file or change.""", user_query="Review this diff carefully. For each file, identify: 1) What changed, 2) Any bugs or issues, 3) Style concerns. Be thorough.", tools=[generate_code_diff(num_files=15, changes_per_file=20)], expected_behavior="Headroom should struggle - code changes are unique and can't be summarized", ) def create_encrypted_analysis_scenario() -> AdversarialScenario: return AdversarialScenario( name="Encrypted Data Analysis", description="Analyze encrypted/encoded data blob", why_adversarial="Random/encrypted data has maximum entropy. No patterns exist to compress. This is mathematically incompressible.", system_prompt="""You are a data analyst examining an encrypted data blob. Describe what you observe about the data format and structure.""", user_query="Examine this encrypted data blob. What can you tell about its format? Is there any visible structure? What's the encoding?", tools=[generate_encrypted_data(size_kb=20)], expected_behavior="Headroom CANNOT compress this - random data has no patterns", ) def create_small_data_scenario() -> AdversarialScenario: return AdversarialScenario( name="Tiny Dataset Analysis", description="Analyz - benchmarks/headroom_worst_case_benchmark.py (34856 bytes)
""" Headroom Worst-Case Benchmark: Where Compression Hurts This benchmark tests scenarios where Headroom's statistical compression may NOT be beneficial - to understand the limits of the approach. Worst cases for Headroom: 1. Highly unique data (no patterns to compress) 2. Data where every item is equally important 3. Data where subtle differences matter 4. Small datasets (not enough data for statistics) 5. Data where you need EXACT recall (audit/legal) """ import hashlib import json import os import random import time from dataclasses import dataclass from typing import Any try: from openai import OpenAI # noqa: F401 OPENAI_AVAILABLE = True except ImportError: OPENAI_AVAILABLE = False try: from headroom import HeadroomClient, OpenAIProvider HEADROOM_AVAILABLE = True except ImportError: HEADROOM_AVAILABLE = False # ============================================================================= # WORST-CASE DATA GENERATORS # ============================================================================= def generate_unique_support_tickets(num_tickets: int = 50) -> dict: """ Customer support tickets where EVERY ticket is unique and important. No redundancy - each customer has a different problem. This is hard for Headroom because: - No repeated patterns to compress - Every ticket needs attention - Can't safely remove any ticket """ products = ["Pro Plan", "Enterprise", "Starter", "Team", "Individual"] issues = [ "billing discrepancy of ${amount} on invoice #{inv}", "cannot access feature '{feature}' despite paying for it", "data export failing with error code {code}", "SSO integration with {provider} not working", "API rate limits hitting at {rate}/min instead of promised {expected}/min", "webhook deliveries delayed by {hours} hours", "user {user} locked out after password reset", "mobile app crashing on {device} with iOS {version}", "search returning wrong results for query '{query}'", "file upload stuck at {percent}% for files over {size}MB", "notification emails going to spam for domain {domain}", "timezone showing {wrong_tz} instead of {correct_tz}", "dashboard metrics {days} days out of date", "cannot downgrade from {from_plan} to {to_plan}", "GDPR data deletion request not completing for user {user_id}", ] severities = ["critical", "high", "medium", "low"] tickets = [] for i in range(num_tickets): # Each ticket is genuinely unique issue_template = issues[i % len(issues)] issue = issue_template.format( amount=random.randint(50, 5000), inv=random.randint(10000, 99999), feature=random.choice( ["advanced analytics", "custom domains", "API access", "SSO", "audit logs"] ), code=f"ERR_{random.randint(1000, 9999)}", provider=random.choice(["Okta", "Azure AD", "Google Workspace", "OneLogin"]), rate=random.randint(100, 500), expected=random.randint(1000, 5000), hours=random.randint(1, 48), user=f"user_{random.randint(1000, 9999)}@company{random.randint(1, 100)}.com", device=random.choice(["iPhone 15", "iPhone 14", "iPad Pro", "iPhone 13"]), version=random.choice(["17.2", "17.1", "16.5", "16.4"]), query=random.choice( ["quarterly report", "user metrics", "revenue data", "team performance"] ), percent=random.randint(45, 95), size=random.randint(10, 500), domain=f"company{random.randint(1, 500)}.com", wrong_tz=random.choice(["UTC", "PST", "EST"]), correct_tz=random.choice(["CET", "JST", "IST"]), days=random.randint(2, 14), from_plan=random.choice(["Enterprise", "Pro"]), to_plan=random.choice(["Starter", "Team"]), user_id=f"usr_{hashlib.md5(str(i).encode()).hexdigest()[:8]}", # nosec B324 ) tickets.append( { "ticket_id": f"TKT-{20000 + i}", "customer": { "id": f"cust_{hashlib.md5(f'customer{i}'.encode()).hexdigest()[:8]}", # nosec B324 "name": f"Customer {i + 1}", "company": f"Company {chr(65 + (i % 26))}{i // 26 + 1} Inc.", "plan": random.choice(products), "mrr": random.randint(99, 9999), "account_age_days": random.randint(30, 1500), }, "issue": issue, "severity": random.choice(severities), "created_at": f"2024-01-{random.randint(10, 17):02d}T{random.randint(0, 23):02d}:{random.randint(0, 59):02d}:00Z", "last_response": f"2024-01-{random.randint(15, 17):02d}T{random.randint(0, 23):02d}:{random.randint(0, 59):02d}:00Z", "response_count": random.randint(1, 8), "tags": random.sample( ["billing", "technical", "feature-request", "bug", "urgent", "escalated"], k=random.randint(1, 3), ), "assignee": None, # Unassigned - needs triage } ) return { "tool": "support_queue", "result": {"queue": "unassigned", "total_tickets": num_tickets, "tickets": tickets}, } def generate_unique_error_traces(num_traces: int = 30) -> dict: """ Unique stack traces where each error is different. This is hard for Headroom because: - Each stack trace has different functions, line numbers - Each error message is unique - All errors need investigation """ languages = ["python", "javascript", "go", "java"] traces = [] for i in range(num_traces): lang = random.choice(languages) if lang == "python": trace = generate_python_trace(i) elif lang == "javascript": trace = generate_js_trace(i) elif lang == "go": trace = generate_go_trace(i) else: trace = generate_java_trace(i) traces.append( { "error_id": f"err_{hashlib.md5(str(i).encode()).hexdigest()[:12]}", # nosec B324 "timestamp": f"2024-01-17T{10 + (i % 12):02d}:{(i * 7) % 60:02d}:00Z", "service": random.choice(["api", "worker", "scheduler", "gateway"]), "environment": "production", "language": lang, "error_type": trace["error_type"], "message": trace["message"], "stack_trace": trace["stack"], "context": { "user_id": f"user_{random.randint(10000, 99999)}", "request_id": hashlib.md5(f"req{i}".encode()).hexdigest()[:16], # nosec B324 "endpoint": trace.get("endpoint", "/api/unknown"), }, "occurrence_count": random.randint(1, 5), # Low count - each is unique } ) return { "tool": "error_tracker", "result": {"time_range": "last_24h", "total_unique_errors": num_traces, "errors": traces}, } def generate_python_trace(seed: int) -> dict: """Generate a unique Python stack trace.""" error_types = [ ("ValueError", f"Invalid value for parameter 'config_{seed}': expected int, got str"), ("KeyError", f"'{random.choice(['user', 'account', 'session', 'token'])}_{seed}'"), ("TypeError", f"unsupported operand type(s) for +: 'NoneType' and 'str' in field_{seed}"), ("AttributeError", f"'NoneType' object has no attribute 'process_{seed}'"), ("RuntimeError", f"Maximum recursion depth exceeded in handler_{seed}"), ("ConnectionError", f"Connection refused to service_{seed}:8080"), ( "TimeoutError", f"Operation timed out after {random.randint(30, 120)}s waiting for resource_{seed}", ), ] error_type, message = random.choice(error_types) functions = [ f"process_request_{seed}", f"validate_input_{seed % 10}", f"transform_data_{seed}", f"save_to_db_{seed % 5}", f"send_notification_{seed}", ] stack_lines = [] for j, func in enumerate(random.sample(functions, k=random.randint(3, 5))): line_no = random.randint(50, 500) file_path = f"/app/services/module_{seed % 20}/{func.split('_')[0]}.py" stack_lines.append(f' File "{file_path}", line {line_no}, in {func}') stack_lines.append(f" result = self.handler_{j}(data)") return { "error_type": error_type, "message": message, "stack": "\n".join(stack_lines), "endpoint": f"/api/v{random.randint(1, 3)}/{random.choice(['users', 'orders', 'products'])}/{seed}", } def generate_js_trace(seed: int) -> dict: """Generate a unique JavaScript stack trace.""" error_types = [ ( "TypeError", f"Cannot read property '{random.choice(['map', 'filter', 'length', 'data'])}' of undefined", ), ("ReferenceError", f"config_{seed} is not defined"), ("SyntaxError", f"Unexpected token in JSON at position {random.randint(100, 1000)}"), ("RangeError", f"Maximum call stack size exceeded in recursive_{seed}"), ] error_type, message = random.choice(error_types) stack = f""" at processData_{seed} (/app/src/handlers/processor_{seed % 10}.js:{random.randint(50, 200)}:15) at async handleRequest_{seed} (/app/src/routes/api_{seed % 5}.js:{random.randint(20, 100)}:23) at async Router.dispatch (/app/node_modules/express/router.js:142:12) at async Layer.handle (/app/node_modules/express/layer.js:95:5)""" return { "error_type": error_type, "message": message, "stack": stack, "endpoint": f"/api/{random.choice(['graphql', 'rest', 'webhook'])}/{seed}", } def generate_go_trace(seed: int) -> dict: """Generate a unique Go stack trace.""" error_types = [ ("panic", f"runtime error: index out of range [{seed}] with length {seed - 1}"), ("panic", "runtime error: invalid memory address or nil pointer dereference"), ("error", f"context deadline exceeded after {random.randint(5, 30)}s"), ("error", f"connection refused to database_{seed % 3}:5432"), ] error_type, message = random.choice(error_types) stack = f"""goroutine {random.randint(1, 100)} [running]: main.processHandler_{seed}(0xc0001{seed:04x}, 0x{random.randint(1000, 9999):x}) /app/internal/handlers/handler_{seed % 10}.go:{random.randint(50, 200)} +0x{random.randint(100, 999):x} main.(*Server).ServeHTTP_{seed}(0xc000{seed:04x}, 0x7f{random.randint(1000, 9999):x}) /app/internal/server/server.go:{random.randint(80, 150)} +0x{random.randint(100, 500):x}""" return { "error_type": error_type, "message": message, "stack": stack, } def generate_java_trace(seed: int) -> dict: """Generate a unique Java stack trace.""" error_types = [ ("NullPointerException", f"Cannot invoke method on null object in Service_{seed}"), ("IllegalArgumentException", f"Parameter 'id_{seed}' cannot be negative"), ("SQLException", f"Connection to database_{seed % 3} timed out"), ("OutOfMemoryError", f"Java heap space exhausted processing batch_{seed}"), ] error_type, message = random.choice(error_types) stack = f"""java.lang.{error_type}: {message} at com.app.services.Handler{seed}.process(Handler{seed}.java:{random.randint(50, 200)}) at com.app.controllers.Api{seed % 10}Controller.handle(Api{seed % 10}Controller.java:{random.randint(30, 100)}) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:897) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)""" return { "error_type": error_type, "message": message, "stack": stack, } def generate_medical_records(num_patients: int = 25) -> dict: """ Medical records where EVERY detail matters. This is hard for Headroom because: - Similar symptoms can have different diagnoses - Missing any detail could be dangerous - "Repetitive" info (vitals) is actually critical data """ conditions = [ "Type 2 Diabetes", "Hypertension", "Asthma", "GERD", "Anxiety Disorder", "Hypothyroidism", "Chronic Back Pain", "Migraine", "Allergic Rhinitis", "Depression", ] medications = [ "Metformin 500mg", "Lisinopril 10mg", "Omeprazole 20mg", "Albuterol inhaler", "Sertraline 50mg", "Levothyroxine 50mcg", "Ibuprofen 400mg PRN", "Sumatriptan 50mg PRN", "Loratadine 10mg", ] records = [] for i in range(num_patients): # Each patient has a unique combination of conditions, meds, vitals patient_conditions = random.sample(conditions, k=random.randint(1, 4)) patient_meds = random.sample(medications, k=random.randint(1, 5)) # Vitals - these look "similar" but each patient's baseline is different systolic = random.randint(110, 160) diastolic = random.randint(70, 100) records.append( { "patient_id": f"PT-{100000 + i}", "name": f"Patient {chr(65 + (i % 26))}{chr(65 + ((i // 26) % 26))}", "age": random.randint(25, 85), "sex": random.choice(["M", "F"]), "visit_date": f"2024-01-{random.randint(15, 17):02d}", "chief_complaint": random.choice( [ f"Chest pain radiating to left arm for {random.randint(1, 6)} hours", f"Shortness of breath worsening over {random.randint(1, 14)} days", "Severe headache, worst of life, sudden onset", f"Abdominal pain, {random.choice(['RLQ', 'LLQ', 'epigastric'])}, {random.randint(1, 72)} hours", f"Dizziness and {random.choice(['syncope', 'near-syncope'])} today", f"Fever {random.randint(100, 104)}°F for {random.randint(1, 5)} days", "Medication refill - stable on current regimen", f"Follow-up for recent {random.choice(['hospitalization', 'procedure', 'diagnosis'])}", ] ), "vitals": { "bp": f"{systolic}/{diastolic}", "hr": random.randint(60, 110), "temp": round(random.uniform(97.5, 100.5), 1), "resp": random.randint(12, 22), "spo2": random.randint(94, 100), }, "conditions": patient_con - benchmarks/i18n_compression_eval.py (10554 bytes)
#!/usr/bin/env python3 """i18n compression-quality eval (zh/ja/ko): does extractive compression keep the answer-bearing content in CJK? No LLM/API calls -- fully local. Part C -- our own DETERMINISTIC needle answer-retention (zh/ja/ko): the always- runs regression gate. A distinctive needle sentence is buried (in the middle) in language-matched distractor sentences; compress query-aware; assert the needle survives. No external data. TextCrusher (query-aware) vs truncate (keep-recent) vs random baselines. Part B -- real-transcript fidelity with CJK-aware salient: optional, anonymized. Part A -- natural-data answer-retention on alexandrainst/multi-wiki-qa (zh-cn/ja/ko): optional, via the [evals] datasets extra, skipped if absent. Usage: python benchmarks/i18n_compression_eval.py [transcript.jsonl] """ from __future__ import annotations import glob import os import random import re import sys import time from headroom.transforms.text_crusher import TextCrusher _REDACT = [ (re.compile(r"/Users/[^/\s]+"), "/Users/USER"), (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), "EMAIL"), (re.compile(r"\b(?:sk|pk|ghp|gho|xox[baprs])-[A-Za-z0-9_-]{10,}\b"), "TOKEN"), (re.compile(r"\b[A-Fa-f0-9]{40,}\b"), "HEX"), ] # Split on ASCII and full-width CJK terminators so baselines segment CJK too. _SEG = re.compile(r"(?<=[.!?。!?])\s*|\n+") _CJK_RUN = re.compile(r"[㐀-鿿-ヿ가-]+") def anon(t: str) -> str: for rx, rep in _REDACT: t = rx.sub(rep, t) return t def norm(s: str) -> str: # CJK has no spaces; drop all whitespace so substring match is robust. return re.sub(r"\s+", "", s.lower()) def _segs(text: str) -> list[str]: return [s for s in _SEG.split(text) if s.strip()] def truncate_keep_last(text: str, ratio: float) -> str: segs = _segs(text) budget = int(sum(len(s) for s in segs) * ratio) kept: list[str] = [] c = 0 for s in reversed(segs): if c >= budget: break kept.append(s) c += len(s) return "".join(reversed(kept)) def random_keep(text: str, ratio: float, seed: int) -> str: segs = _segs(text) idx = list(range(len(segs))) random.Random(seed).shuffle(idx) budget = int(sum(len(s) for s in segs) * ratio) kept: set[int] = set() c = 0 for i in idx: if c >= budget: break kept.add(i) c += len(segs[i]) return "".join(segs[i] for i in sorted(kept)) # --- Part C: deterministic needle retention (zh / ja / ko) --------------------- # Each needle carries a distinctive verbatim KEY that must survive. Distractors # are generated (deterministic, distinct, topic-unrelated to the query) so the # haystack is large enough to FORCE real compression -- the needle only survives # under TextCrusher because it is query-relevant, not because of passthrough. _NEEDLES = { "zh": { "query": "认证令牌缓存淘汰策略", "key": "最近最少使用淘汰", "needle": "认证令牌的缓存采用最近最少使用淘汰算法来管理过期条目。", "distractor": lambda i: f"第{i}号监控服务器的日志显示子系统{i}今天运行平稳没有出现异常。", }, "ja": { "query": "認証トークン キャッシュ 破棄 アルゴリズム", "key": "最長未使用", "needle": "認証トークンのキャッシュは最長未使用アルゴリズムで管理される。", "distractor": lambda i: ( f"{i}番目の監視サーバーのログには{i}番のサブシステムが本日も正常に稼働したと記録されている。" ), }, "ko": { "query": "인증 토큰 캐시 제거 알고리즘", "key": "최근 최소 사용", "needle": "인증 토큰 캐시는 최근 최소 사용 알고리즘으로 관리된다.", "distractor": lambda i: ( f"{i}번 모니터링 서버의 로그에는 {i}번 하위 시스템이 오늘도 정상 작동했다고 기록되어 있다." ), }, } def _haystack(spec: dict, n_distract: int = 24) -> str: half = n_distract // 2 before = [spec["distractor"](i) for i in range(half)] after = [spec["distractor"](i) for i in range(half, n_distract)] # needle in the MIDDLE so keep-recent (truncate) reliably misses it. return "".join(before + [spec["needle"]] + after) def retention_synthetic(lang: str, ratio: float = 0.3, seed: int = 0) -> dict[str, bool]: spec = _NEEDLES[lang] hay = _haystack(spec) key = norm(spec["key"]) tc = TextCrusher() out_tc = tc.compress(hay, spec["query"], ratio).compressed return { "text_crusher": key in norm(out_tc), "truncate": key in norm(truncate_keep_last(hay, ratio)), "random": key in norm(random_keep(hay, ratio, seed)), } def eval_synthetic(ratio: float = 0.3) -> None: print(f"\n=== Part C: synthetic needle retention (zh/ja/ko, target_ratio={ratio}) ===") print(f" {'lang':5} {'text_crusher':>13} {'truncate':>9} {'random':>7}") for lang in ("zh", "ja", "ko"): r = retention_synthetic(lang, ratio) print( f" {lang:5} {str(r['text_crusher']):>13} {str(r['truncate']):>9} {str(r['random']):>7}" ) print(" (needle must survive under TextCrusher; baselines are the contrast)") # --- Part B: real CJK transcript fidelity (CJK-aware salient) ------------------ # ASCII salient (identifiers/numbers/errors) STILL matters in CJK coding context. _SALIENT_ASCII = re.compile( r"\b(?:error|exception|fail(?:ed|ure)?|warning|traceback|assert|todo|fixme)\b" r"|\b[A-Z]{2,}\b|\b[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*\b|\b\d+\b" ) def _cjk_hapax(text: str) -> set[str]: # distinctive CJK content = char-bigrams occurring exactly once (rare = must-keep) grams: dict[str, int] = {} for run in _CJK_RUN.findall(text): for i in range(len(run) - 1): g = run[i : i + 2] grams[g] = grams.get(g, 0) + 1 return {g for g, c in grams.items() if c == 1} def salient_set(text: str) -> set[str]: return set(_SALIENT_ASCII.findall(text)) | _cjk_hapax(text) def _block_texts(jsonl_path: str, min_chars: int, limit: int) -> list[str]: import json out: list[str] = [] with open(jsonl_path, encoding="utf-8") as fh: for line in fh: try: o = json.loads(line) except json.JSONDecodeError: continue c = (o.get("message") or {}).get("content") parts = ( [c] if isinstance(c, str) else [ p["text"] for p in c if isinstance(p, dict) and isinstance(p.get("text"), str) ] if isinstance(c, list) else [] ) for t in parts: if len(t) >= min_chars and _CJK_RUN.search(t): # CJK-bearing only out.append(anon(t)) if len(out) >= limit: break return out[:limit] def eval_transcript( jsonl_path: str, ratio: float = 0.4, min_chars: int = 600, limit: int = 40 ) -> None: blocks = _block_texts(jsonl_path, min_chars, limit) if not blocks: print( f"\n=== Part B: no CJK blocks >= {min_chars} chars in {os.path.basename(jsonl_path)} ===" ) return tc = TextCrusher() ratios: list[float] = [] times: list[float] = [] retentions: list[float] = [] for b in blocks: sal_before = salient_set(b) t0 = time.perf_counter() out = tc.compress(b, "", ratio).compressed times.append((time.perf_counter() - t0) * 1000) retentions.append(len(sal_before & salient_set(out)) / max(1, len(sal_before))) ratios.append(len(out) / max(1, len(b))) n = len(blocks) print( f"\n=== Part B: real CJK transcript fidelity (n={n}, anonymized, target_ratio={ratio}) ===" ) print(f" mean char-ratio kept: {sum(ratios) / n:.2f}") print(f" mean speed: {sum(times) / n:.1f} ms/block") print(f" CJK-aware salient retention: {sum(retentions) / n:.1%}") # --- Part A: optional natural-data retention (multi-wiki-qa zh/ja/ko) ---------- # Schema verified: row = {id, title, context, question, answers:{text:[...]}}. # Answers are guaranteed verbatim substrings of the (long) context; CC-BY-NC-SA. def eval_multiwiki( langs=("zh-cn", "ja", "ko"), n: int = 80, ratio: float = 0.3, seed: int = 0 ) -> None: try: from datasets import load_dataset except ImportError: print( "\n=== Part A: `datasets` not installed; skipping (pip install headroom-ai[evals]) ===" ) return tc = TextCrusher() print(f"\n=== Part A: multi-wiki-qa answer-retention (n={n}/lang, target_ratio={ratio}) ===") print(f" {'lang':6} {'text_crusher':>13} {'truncate':>9} {'random':>7}") for lang in langs: try: ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n * 2}]") except Exception as e: # noqa: BLE001 -- optional path, fail-open print(f" {lang}: load failed ({e}); skipping") continue ex = [] for r in ds: ans = r.get("answers") a = ans["text"][0] if isinstance(ans, dict) and ans.get("text") else None if r.get("context") and r.get("question") and a: ex.append((r["context"], r["question"], a)) random.Random(seed).shuffle(ex) ex = ex[:n] hit = {"text_crusher": 0, "truncate": 0, "random": 0} for ctx, q, ans in ex: a = norm(ans) hit["text_crusher"] += a in norm(tc.compress(ctx, q, ratio).compressed) hit["truncate"] += a in norm(truncate_keep_last(ctx, ratio)) hit["random"] += a in norm(random_keep(ctx, ratio, seed)) m = max(1, len(ex)) print( f" {lang:6} {hit['text_crusher'] / m:>12.0%} {hit['truncate'] / m:>9.0%} {hit['random'] / m:>7.0%}" ) if __name__ == "__main__": eval_synthetic() tx = sys.argv[1] if len(sys.argv) > 1 else None if tx is None: found = glob.glob(os.path.expanduser("~/.claude/projects/*headroom*/*.jsonl")) tx = max(found, key=os.path.getsize) if found else None if tx and os.path.exists(tx): eval_transcript(tx) else: print("\nno transcript jsonl found; skipping Part B") eval_multiwiki() - benchmarks/index_proof_table.py (5027 bytes)
"""Regenerate the "Proof" savings table published on the docs landing page. WHY THIS EXISTS --------------- docs/content/docs/index.mdx published four precise before/after token counts with no reproducible source. The nearest harness, ``real_world_agent_benchmark.py``, seeded nothing, so its corpus differed on every run and the published figures could not be reproduced by anyone, including us. A number on the front page of the docs that nobody can regenerate is a liability, not evidence. This script fixes the reproducibility half. It seeds the generators, builds the same scenarios, and measures tokens through the real tokenizer and the real ``compress()`` path. No network, no API key, no model call: the table is a statement about token counts, and token counts are computable locally. uv run python benchmarks/index_proof_table.py Deterministic: same seed in, same numbers out, on any machine. """ from __future__ import annotations import argparse import json from real_world_agent_benchmark import ( # noqa: E402 DEFAULT_SEED, create_codebase_exploration_scenario, create_issue_triage_scenario, create_sre_debugging_scenario, generate_github_code_search, seed_everything, ) from headroom import CompressConfig, compress from headroom.providers.openai_compatible import OpenAICompatibleTokenCounter MODEL = "gpt-5.6" def _tool_messages(tools: list[dict]) -> list[dict]: """The tool payloads as the proxy would actually see them on the wire.""" return [ { "role": "tool", "tool_call_id": f"call_{i}", "content": json.dumps(t["result"]), } for i, t in enumerate(tools) ] def measure(label: str, tools: list[dict], tok, config: CompressConfig) -> dict: msgs = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Analyse the tool output and answer."}, *_tool_messages(tools), ] before = sum(tok.count_text(m["content"]) for m in msgs if m["role"] == "tool") result = compress(msgs, model=MODEL, config=config) after = sum( tok.count_text(m["content"]) for m in result.messages if m.get("role") == "tool" and isinstance(m.get("content"), str) ) saved = before - after return { "scenario": label, "before": before, "after": after, "saved": saved, "savings_pct": (saved / before * 100) if before else 0.0, "transforms": sorted(set(result.transforms_applied)), } def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--seed", type=int, default=DEFAULT_SEED) args = ap.parse_args() tok = OpenAICompatibleTokenCounter(model=MODEL) # One configuration: the default, i.e. what a coding agent actually gets. # # A second `protect_recent=0` run used to be printed here, and came out # identical in every row and in the total (#4732). Not a coincidence: the # router applies `protect_recent` only to content detected as SOURCE_CODE # ("Don't compress recent CODE"). These scenarios are JSON tool payloads -- # JSON_ARRAY even under the native detector -- so the guard never fires and # the second table carried no information. Printing it again would only # relabel a duplicate, so it is stated here once instead. cfg = CompressConfig() # Built in a fixed order: every generator draws from the same global RNG, # so re-ordering these lines changes every number below. print(f"seed={args.seed} model={MODEL} tokenizer={type(tok._tokenizer).__name__}") seed_everything(args.seed) rows = [ measure( "Code search (100 results)", [generate_github_code_search("JWT authentication middleware", num_results=100)], tok, cfg, ), measure("SRE incident debugging", create_sre_debugging_scenario().tools, tok, cfg), measure("Codebase exploration", create_codebase_exploration_scenario().tools, tok, cfg), measure("GitHub issue triage", create_issue_triage_scenario().tools, tok, cfg), ] print("\n=== default (protect_recent=4) ===") print(f"{'Scenario':<30} {'Before':>10} {'After':>10} {'Savings':>9}") print("-" * 62) for r in rows: print(f"{r['scenario']:<30} {r['before']:>10,} {r['after']:>10,} {r['savings_pct']:>8.0f}%") tb = sum(r["before"] for r in rows) ta = sum(r["after"] for r in rows) print("-" * 62) print(f"{'TOTAL':<30} {tb:>10,} {ta:>10,} {(tb - ta) / tb * 100:>8.0f}%") print( "\n\nMarkdown for docs/content/docs/index.mdx " "(default config, which must be stated on the page):\n" ) print("| Scenario | Before | After | Savings |") print("|---|---|---|---|") for r in rows: print( f"| {r['scenario']} | {r['before']:,} | {r['after']:,} | **{r['savings_pct']:.0f}%** |" ) return 0 if __name__ == "__main__": raise SystemExit(main()) - benchmarks/memory_fts_reindex_benchmark.py (3018 bytes)
#!/usr/bin/env python3 """Measure FTS5 reindex throughput: per-record commits vs batched transactions.""" from __future__ import annotations import argparse import asyncio import json import statistics import sys import tempfile import time from pathlib import Path # Run as a script sys.path[0] is benchmarks/, so an editable install of another # checkout would win and we would silently measure the wrong tree. sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) from headroom.cli.memory import _REINDEX_PAGE_SIZE from headroom.memory.adapters.fts5 import FTS5TextIndex from headroom.memory.models import Memory def make_memories(count: int, content_words: int) -> list[Memory]: body = " ".join(f"token{n}" for n in range(content_words)) return [ Memory( id=f"mem-{index:06d}", content=f"{body} unique{index}", user_id="bench-user", session_id=f"session-{index % 7}", ) for index in range(count) ] def benchmark(count: int, content_words: int, page_size: int, runs: int) -> dict[str, object]: memories = make_memories(count, content_words) per_record: list[float] = [] batched: list[float] = [] for _ in range(runs): # Fresh temporary database per mode per run so neither inherits the # other's page cache or index state. with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp: fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db")) start = time.perf_counter() for memory in memories: asyncio.run(fts.index_memory(memory)) per_record.append((time.perf_counter() - start) * 1000) with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp: fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db")) start = time.perf_counter() for offset in range(0, len(memories), page_size): asyncio.run(fts.index_batch_memories(memories[offset : offset + page_size])) batched.append((time.perf_counter() - start) * 1000) return { "records": count, "content_words": content_words, "page_size": page_size, "runs": runs, "per_record_ms": statistics.median(per_record), "batched_ms": statistics.median(batched), "speedup": round(statistics.median(per_record) / statistics.median(batched), 1), } def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--records", type=int, nargs="+", default=[1000]) parser.add_argument("--content-words", type=int, default=40) parser.add_argument("--page-size", type=int, default=_REINDEX_PAGE_SIZE) parser.add_argument("--runs", type=int, default=3) args = parser.parse_args() results = [ benchmark(count, args.content_words, args.page_size, args.runs) for count in args.records ] print(json.dumps(results, indent=2)) if __name__ == "__main__": main() - benchmarks/prefix_cache_benchmark.py (27424 bytes)
#!/usr/bin/env python3 """ Prefix Cache Strategy Benchmark — Real API Calls Sends a 25-turn conversation through 4 different caching strategies and measures actual cache_read_input_tokens vs cache_creation_input_tokens from the Anthropic API. Strategies: 1. Baseline — no Headroom, no markers 2. Headroom compression — full pipeline, CompressionCache keeps bytes stable 3. Headroom + prefix freeze — pipeline skips frozen (already-cached) messages 4. Headroom + explicit markers — pipeline + 4 cache_control breakpoints Usage: # Load API key from .env and run source .env && python benchmarks/prefix_cache_benchmark.py # Quick test with fewer turns source .env && python benchmarks/prefix_cache_benchmark.py --turns 5 # With specific model source .env && python benchmarks/prefix_cache_benchmark.py --model claude-sonnet-4-6 Estimated cost: ~$0.50-1.00 total across all strategies. """ from __future__ import annotations import argparse import copy import json import os import sys import time from dataclasses import dataclass, field from typing import Any import httpx # --------------------------------------------------------------------------- # Pricing (per token) # --------------------------------------------------------------------------- PRICING = { "claude-sonnet-4-6": { "input": 3.00 / 1_000_000, "output": 15.00 / 1_000_000, "cache_read": 0.30 / 1_000_000, "cache_write": 3.75 / 1_000_000, }, "claude-haiku-4-5-20251001": { "input": 0.80 / 1_000_000, "output": 4.00 / 1_000_000, "cache_read": 0.08 / 1_000_000, "cache_write": 1.00 / 1_000_000, }, } # --------------------------------------------------------------------------- # Data classes # --------------------------------------------------------------------------- @dataclass class TurnMetrics: turn: int cache_read_tokens: int = 0 cache_creation_tokens: int = 0 input_tokens: int = 0 output_tokens: int = 0 cost_usd: float = 0.0 @dataclass class StrategyResult: name: str turns: list[TurnMetrics] = field(default_factory=list) @property def total_input(self) -> int: return sum( t.input_tokens + t.cache_read_tokens + t.cache_creation_tokens for t in self.turns ) @property def total_cache_read(self) -> int: return sum(t.cache_read_tokens for t in self.turns) @property def total_cache_write(self) -> int: return sum(t.cache_creation_tokens for t in self.turns) @property def total_output(self) -> int: return sum(t.output_tokens for t in self.turns) @property def total_cost(self) -> float: return sum(t.cost_usd for t in self.turns) @property def cache_hit_rate(self) -> float: total = ( self.total_cache_read + self.total_cache_write + sum(t.input_tokens for t in self.turns) ) return (self.total_cache_read / total * 100) if total > 0 else 0.0 # --------------------------------------------------------------------------- # Conversation builder # --------------------------------------------------------------------------- SYSTEM_PROMPT = """You are an expert software engineering assistant. You help users debug code, analyze logs, query databases, and search codebases. You have access to several tools. When analyzing data, be thorough but concise. Focus on anomalies, errors, and actionable insights. Always explain your reasoning step by step. Important guidelines: - When you see error patterns, highlight them immediately - For database queries, suggest optimizations if the result set is large - For code analysis, focus on potential bugs and security issues - Always provide actionable next steps You are working in a large Python monorepo with FastAPI services, PostgreSQL databases, and Redis caching. The codebase uses pytest for testing and has CI/CD via GitHub Actions.""" TOOLS = [ { "name": "search_codebase", "description": "Search the codebase for patterns, function definitions, or references.", "input_schema": { "type": "object", "properties": { "query": {"type": "string", "description": "Search pattern or keyword"}, "file_pattern": {"type": "string", "description": "Glob pattern for files"}, }, "required": ["query"], }, }, { "name": "read_file", "description": "Read the contents of a file.", "input_schema": { "type": "object", "properties": { "file_path": {"type": "string", "description": "Path to the file"}, "offset": {"type": "integer", "description": "Line offset to start from"}, "limit": {"type": "integer", "description": "Number of lines to read"}, }, "required": ["file_path"], }, }, { "name": "query_database", "description": "Execute a read-only SQL query against the application database.", "input_schema": { "type": "object", "properties": { "query": {"type": "string", "description": "SQL SELECT query"}, "database": {"type": "string", "description": "Database name"}, }, "required": ["query"], }, }, { "name": "search_logs", "description": "Search application logs for patterns within a time range.", "input_schema": { "type": "object", "properties": { "pattern": {"type": "string", "description": "Log pattern to search"}, "service": {"type": "string", "description": "Service name"}, "hours": {"type": "integer", "description": "Hours to look back"}, }, "required": ["pattern"], }, }, ] USER_QUERIES = [ "Can you search for all usages of the `authenticate_user` function?", "Read the file src/auth/middleware.py so I can understand the auth flow.", "Query the database for failed login attempts in the last hour: SELECT user_id, attempt_time, error_code FROM auth_logs WHERE status='failed' AND attempt_time > NOW() - INTERVAL '1 hour' ORDER BY attempt_time DESC LIMIT 50", "Search the logs for 'ConnectionRefused' errors in the auth-service from the past 2 hours.", "Read src/auth/token_validator.py — I think the bug might be there.", "Search for all files that import from `auth.middleware`.", "Query for users who had more than 5 failed attempts: SELECT user_id, COUNT(*) as fails FROM auth_logs WHERE status='failed' AND attempt_time > NOW() - INTERVAL '24 hours' GROUP BY user_id HAVING COUNT(*) > 5", "Search logs for 'JWT expired' in auth-service.", "Read the test file tests/test_auth.py to see what's covered.", "Search for `rate_limit` in the codebase.", "Read src/config/settings.py to check the rate limit configuration.", "Query the metrics table: SELECT endpoint, avg_latency_ms, p99_latency_ms, error_rate FROM api_metrics WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY error_rate DESC", "Search logs for any 5xx errors across all services.", "Read src/api/routes.py to check the endpoint definitions.", "Search for usages of the Redis cache client.", "Read src/cache/redis_client.py for the connection pooling setup.", "Query cache hit rates: SELECT cache_key_prefix, hit_count, miss_count, hit_count::float/(hit_count+miss_count) as hit_rate FROM cache_stats WHERE period='hourly' ORDER BY miss_count DESC LIMIT 20", "Search logs for 'cache eviction' warnings.", "Read the Dockerfile to check the base image version.", "Search for any TODO or FIXME comments in the auth module.", "Read .github/workflows/ci.yml for the CI pipeline config.", "Query deployment history: SELECT version, deployed_at, deployed_by, status FROM deployments WHERE service='auth-service' ORDER BY deployed_at DESC LIMIT 10", "Search for error handling patterns — look for bare `except:` blocks.", "Read src/auth/oauth.py for the OAuth integration.", "Search logs for memory usage spikes in the last 4 hours.", ] # Fake tool responses (JSON data that would come from tools) TOOL_RESPONSES = { "search_codebase": lambda q: json.dumps( [ { "file": f"src/auth/{f}.py", "line": 10 + i * 5, "match": f"def authenticate_user(request): # {q}", } for i, f in enumerate(["middleware", "token_validator", "oauth", "session", "utils"]) ] + [ { "file": f"tests/test_{f}.py", "line": 20 + i * 3, "match": f"from auth.middleware import {q.split()[0] if q.split() else 'auth'}", } for i, f in enumerate(["auth", "api", "cache"]) ] ), "read_file": lambda q: ( "# File contents (simulated)\nimport logging\nfrom typing import Optional\n\nlogger = logging.getLogger(__name__)\n\n" + "\n".join( [ f"def function_{i}(arg: str) -> Optional[dict]:\n \"\"\"Process {q}.\"\"\"\n result = {{}}\n for key in ['id', 'name', 'status']:\n result[key] = f'value_{{key}}_{{arg}}'\n logger.info(f'Processed {{arg}}')\n return result\n" for i in range(8) ] ) ), "query_database": lambda q: json.dumps( [ { "user_id": f"user_{i:04d}", "attempt_time": f"2025-01-15T10:{i:02d}:00Z", "error_code": [ "INVALID_PASSWORD", "EXPIRED_TOKEN", "RATE_LIMITED", "ACCOUNT_LOCKED", ][i % 4], "status": "failed", } for i in range(15) ] ), "search_logs": lambda q: "\n".join( [ f"2025-01-15T10:{i:02d}:{j:02d}Z [ERROR] auth-service: {q} - connection to db-primary:5432 refused (attempt {j + 1}/3)" for i in range(5) for j in range(3) ] ), } def build_turn_messages( turn_idx: int, history: list[dict[str, Any]], ) -> tuple[list[dict[str, Any]], str]: """Build messages for a specific turn, return (messages, user_query).""" query = USER_QUERIES[turn_idx % len(USER_QUERIES)] messages = list(history) + [{"role": "user", "content": query}] return messages, query # --------------------------------------------------------------------------- # API call helper # --------------------------------------------------------------------------- def call_anthropic( api_key: str, model: str, messages: list[dict[str, Any]], tools: list[dict] | None = None, max_tokens: int = 100, ) -> dict[str, Any]: """Make a real Anthropic API call, return the full response JSON.""" # Separate system from messages (Anthropic format) system_content = None api_messages = [] for msg in messages: if msg["role"] == "system": system_content = msg["content"] else: api_messages.append(msg) body: dict[str, Any] = { "model": model, "max_tokens": max_tokens, "messages": api_messages, } if system_content: body["system"] = system_content if tools: body["tools"] = tools headers = { "x-api-key": api_key, "anthropic-version": "2023-06-01", "content-type": "application/json", } with httpx.Client(timeout=60) as client: resp = client.post( "https://api.anthropic.com/v1/messages", json=body, headers=headers, ) resp.raise_for_status() return resp.json() def extract_metrics(resp: dict, turn: int, pricing: dict) -> TurnMetrics: """Extract cache metrics from Anthropic response.""" usage = resp.get("usage", {}) cr = usage.get("cache_read_input_tokens", 0) cw = usage.get("cache_creation_input_tokens", 0) inp = usage.get("input_tokens", 0) out = usage.get("output_tokens", 0) cost = ( cr * pricing["cache_read"] + cw * pricing["cache_write"] + inp * pricing["input"] + out * pricing["output"] ) return TurnMetrics( turn=turn, cache_read_tokens=cr, cache_creation_tokens=cw, input_tokens=inp, output_tokens=out, cost_usd=cost, ) def extract_assistant_content(resp: dict) -> dict[str, Any]: """Convert Anthropic response to a message dict for conversation history.""" content = resp.get("content", []) # Check for tool use has_tool_use = any(b.get("type") == "tool_use" for b in content if isinstance(b, dict)) if has_tool_use: return {"role": "assistant", "content": content} else: # Extract text text = "" for block in content: if isinstance(block, dict) and block.get("type") == "text": text += block.get("text", "") return {"role": "assistant", "content": text} def make_tool_result(assistant_msg: dict) -> list[dict[str, Any]]: """Generate fake tool results for any tool_use blocks in the assistant message.""" results = [] content = assistant_msg.get("content", []) if not isinstance(content, list): return results for block in content: if isinstance(block, dict) and block.get("type") == "tool_use": tool_name = block.get("name", "search_codebase") tool_id = block.get("id", "") query = json.dumps(block.get("input", {})) gen = TOOL_RESPONSES.get(tool_name, TOOL_RESPONSES["search_codebase"]) fake_output = gen(query) results.append( { "role": "user", "content": [ { "type": "tool_result", "tool_use_id": tool_id, "content": fake_output, } ], } ) return results # --------------------------------------------------------------------------- # Strategy: inject explicit cache_control markers # --------------------------------------------------------------------------- def inject_cache_markers( system_content: str | None, api_messages: list[dict[str, Any]], ) -> tuple[str | list | None, list[dict[str, Any]]]: """Inject up to 4 cache_control breakpoints at strategic positions. Marker 1: End of system prompt Marker 2: ~1/3 through messages Marker 3: ~2/3 through messages Marker 4: Last message """ # Marker 1: system prompt if system_content and isinstance(system_content, str): system_content = [ {"type": "text", - benchmarks/proxy_mode_benchmark.py (10028 bytes)
#!/usr/bin/env python3 """Local benchmark for proxy run modes (no API calls). Compares: - baseline: no compression - token mode: prioritize compression - cache mode: preserve prior-turn prefix stability Includes an optional real-test harness printout for Claude Code, but does not invoke external APIs unless the user does so manually. """ from __future__ import annotations import argparse import copy import json import logging from dataclasses import dataclass from typing import Any from headroom.cache.compression_cache import CompressionCache from headroom.cache.prefix_tracker import PrefixCacheTracker from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin from headroom.proxy.models import ProxyConfig from headroom.proxy.modes import PROXY_MODE_CACHE, PROXY_MODE_TOKEN from headroom.proxy.server import HeadroomProxy from headroom.tokenizers import get_tokenizer from headroom.utils import extract_user_query MODEL = "claude-sonnet-4-6" @dataclass class ModeBenchmarkResult: mode: str total_original_tokens: int = 0 total_sent_tokens: int = 0 total_tokens_saved: int = 0 total_cache_read_tokens: int = 0 total_cache_write_tokens: int = 0 total_uncached_tokens: int = 0 @property def compression_pct(self) -> float: if self.total_original_tokens <= 0: return 0.0 return self.total_tokens_saved / self.total_original_tokens * 100.0 @property def cache_hit_pct(self) -> float: total = ( self.total_cache_read_tokens + self.total_cache_write_tokens + self.total_uncached_tokens ) if total <= 0: return 0.0 return self.total_cache_read_tokens / total * 100.0 def _build_tool_result(turn: int, rows: int = 240) -> str: payload = [] for i in range(rows): payload.append( { "id": f"{turn:02d}-{i:04d}", "status": "ok" if i % 37 else "warning", "service": "auth-api" if i % 2 else "gateway", "latency_ms": 100 + (i % 13), "hint": "retry with exponential backoff" if i % 89 == 0 else "none", } ) return json.dumps(payload) def _build_conversation(turn: int) -> list[dict[str, Any]]: messages: list[dict[str, Any]] = [] for t in range(1, turn): messages.extend( [ { "role": "user", "content": f"Analyze tool output turn {t} and summarize anomalies.", }, { "role": "user", "content": [ { "type": "tool_result", "tool_use_id": f"tool-{t}", "content": _build_tool_result(t), } ], }, {"role": "assistant", "content": f"Turn {t} acknowledged."}, ] ) # Current turn: user request + fresh tool output, no assistant response yet. messages.extend( [ { "role": "user", "content": f"Analyze tool output turn {turn} and summarize anomalies.", }, { "role": "user", "content": [ { "type": "tool_result", "tool_use_id": f"tool-{turn}", "content": _build_tool_result(turn), } ], }, ] ) return messages def _common_prefix_tokens( prev: list[dict[str, Any]], curr: list[dict[str, Any]], tokenizer: Any ) -> tuple[int, list[int]]: common = 0 counts: list[int] = [] for msg in curr: counts.append(tokenizer.count_message(msg)) for i, (a, b) in enumerate(zip(prev, curr)): if a != b: break common += counts[i] return common, counts def _make_proxy(mode: str) -> HeadroomProxy: cfg = ProxyConfig( mode=mode, optimize=True, image_optimize=False, smart_routing=False, code_aware_enabled=False, read_lifecycle=False, cache_enabled=False, rate_limit_enabled=False, cost_tracking_enabled=False, log_requests=False, ccr_inject_tool=False, ccr_handle_responses=False, ccr_context_tracking=False, ) return HeadroomProxy(cfg) def _simulate_mode(turns: int, mode: str) -> ModeBenchmarkResult: tokenizer = get_tokenizer(MODEL) result = ModeBenchmarkResult(mode=mode) if mode == "baseline": prev_forwarded: list[dict[str, Any]] = [] for turn in range(1, turns + 1): messages = _build_conversation(turn) before = tokenizer.count_messages(messages) common, counts = _common_prefix_tokens(prev_forwarded, messages, tokenizer) uncached = max(0, before - common) result.total_original_tokens += before result.total_sent_tokens += before result.total_cache_read_tokens += common result.total_cache_write_tokens += 0 result.total_uncached_tokens += uncached prev_forwarded = copy.deepcopy(messages) return result proxy = _make_proxy(mode) prefix_tracker = PrefixCacheTracker("anthropic") comp_cache = CompressionCache() prev_forwarded = [] for turn in range(1, turns + 1): messages = _build_conversation(turn) before = tokenizer.count_messages(messages) frozen = prefix_tracker.get_frozen_message_count() if mode == PROXY_MODE_CACHE: frozen = AnthropicHandlerMixin._strict_previous_turn_frozen_count(messages, frozen) working = messages if mode == PROXY_MODE_TOKEN: working = comp_cache.apply_cached(messages) frozen = min(frozen, comp_cache.compute_frozen_count(messages)) context_limit = proxy.anthropic_provider.get_context_limit(MODEL) pipeline_result = proxy.anthropic_pipeline.apply( messages=working, model=MODEL, model_limit=context_limit, context=extract_user_query(working), frozen_message_count=frozen, ) forwarded = pipeline_result.messages if mode == PROXY_MODE_TOKEN: comp_cache.update_from_result(messages, forwarded) if mode == PROXY_MODE_CACHE: forwarded, _ = AnthropicHandlerMixin._restore_frozen_prefix( messages, forwarded, frozen_message_count=frozen ) after = tokenizer.count_messages(forwarded) common, msg_counts = _common_prefix_tokens(prev_forwarded, forwarded, tokenizer) uncached = max(0, after - common) result.total_original_tokens += before result.total_sent_tokens += after result.total_tokens_saved += max(0, before - after) result.total_cache_read_tokens += common result.total_uncached_tokens += uncached prefix_tracker.update_from_response( cache_read_tokens=common, cache_write_tokens=uncached, messages=forwarded, message_token_counts=msg_counts, ) result.total_cache_write_tokens += uncached prev_forwarded = copy.deepcopy(forwarded) return result def run_local_benchmark(turns: int = 12) -> dict[str, ModeBenchmarkResult]: return { "baseline": _simulate_mode(turns, "baseline"), PROXY_MODE_TOKEN: _simulate_mode(turns, PROXY_MODE_TOKEN), PROXY_MODE_CACHE: _simulate_mode(turns, PROXY_MODE_CACHE), } def _print_results(results: dict[str, ModeBenchmarkResult]) -> None: print( "\nMode benchmark (higher compression + higher cache_hit is better for total cost):\n" "mode orig_tok sent_tok saved_tok compression cache_hit uncached_tok" ) for key in ("baseline", PROXY_MODE_TOKEN, PROXY_MODE_CACHE): r = results[key] print( f"{r.mode:<9} {r.total_original_tokens:>9,} {r.total_sent_tokens:>10,} " f"{r.total_tokens_saved:>10,} {r.compression_pct:>10.1f}% " f"{r.cache_hit_pct:>9.1f}% {r.total_uncached_tokens:>12,}" ) token = results[PROXY_MODE_TOKEN] cache = results[PROXY_MODE_CACHE] print("\nDelta (cache - token):") print(f" cache_hit_pct: {cache.cache_hit_pct - token.cache_hit_pct:+.1f}%") print(f" compression_pct: {cache.compression_pct - token.compression_pct:+.1f}%") print(f" uncached_tokens: {cache.total_uncached_tokens - token.total_uncached_tokens:+,}") def _print_real_harness() -> None: print("\nReal test harness (manual; optional, not executed by this benchmark):") print(" 1) Start proxy in cache mode: HEADROOM_MODE=cache headroom proxy --port 8787") print(" 2) Start proxy in token mode: HEADROOM_MODE=token headroom proxy --port 8787") print(" 3) Run Claude Code against each:") print(" ANTHROPIC_BASE_URL=http://localhost:8787 claude") print(" 4) Compare /stats prefix_cache and compression sections per run.") def main() -> None: logging.getLogger("headroom").setLevel(logging.WARNING) logging.getLogger("sentence_transformers").setLevel(logging.WARNING) logging.getLogger("huggingface_hub").setLevel(logging.WARNING) parser = argparse.ArgumentParser(description="Local benchmark for proxy token/cache modes") parser.add_argument("--turns", type=int, default=12, help="Conversation turns to simulate") parser.add_argument( "--show-real-harness", action="store_true", help="Print manual steps for optional Claude Code real testing", ) args = parser.parse_args() results = run_local_benchmark(turns=args.turns) _print_results(results) if args.show_real_harness: _print_real_harness() if __name__ == "__main__": main() - Cargo.toml (6840 bytes)
[workspace] resolver = "2" members = [ "crates/headroom-core", "crates/headroom-proxy", "crates/headroom-simulators", "crates/headroom-py", "crates/headroom-parity", ] # headroom-py is a Python extension module — it must be built via maturin, not # plain cargo (the "extension-module" feature tells pyo3 not to link libpython, # which is required for `import` to work). `cargo build --workspace` without # explicit members skips it; `cargo test --workspace` still runs its tests # because pyo3 can dynamically link here for the cdylib used by tests. default-members = [ "crates/headroom-core", "crates/headroom-proxy", "crates/headroom-simulators", "crates/headroom-parity", ] [workspace.package] edition = "2021" rust-version = "1.80" license = "Apache-2.0" repository = "https://github.com/headroomlabs-ai/headroom" authors = ["Headroom Maintainers"] [workspace.dependencies] serde = { version = "1", features = ["derive"] } # `preserve_order` makes `serde_json::Value::Object` use IndexMap so JSON # parse order is preserved through Value→string→Value round-trips. The # smart_crusher port relies on this to match Python's `str(dict)` output, # which preserves insertion order; otherwise BTreeMap's sorted-key default # would diverge from Python on every multi-key object. # # `arbitrary_precision` keeps the literal numeric token from the source # JSON intact: `Value::Number` becomes a wrapper around the original # digit string, so `1.0` does NOT collapse to `1`, and `12345678901234567` # does NOT lose precision through f64. Required by Realignment invariant # I1 (byte-faithful passthrough on unmutated bytes; see REALIGNMENT/02- # architecture.md §2.2) and PR-A4 (see REALIGNMENT/03-phase-A-lockdown.md). # # `raw_value` exposes `serde_json::value::RawValue`, the unparsed JSON # fragment type. Phase B PR-B2 uses this to forward unmodified # `messages[*]` entries as exact byte copies — the parser captures the # original byte slice, so byte-for-byte round-trips work even with # whitespace, key order, or escape preferences the producer chose. # Enabled here in Phase A so PR-B2 can land as a pure consumer change. serde_json = { version = "1", features = ["preserve_order", "arbitrary_precision", "raw_value"] } bytes = "1" thiserror = "2" # `log` compat: when no tracing subscriber is active (the case inside the # headroom-py cdylib), events are re-emitted as `log` records so pyo3-log # can forward them to Python's logging. No effect on binaries that install # a real subscriber. tracing = { version = "0.1", features = ["log"] } anyhow = "1" clap = { version = "4", features = ["derive"] } tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] } axum = "0.8" tower = "0.5" reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } pyo3 = { version = "0.30", features = ["abi3-py310"] } # Forwards Rust `log` records (incl. tracing events via the `log` compat # feature above) into Python's `logging` inside the _core extension module. pyo3-log = "0.13" # Phase D PR-D1: AWS SigV4 signing for native Bedrock InvokeModel route. # `aws-sigv4` provides the canonical-request + signing-key implementation; # `aws-config` resolves credentials from the standard provider chain # (env vars, profiles, IMDS, ECS task role, etc); `aws-credential-types` # exposes `Credentials` so the signer accepts whatever the chain returned. aws-sigv4 = { version = "1", default-features = false, features = ["sign-http", "http1"] } aws-config = { version = "1", default-features = false, features = ["behavior-version-latest", "rustls", "rt-tokio", "sso"] } aws-credential-types = { version = "1", default-features = false } # `Identity` lives in aws-smithy-runtime-api; the SigV4 builder # accepts `&Identity`. Pinning the version explicitly avoids a # silent semver bump from the transitive dep tree. aws-smithy-runtime-api = { version = "1", default-features = false, features = ["client"] } # PR-D4: Vertex publisher path uses GCP Application Default Credentials # (ADC) → bearer token for the `Authorization: Bearer <token>` header. # `gcp_auth` resolves the chain (gcloud user creds, GCE/GKE metadata # server, service-account JSON, workload-identity federation) without # us baking provider-specific knowledge in. The token source is wrapped # in a `TokenSource` trait so tests inject a static-token mock. gcp_auth = "0.12" # ── Release profile — wheel size optimization ─────────────────────── # # PyPI imposes a 10 GB cumulative storage limit per project. We hit it # at version 0.21.36 (191 versions × ~213 MB/release = 10.00 GB # exactly). Recent wheels were ~16-18 MB each, of which ~6.4 MB was # pure debug metadata (`.strtab` + `.symtab` ELF sections; uncovered # by post-mortem inspection of an actual production wheel). # # This profile shrinks each Linux wheel from ~18 MB → ~10-11 MB by: # * Stripping symbol/string tables (~6.4 MB direct savings) # * Link-time optimization across crate boundaries (~5-10% .text # savings via dead-code elim across the workspace) # * Single codegen unit (better inlining + dead-code elim, at the # cost of slightly slower release builds) # # We deliberately do NOT set ``panic = "abort"``. The proxy is a # long-lived async process — a single misbehaving request triggering # panic-abort would terminate the whole proxy and disconnect every # concurrent client. Accept the smaller savings; keep unwind behaviour. # # Estimated impact: 213 MB/release → ~130 MB/release. Buys ~30+ more # release slots within the 10 GB ceiling at the current release # cadence. Per-PyPI-version savings AND faster downloads for end # users. Tradeoff: release builds take ~30-50% longer due to # `codegen-units = 1` + LTO; acceptable for the size win. [profile.release] strip = "symbols" lto = "thin" codegen-units = 1 # The proxy binary turns untrusted network input into byte and token counts; # a wrapping add there is a silent correctness bug at best, so its release # build keeps overflow checks (a panic is caught per task by tokio and # surfaces in the log; a wrong number does not). Scoped to this package: # `headroom-core` is also the `_core` Python extension's compression hot # path and is not included until the cost is benchmarked there. [profile.release.package.headroom-proxy] overflow-checks = true # Fast-to-compile profile for CI test wheels. The shipped wheel uses # `release` (lto + codegen-units=1) for runtime/size; CI only needs a working # extension, so trade runtime perf for ~parallel, lto-free compilation. Used # via `maturin build --profile ci`. Does NOT affect `--release` builds. [profile.ci] inherits = "release" lto = false codegen-units = 256 opt-level = 1 strip = "none" debug = false incremental = false - claude_analysis_ttl.py (12403 bytes)
#!/usr/bin/env python3 """Cache reconstruction cost: cache_creation on first turn after idle gap vs in-window. Prints a pretty distribution table plus a final cost-comparison summary across three caching strategies: current (5m default), naive flip to 1h, and conditional 1h-after-idle. """ import json from collections import defaultdict from datetime import datetime from pathlib import Path PROJECTS = Path.home() / ".claude" / "projects" # $ per million tokens. PRICING = { "claude-sonnet-4-6": {"w5": 3.75, "w1h": 6.00, "r": 0.30, "in": 3.00}, "claude-opus-4-6": {"w5": 6.25, "w1h": 10.00, "r": 0.50, "in": 5.00}, "claude-haiku-4-5": {"w5": 1.25, "w1h": 2.00, "r": 0.10, "in": 1.00}, "claude-opus-4-7": {"w5": 18.75, "w1h": 30.00, "r": 1.50, "in": 15.00}, } DEFAULT_PRICE = PRICING["claude-sonnet-4-6"] unknown_models = set() def price_for(model: str) -> dict[str, float]: if not model: return DEFAULT_PRICE if model in PRICING: return PRICING[model] base = model.split("[")[0] for k in PRICING: if base.startswith(k) or k in base: return PRICING[k] unknown_models.add(model) return DEFAULT_PRICE def parse_ts(s: str) -> datetime: if s.endswith("Z"): s = s[:-1] + "+00:00" return datetime.fromisoformat(s) def turns_of(path: Path, seen_ids: set[str]) -> list[tuple[datetime, int, int, int, str]]: """Parse one JSONL. Skip turns whose message.id was already counted globally.""" out = [] with path.open(errors="replace") as f: for line in f: line = line.strip() if not line: continue try: obj = json.loads(line) except Exception: continue msg = obj.get("message") if not isinstance(msg, dict): continue usage = msg.get("usage") if not isinstance(usage, dict): continue try: ts = parse_ts(obj["timestamp"]) except Exception: continue mid = msg.get("id") if mid: if mid in seen_ids: continue seen_ids.add(mid) cc = usage.get("cache_creation_input_tokens", 0) or 0 cr = usage.get("cache_read_input_tokens", 0) or 0 inp = usage.get("input_tokens", 0) or 0 model = msg.get("model") or usage.get("model") or "" out.append((ts, cc, cr, inp, model)) out.sort(key=lambda x: x[0]) return out BUCKET_ORDER = ["<5min", "5-15min", "15-30min", "30-60min", "1-4hr", ">4hr"] def bucket(g: float) -> str: if g < 5: return "<5min" if g < 15: return "5-15min" if g < 30: return "15-30min" if g < 60: return "30-60min" if g < 240: return "1-4hr" return ">4hr" def main() -> None: buckets: dict[str, list[int]] = {b: [] for b in BUCKET_ORDER} bucket_by_model: dict[str, dict[str, int]] = {b: defaultdict(int) for b in BUCKET_ORDER} total_sessions = 0 first_turn_tokens_by_model: dict[str, int] = defaultdict(int) seen_ids: set[str] = set() for path in sorted(PROJECTS.rglob("*.jsonl")): # sort for deterministic dedupe winner try: t = turns_of(path, seen_ids) except Exception: continue if len(t) < 2: continue total_sessions += 1 # First turn of session: no prior, but the cache_creation IS a fresh write. ts0, cc0, _, _, m0 = t[0] first_turn_tokens_by_model[m0] += cc0 for i in range(1, len(t)): gap = (t[i][0] - t[i - 1][0]).total_seconds() / 60.0 if gap < 0: continue cc = t[i][1] m = t[i][4] b = bucket(gap) buckets[b].append(cc) bucket_by_model[b][m] += cc def stats(lst: list[int]) -> dict[str, int] | None: if not lst: return None s = sorted(lst) n = len(s) return { "n": n, "min": s[0], "p25": s[n // 4], "median": s[n // 2], "p75": s[3 * n // 4], "p95": s[min(n - 1, int(n * 0.95))], "max": s[-1], "mean": sum(s) // n, "total": sum(s), } # ----- Pretty distribution table ----- print() print("=" * 88) print(f" CACHE RECONSTRUCTION COST — {total_sessions:,} sessions analyzed") print("=" * 88) print() print(" cache_creation tokens, bucketed by gap since previous turn") print() header = f" {'bucket':<10} {'count':>8} {'median':>12} {'mean':>12} {'p75':>12} {'p95':>12} {'total':>16}" print(header) print(" " + "-" * (len(header) - 2)) for b in BUCKET_ORDER: st = stats(buckets[b]) if st: print( f" {b:<10} {st['n']:>8,} {st['median']:>12,} {st['mean']:>12,} " f"{st['p75']:>12,} {st['p95']:>12,} {st['total']:>16,}" ) print() # ----- Smoking-gun ratios ----- in_window = buckets["<5min"] post_idle_short = buckets["5-15min"] post_idle_5to60 = buckets["5-15min"] + buckets["15-30min"] + buckets["30-60min"] med_in = sorted(in_window)[len(in_window) // 2] if in_window else 0 med_5_15 = sorted(post_idle_short)[len(post_idle_short) // 2] if post_idle_short else 0 med_5_60 = sorted(post_idle_5to60)[len(post_idle_5to60) // 2] if post_idle_5to60 else 0 print("-" * 88) print(" RECONSTRUCTION RATIO — the smoking gun") print("-" * 88) print(f" Median in-window write (<5min gap) : {med_in:>10,} tokens") print( f" Median post-idle write (5-15min gap) : {med_5_15:>10,} tokens " f"({med_5_15 / max(med_in, 1):>5.0f}x)" ) print( f" Median post-idle write (5-60min gap) : {med_5_60:>10,} tokens " f"({med_5_60 / max(med_in, 1):>5.0f}x)" ) print() # ----- Cost comparison across strategies ----- # Strategy A — current: all writes at 5m price. # cost_A = sum_m (in_window_m + post_idle_5to60_m + post_idle_over60_m) * w5 # Strategy B — naive 1h: every write becomes a 1h write; 5-60min rewrites flip to reads. # cost_B = sum_m [(in_window_m + post_idle_over60_m) * w1h + post_idle_5to60_m * r] # Strategy C — conditional 1h-after-idle: write 5m on in-window deltas, write 1h # only on first turn after >=5min idle. Then the 5-60min rewrites become # reads on the *next* gap event (they already are, post-write), and the # >60min rewrites still cost a 1h write (they expired even the 1h cache). # cost_C = sum_m [in_window_m * w5 + post_idle_5to60_m * r + post_idle_over60_m * w1h] # # NOTE: Strategy C model assumes the post-idle rewrite events we measured today would # become reads under conditional-1h. That's accurate for gaps in [5min, 60min) because # the previous turn (now written at 1h) is still cached when the next turn arrives. def cost(tok: int, ppm: float) -> float: return tok * ppm / 1_000_000.0 # Aggregate per-model token totals. by_model: dict[str, dict[str, int]] = defaultdict( lambda: {"in": 0, "p_5to60": 0, "p_over60": 0, "first": 0} ) for b in BUCKET_ORDER: for m, tok in bucket_by_model[b].items(): if b == "<5min": by_model[m]["in"] += tok elif b in ("5-15min", "15-30min", "30-60min"): by_model[m]["p_5to60"] += tok else: by_model[m]["p_over60"] += tok for m, tok in first_turn_tokens_by_model.items(): # First turn of a session is a fresh write; treat it as a >5min "post-idle" # since there's no prior to refresh. Conservative: bucket as p_over60 so # conditional-1h pays 1h for it too. by_model[m]["p_over60"] += tok rows: list[tuple[str, dict[str, int], float, float, dict[str, float]]] = [] A_total = 0.0 B_total = 0.0 for m, agg in by_model.items(): p = price_for(m) A = ( cost(agg["in"], p["w5"]) + cost(agg["p_5to60"], p["w5"]) + cost(agg["p_over60"], p["w5"]) ) B = ( cost(agg["in"], p["w1h"]) + cost(agg["p_5to60"], p["r"]) + cost(agg["p_over60"], p["w1h"]) ) A_total += A B_total += B rows.append((m, agg, A, B, p)) print("-" * 88) print(" COST COMPARISON — two caching strategies") print("-" * 88) print() print(" Strategies:") print(" A) Current — all cache writes at 5m TTL") print(" B) Naive 1h — flip default: all writes at 1h TTL; 5-60min rewrites become reads") print() # simpler totals tot_in = sum(a["in"] for a in by_model.values()) tot_5to60 = sum(a["p_5to60"] for a in by_model.values()) tot_over60 = sum(a["p_over60"] for a in by_model.values()) grand = tot_in + tot_5to60 + tot_over60 print() print(f" {'category':<40} {'tokens':>16} {'% of total':>12}") print(" " + "-" * 70) print(f" {'in-window deltas (<5min)':<40} {tot_in:>16,} {tot_in / grand * 100:>11.1f}%") print( f" {'avoidable rewrites (5-60min idle)':<40} {tot_5to60:>16,} {tot_5to60 / grand * 100:>11.1f}%" ) print( f" {'unavoidable rewrites (>60min + first)':<40} {tot_over60:>16,} {tot_over60 / grand * 100:>11.1f}%" ) print(f" {'TOTAL cache_creation':<40} {grand:>16,} {100.0:>11.1f}%") print() # Per-model cost rows print(f" {'model':<22} {'A: current 5m':>15} {'B: naive 1h':>15} {'B vs A':>10}") print(" " + "-" * 70) for m, _agg, A, B, _p in sorted(rows, key=lambda r: -r[2]): name = m or "<unknown>" if len(name) > 22: name = name[:21] + "…" dB = B - A print(f" {name:<22} ${A:>13,.2f} ${B:>13,.2f} ${dB:>+9,.2f}") print(" " + "-" * 70) dB_total = B_total - A_total print(f" {'TOTAL':<22} ${A_total:>13,.2f} ${B_total:>13,.2f} ${dB_total:>+9,.2f}") print() # ----- Hypothetical: same token mix priced as if 100% Sonnet vs 100% Opus ----- print("-" * 88) print(" HYPOTHETICAL — same token mix, all on one model") print("-" * 88) print() print(" Re-prices the observed cache_creation token mix as if every token had") print(" been written by a single model. Lets you compare TTL impact at each tier.") print() hypos = [ ("All Sonnet 4.6", PRICING["claude-sonnet-4-6"]), ("All Opus 4.7", PRICING["claude-opus-4-7"]), ] print( f" {'scenario':<18} {'A: current 5m':>15} {'B: naive 1h':>15} {'B vs A':>10} {'B vs A %':>10}" ) print(" " + "-" * 80) for name, p in hypos: A = cost(tot_in, p["w5"]) + cost(tot_5to60, p["w5"]) + cost(tot_over60, p["w5"]) B = cost(tot_in, p["w1h"]) + cost(tot_5to60, p["r"]) + cost(tot_over60, p["w1h"]) d = B - A pct = (d / A * 100) if A else 0 print(f" {name:<18} ${A:>13,.2f} ${B:>13,.2f} ${d:>+9,.2f} {pct:>+9.1f}%") print() # ----- Bottom line ----- print("=" * 88) print(" BOTTOM LINE") print("=" * 88) print() print(f" Sample: {total_sessions:,} sessions, {grand:,} total cache_creation tokens") print() print(f" A) Current 5m default : ${A_total:>10,.2f} (baseline)") sign_B = "+" if dB_total >= 0 else "-" print( f" B) Naive flip to 1h : ${B_total:>10,.2f} ({sign_B}${abs(dB_total):,.2f} vs current)" ) print() if dB_total > 0: print( f" Verdict: naive flip COSTS MORE because the 1.6x premium on {tot_in / grand * 100:.0f}% of tokens" ) print( f" (in-window deltas) exceeds savings on {tot_5to60 / grand * 100:.0f}% (post-idle rewrites)." ) elif dB_total < 0: print(f" Verdict: naive 1h flip saves ${abs(dB_total):,.2f} on this sample.") else: print(" Verdict: 1h flip is cost-neutral on this sample.") print() if unknown_models: print(f" Note: unknown models defaulted to Sonnet pricing: {sorted(unknown_models)}") print() if __name__ == "__main__": main() - CODE_OF_CONDUCT.md (5492 bytes)
# Contributor Covenant Code of Conduct ## Our Pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at **[email protected]**. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. [homepage]: https://www.contributor-covenant.org [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html [Mozilla CoC]: https://github.com/mozilla/diversity [FAQ]: https://www.contributor-covenant.org/faq [translations]: https://www.contributor-covenant.org/translations
- codecov.yml (926 bytes)
codecov: require_ci_to_pass: true coverage: status: # Gate on the comprehensive unit suite (`python` flag from ci.yml's 4 test # shards), NOT the narrow native-e2e smoke flags (install-native / # wrap-native). Those e2e jobs upload first and barely exercise new code, # so an unscoped status computes patch at ~6% off the e2e flags alone and # flaps to FAILURE before the unit shards report. Scoping to `python` makes # the status reflect real coverage of the diff. project: default: target: auto flags: - python patch: default: target: auto flags: - python flags: python: carryforward: false ignore: - "tests/**" - "scripts/tests/**" - ".github/**" - ".claude-plugin/**" - "headroom/dashboard/templates/**" - "plugins/headroom-agent-hooks/.claude-plugin/**" - "plugins/headroom-agent-hooks/.github/**" - CONTRIBUTING.md (8197 bytes)
# Contributing to Headroom Thanks for contributing! Please skim this before opening a PR : the policies exist because we've been burned skipping them, not because we love paperwork. By participating, you agree to our [Code of Conduct](CODE_OF_CONDUCT.md). ## Where does my contribution go? | Type | What to do | | --- | --- | | 🐛 Bug or small fix | **Open a PR** (with repro + test) | | ✨ New feature / architectural change | **Open an issue or ask in Discord first.** | | 🧹 Refactor-only | **Don't.** Only if a maintainer asked, as part of a concrete fix. | | 🧪 Test/CI-only PR chasing a known `main` failure | **Don't.** We're tracking it. | | 📦 New dep or version bump | **PR with written justification.** | | ❓ Question | Ask in **Discord `#help`** | **Open PR cap: 10 per author.** Get existing ones merged before opening more. ## Guiding principles - **Verification is the author's job, not the reviewer's.** - **Supply chain is a real threat.** Dependency changes get human review, every time. ## Bug fixes Every bug-fix PR must include: 1. **A reproduction** — minimal code, failing test, or steps. 2. **A test that fails before your fix and passes after** (unit, integration, or e2e). If you genuinely can't write a test, say so explicitly and explain how you verified. ## "Real behavior proof" — required on every external PR We can't merge what we can't verify. Include a **`Real behavior proof`** section in the PR body covering: - **Setup you tested on** (OS, Python, config, provider/model) - **Exact command or steps you ran after the patch** - **After-fix evidence** + **observed result** - **What you did *not* test** ✅ Counts: screenshots, recordings, terminal output, copied live output, linked artifacts, redacted runtime logs. ❌ Does **not** count alone: unit tests, mocks, snapshots, lint, typechecks, green CI. Have them too — but they prove the test passes, not that the feature works. **PRs missing this may be autoclosed.** ## New features Before writing code: 1. **Open a feature-request issue** (or raise in Discord). 2. **Get a 👍 from a core maintainer** before implementing. 3. **Include a short spec** covering: - **API surface** (public functions, config, CLI flags) - **Changes to existing behavior** - **User stories** — Given / When / Then, golden path + one edge case - **Failure modes** - **Recovery / resilience** - **Security considerations** Short and concrete beats long. ## Dependencies & supply chain A human maintainer reviews every dep change. PRs that add or bump a package must justify: - **Why this package** (vs. doing it ourselves / using existing deps) - **Who maintains it** (activity, release cadence, security history) - **Install surface** (transitive deps, native code, install/runtime network) - **Why this version** — permitted reasons: **bug fix**, **security patch**, **required new functionality**. Cosmetic bumps will be closed. ## PR workflow 1. Fork, branch from `main`. 2. Install **Node 18+** and run `uv sync --extra dev` then `make install-git-hooks` — installs repo pre-commit checks on every commit, commitlint on every commit message, and ci-precheck on every push. 3. One logical change per PR. 4. Add tests. 5. `uv run pytest` · `uv run ruff check .` · `uv run ruff format .` 6. Do **not** edit `CHANGELOG.md` — release-please generates it from your Conventional Commit PR title, so a clear `fix(...)`/`feat(...)` title *is* your changelog entry. A CI guard rejects manual edits. 7. Open the PR with a clear description + `Real behavior proof` + any spec/justification required, and keep the PR in draft until the `Review Readiness` boxes are complete. **Title format** (conventional commits): `feat:`, `fix:`, `docs:`, `test:`, `refactor:`. **Commit message format** is enforced locally by the repo's `commit-msg` hook and again in CI. **Review:** CI green, one maintainer review, coverage held/improved. ## Posting PR/issue comments from scripts or agents A successful API call (exit 0) only proves a comment was created, not that its body is right. Scripts must: - Pass the body as a single plain string via a file (`gh pr comment --body-file`, `gh issue comment --body-file`, or `gh api --input <json-file>`). Avoid inline multi-line strings, which differ across shells, and never pass a collection or object as a field value (`-f body=...`). - Re-fetch the posted comment and confirm the body reads as intended before reporting success. - If the body is a bare type name (e.g. `System.Collections.Hashtable`) or otherwise garbled, delete it and repost: `gh api -X DELETE repos/<owner>/<repo>/pulls/comments/<id>` for review comments, `.../issues/comments/<id>` for issue and top-level PR comments. ## Development setup ```bash git clone https://github.com/headroomlabs-ai/headroom.git cd headroom python -m venv .venv && source .venv/bin/activate node --version # Node 18+ required for commitlint hooks python -m pip install --upgrade pip python -m pip install -e ".[dev,relevance,proxy]" python -m pytest ``` Headroom uses a `pyproject.toml`/`maturin` build backend. Older `pip` versions may fail editable installs by looking for `setup.py`; upgrade `pip` first or use `uv sync --extra dev`. Handler tests that read request bodies should use a real Starlette `Request` with ASGI receive messages, including streamed chunks, rather than patching the body reader. Size-limit regressions must configure an explicit test ceiling; the production ceiling is independent of the compression admission threshold. Avoid replacing imported proxy packages in `sys.modules`: it can split live helper and exception identities across full-suite test orderings. ### Dev Containers Two configs ship for VS Code / Codespaces: - **`.devcontainer/devcontainer.json`** — Python 3.12, `uv`, Node.js, `gh`. - **`.devcontainer/memory-stack/devcontainer.json`** — adds Qdrant + Neo4j sidecars (use `qdrant:6333`, `neo4j://neo4j:7687`). Inside, use: `uv run ruff check .`, `uv run pytest`, etc. ## Optional automated review This repository includes `.github/copilot-instructions.md` so maintainers can opt into GitHub Copilot code review without adding workflow billing noise to every PR. Enable or disable automatic Copilot review in **Settings → Rules → Rulesets → Automatically request Copilot code review**. Keep it off unless maintainers explicitly want the extra review traffic. ## Coding standards - [Ruff](https://github.com/astral-sh/ruff) for lint + format, line length 100, PEP 8. - Type hints on public functions; Google-style docstrings. - Cover new behavior + edge cases; aim >80% coverage on new code. - Python 3.11+. Optional features go behind extras. - For an optional-integration change, run its owner tests and `mypy headroom` with the relevant extra installed. A hook environment without that dependency does not check the imported library's types. CI lint installs `langchain-core` for its normal mypy run; keep typechecking there rather than launching a second mypy subprocess from pytest. - **Headroom writes LF and normalizes on read, on every platform.** Pass `newline="\n"` to every `write_text`/`open` that writes a context, memory, or state file, and normalize `\r\n`/`\r` when you read one back. Without the pin, `TextIOWrapper` translates `\n` to `\r\n` on Windows, and a reader that decodes bytes directly then re-writes accumulates carriage returns (#3594); even with a normalizing reader, two subsystems writing the same file (`CLAUDE.md`, `AGENTS.md`) flip it between LF and CRLF (#3698). Cover new write sites with `@pytest.mark.windows_newline` — those tests assert the `newline=` kwarg (an artifact assertion cannot fail on POSIX) and also run on `windows-latest` in CI. ## Architecture principles **Safety first:** never drop user/assistant content, never break tool call/response pairing, malformed content passes through unchanged, prefer false negatives. **Performance:** transforms <50ms at P99, lazy-load optional deps, profile before optimizing. Contributors are credited in `CHANGELOG`, the GitHub contributors page, and release notes. Thanks again. 💚
- deny.toml (578 bytes)
# cargo-deny configuration for the Rust workspace. Intentionally permissive # during Phase 0 — tighten before Phase 2 goes to production. [graph] all-features = false [licenses] version = 2 allow = [ "MIT", "Apache-2.0", "Apache-2.0 WITH LLVM-exception", "BSD-2-Clause", "BSD-3-Clause", "ISC", "Unicode-3.0", "Unicode-DFS-2016", "CC0-1.0", "Zlib", "0BSD", "MPL-2.0", ] confidence-threshold = 0.8 exceptions = [] [bans] multiple-versions = "allow" wildcards = "allow" [sources] unknown-registry = "warn" unknown-git = "warn" - docker-compose.yml (6170 bytes)
# ============================================================================= # Headroom — full "memory stack" compose # ============================================================================= # Brings up the Headroom proxy together with the two datastores it needs for # semantic memory: Qdrant (vector search) and Neo4j (relationship graph). # # Quick start: # 1. cp .env.example .env # then set a real NEO4J_AUTH before any non-local use # 2. docker compose up -d # 3. point your LLM client at http://localhost:8787 (proxy) # # Just want the proxy without the memory features? You can run the proxy image # on its own (`docker run --rm -p 127.0.0.1:8787:8787 -e HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 # ghcr.io/headroomlabs-ai/headroom --host 0.0.0.0 --port 8787`; the env var acknowledges that the # 127.0.0.1 publication, not a token, is what keeps the port private); the two # database services below are only required for the memory/relevance features. # # Ports published on the host — all bound to 127.0.0.1 (this machine only): # 8787 proxy (OpenAI-compatible endpoint) # 6333 Qdrant REST 6334 Qdrant gRPC # 7474 Neo4j Browser 7687 Neo4j Bolt # # None of these three services authenticates inbound callers by default: the # proxy's /v1/* data plane is open unless HEADROOM_PROXY_TOKEN is set, Qdrant # has no API key, and Neo4j falls back to a published dev password. Publishing # them on 0.0.0.0 therefore hands any peer on your network a relay through the # proxy plus direct read/write on the embeddings and graph derived from your # prompts. They are bound to loopback so that `docker compose up -d` is safe on # a shared or untrusted network. # # To reach the proxy from another machine, publish it deliberately AND require # a token — never one without the other: # HEADROOM_PROXY_TOKEN=$(openssl rand -hex 32) # put this in .env # ports: ["8787:8787"] # override in a compose override file # ============================================================================= services: # Headroom proxy — the OpenAI-compatible endpoint your client talks to. # Built from the repo Dockerfile so it tracks your local checkout. headroom-proxy: build: context: . args: HEADROOM_BUILD_VERSION: ${HEADROOM_BUILD_VERSION:-source-build} # Bind to all interfaces inside the container so the published port is reachable. command: ["--host", "0.0.0.0"] environment: - HEADROOM_HOST=0.0.0.0 # The proxy binds 0.0.0.0 *inside* the container (required for Docker port # forwarding); it is confined to host loopback by the published port below. # A proxy token is required so the data plane is never open if you widen the # bind. Generate one with: openssl rand -hex 32 - HEADROOM_PROXY_TOKEN=${HEADROOM_PROXY_TOKEN:?set HEADROOM_PROXY_TOKEN (see .env.example; e.g. openssl rand -hex 32)} - HOME=/home/nonroot # Keep all Headroom read/write state on the named volume below. - HEADROOM_WORKSPACE_DIR=/home/nonroot/.headroom - HEADROOM_CONFIG_DIR=/home/nonroot/.headroom/config # if you want to use a custom OpenAI-compatible API endpoint, # uncomment and set the following line with the desired URL # - OPENAI_TARGET_API_URL=https://api.x.ai # Required before publishing this port beyond loopback: without it the # /v1/* data plane accepts unauthenticated callers. # - HEADROOM_PROXY_TOKEN=${HEADROOM_PROXY_TOKEN} ports: # Loopback-only. The container still listens on 0.0.0.0 (above) so the # other compose services can reach it by name; this line controls only # which host interfaces the port is published on. - "127.0.0.1:8787:8787" volumes: - headroom_workspace:/home/nonroot/.headroom # Readiness probe: the orchestrator polls /readyz so dependents and # `docker compose up --wait` only see the proxy as healthy once it's serving. healthcheck: test: ["CMD", "curl", "--fail", "--silent", "http://127.0.0.1:8787/readyz"] interval: 30s timeout: 5s retries: 3 start_period: 20s # Start the datastores first. Note: this waits for the containers to start, # not for them to be fully ready — the proxy retries its connections, so a # brief "database not ready yet" window on first boot is expected. depends_on: - qdrant - neo4j # Vector database for semantic search. # Stores embeddings so the proxy can retrieve semantically similar context. qdrant: image: qdrant/qdrant:v1.17.1 ports: # Loopback-only: Qdrant runs unauthenticated here and holds embeddings # derived from your prompts. - "127.0.0.1:6333:6333" # REST API - "127.0.0.1:6334:6334" # gRPC # Named volume keeps the vector index across container restarts/recreates. volumes: - qdrant_data:/qdrant/storage environment: - QDRANT__SERVICE__GRPC_PORT=6334 # Graph database for relationships and multi-hop reasoning. # Backs the memory features that traverse links between stored items. neo4j: image: neo4j:5.26 ports: # Loopback-only to keep the graph store off the network. - "127.0.0.1:7474:7474" # HTTP (Browser) - "127.0.0.1:7687:7687" # Bolt # Named volume persists the graph data across container restarts/recreates. volumes: - neo4j_data:/data environment: # No default credential — must be supplied (see .env.example). - NEO4J_AUTH=${NEO4J_AUTH:?set NEO4J_AUTH, e.g. neo4j/<strong-password>} # APOC: Neo4j's standard procedure library, needed by Headroom's queries. - NEO4J_PLUGINS=["apoc"] # APOC file import/export stays disabled (its Neo4j default) — it grants # filesystem read/write via stored procedures. Do not enable unless required. # Named volumes — managed by Docker, survive `docker compose down` (use # `docker compose down -v` to delete the stored data as well). volumes: headroom_workspace: # persists dashboard savings/history, logs, config, memory state, session stats, and TOIN qdrant_data: neo4j_data: - Dockerfile (10945 bytes)
ARG PYTHON_VERSION=3.13 ARG UV_VERSION=0.11.18 ARG DISTROLESS_IMAGE=gcr.io/distroless/python3-debian13 ARG PYTHON_SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/site-packages # ---- Build stage: compile native extensions, build wheel ---- FROM python:${PYTHON_VERSION}-slim AS builder ARG UV_VERSION ARG PYTHON_SITE_PACKAGES ARG HEADROOM_BUILD_VERSION="" # build-essential / g++ for any C extension wheels uv may need to build # from source. curl + ca-certificates are required by the rustup # bootstrap below. patchelf for maturin's wheel-link repair on linux. # No OpenSSL system deps required: the rustls-everywhere refactor # eliminated `openssl-sys` from our build tree by switching fastembed # to `hf-hub-rustls-tls` + `ort-download-binaries-rustls-tls`. RUN apt-get update && \ apt-get install -y --no-install-recommends \ build-essential \ g++ \ curl \ ca-certificates \ patchelf \ && rm -rf /var/lib/apt/lists/* RUN python -m pip install --no-cache-dir uv==${UV_VERSION} # Rust toolchain for the headroom._core extension. With single-wheel # architecture (post-#355), `pip install -e .` invokes maturin via # pyproject.toml's [build-system], which calls cargo. No more separate # headroom-core-py package. ENV CARGO_HOME=/usr/local/cargo \ RUSTUP_HOME=/usr/local/rustup \ PATH=/usr/local/cargo/bin:${PATH} COPY scripts/install-rustup.sh /tmp/install-rustup.sh RUN sh /tmp/install-rustup.sh -y --no-modify-path --profile minimal -c rustfmt -c clippy --default-toolchain 1.95.0 WORKDIR /build # Copy the full set of files maturin needs to build the wheel: the root # pyproject.toml + Cargo workspace + Rust crates + Python source. The # uv install builds + installs the wheel in one shot. COPY pyproject.toml uv.lock README.md README-pypi.md ./ COPY Cargo.toml Cargo.lock rust-toolchain.toml ./ COPY crates/ crates/ COPY headroom/ headroom/ # The standalone Dockerfile must support every backend advertised by # `headroom proxy --backend`, including Bedrock temporary/SSO credentials. # Those credentials require botocore (GH #1551), supplied by [bedrock]. # [redis] backs the replica-shared stores: without it HEADROOM_COMPRESS_SESSION_STORE # stops startup and HEADROOM_CCR_BACKEND=redis falls back to a per-process store. ARG HEADROOM_EXTRAS=proxy,code,bedrock,redis # `uv pip install` ignores [tool.uv] override-dependencies (uv.lock only), so # pass them explicitly: the image gets the security overrides the lock has # (pillow 13 over fastembed's <13 cap). RUN --mount=type=cache,target=/root/.cache/uv \ --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git \ --mount=type=cache,target=/build/target \ python -c "exec('try: import tomllib\\nexcept ImportError: from pip._vendor import tomli as tomllib'); print('\\n'.join(tomllib.load(open('pyproject.toml', 'rb'))['tool']['uv']['override-dependencies']))" > /tmp/overrides.txt && \ uv pip install --system --overrides /tmp/overrides.txt ".[${HEADROOM_EXTRAS}]" RUN --mount=type=bind,source=.,target=/context,readonly \ HEADROOM_BUILD_VERSION="${HEADROOM_BUILD_VERSION}" PYTHON_SITE_PACKAGES="${PYTHON_SITE_PACKAGES}" python - <<'PY' import hashlib import os from pathlib import Path def git_revision(context: Path) -> str | None: git_dir = context / ".git" head_path = git_dir / "HEAD" if not head_path.exists(): return None head = head_path.read_text(encoding="utf-8").strip() if head.startswith("ref: "): ref_name = head.removeprefix("ref: ").strip() ref_path = git_dir / ref_name if ref_path.exists(): head = ref_path.read_text(encoding="utf-8").strip() else: packed_refs = git_dir / "packed-refs" if not packed_refs.exists(): return None for line in packed_refs.read_text(encoding="utf-8").splitlines(): if line.startswith("#") or not line.strip(): continue sha, _, name = line.partition(" ") if name.strip() == ref_name: head = sha break else: return None return head[:12] if len(head) >= 7 and all(c in "0123456789abcdef" for c in head.lower()) else None def source_digest(root: Path) -> str: digest = hashlib.sha256() inputs = ( "pyproject.toml", "uv.lock", "README.md", "README-pypi.md", "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "crates", "headroom", ) for name in inputs: path = root / name if not path.exists(): continue files = [path] if path.is_file() else sorted(p for p in path.rglob("*") if p.is_file()) for file in files: digest.update(file.relative_to(root).as_posix().encode("utf-8")) digest.update(b"\0") digest.update(file.read_bytes()) digest.update(b"\0") return digest.hexdigest()[:12] build_version = os.environ["HEADROOM_BUILD_VERSION"].strip() if not build_version: print("no Headroom build version override provided; using installed package metadata") raise SystemExit(0) if build_version == "source-build": revision = git_revision(Path("/context")) build_version = ( f"source-build+g{revision}" if revision else f"source-build+sha256.{source_digest(Path('/build'))}" ) package_dir = Path(os.environ["PYTHON_SITE_PACKAGES"]) / "headroom" (package_dir / "_build_info.py").write_text( "BUILD_VERSION = " + repr(build_version) + "\n", encoding="utf-8", ) print("baked Headroom build version: " + build_version) PY # Build-stage smoke check: verify the extension loads end-to-end inside # the build image before we copy site-packages into the runtime image. # If this fails, the runtime image would fail Phase A0's fail-loud # startup check on every restart. Run from /tmp so cwd doesn't shadow # site-packages with /build/headroom/ (which has no _core.so since # maturin installed the .so into site-packages). RUN cd /tmp && python -c "from headroom._core import DiffCompressor, SmartCrusher; \ print(f'build-stage rust core verify OK: {DiffCompressor.__name__}, {SmartCrusher.__name__}')" # Build the native Rust reverse proxy binary and stage it for the runtime # images (issue #976). These images already run "the proxy"; bundling the # native `headroom-proxy` binary lets operators front the Python proxy with # the Rust SigV4 / live-zone compression path from the same image. The # binary is copied out of the cache-mounted target dir into a persistent # path so the COPY in the runtime stages can pick it up. RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/build/target \ cargo build --release --locked --bin headroom-proxy && \ cp target/release/headroom-proxy /usr/local/bin/headroom-proxy # ---- Runtime stage (python-slim): supports root/nonroot via build arg ---- FROM python:${PYTHON_VERSION}-slim AS runtime-slim-base ARG RUNTIME_USER=nonroot ARG RUNTIME_HOME=/home/nonroot ARG PYTHON_SITE_PACKAGES RUN apt-get update && \ apt-get install -y --no-install-recommends curl && \ rm -rf /var/lib/apt/lists/* COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES} COPY --from=builder /usr/local/bin/headroom /usr/local/bin/headroom # Native Rust reverse proxy binary (issue #976). COPY --from=builder /usr/local/bin/headroom-proxy /usr/local/bin/headroom-proxy RUN mkdir -p /home/nonroot /data && \ if [ "$RUNTIME_USER" = "nonroot" ]; then \ groupadd --gid 1000 nonroot && \ useradd --uid 1000 --gid nonroot --create-home nonroot && \ mkdir -p /home/nonroot/.headroom && \ chown -R nonroot:nonroot /data /home/nonroot; \ else \ mkdir -p /root/.headroom; \ fi USER ${RUNTIME_USER} WORKDIR ${RUNTIME_HOME} ENV HEADROOM_HOST=127.0.0.1 \ PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 # Declare ~/.headroom as a volume so Docker (and ACA) can attach persistent # storage here. Bare `docker run` gets an anonymous volume as a fallback so # state is never silently written to the ephemeral container layer. # RUNTIME_HOME defaults to /home/nonroot (the published image default); pass # --build-arg RUNTIME_HOME=/root when building with RUNTIME_USER=root. VOLUME ${RUNTIME_HOME}/.headroom EXPOSE 8787 # Shell form so ${HEADROOM_PORT} expands at probe time: `headroom deploy # --port N` sets it in the container env, and an exec-form CMD would keep # probing 8787 and report a working deployment as unhealthy (issue #2432). HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \ CMD curl --fail --silent "http://127.0.0.1:${HEADROOM_PORT:-8787}/readyz" ENTRYPOINT ["headroom", "proxy"] # Keep host and port defaults in the CLI so HEADROOM_HOST and HEADROOM_PORT # remain authoritative for the published image. # Distroless has no shell, so the ~/.headroom state directory cannot be made # with RUN there. Build it here and COPY it in with the right owner (#4443). FROM python:${PYTHON_VERSION}-slim AS runtime-slim-state RUN mkdir -p /state/.headroom FROM ${DISTROLESS_IMAGE} AS runtime-slim ARG RUNTIME_USER=nonroot ARG RUNTIME_HOME=/home/nonroot ARG PYTHON_SITE_PACKAGES COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES} # Native Rust reverse proxy binary (issue #976). COPY --from=builder /usr/local/bin/headroom-proxy /usr/local/bin/headroom-proxy # Owned by the runtime user so the volume below is writable (user and group # are both named after RUNTIME_USER in the distroless image). COPY --from=runtime-slim-state --chown=${RUNTIME_USER}:${RUNTIME_USER} /state/ ${RUNTIME_HOME}/ USER ${RUNTIME_USER} WORKDIR /app ENV HEADROOM_HOST=127.0.0.1 \ PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONPATH=${PYTHON_SITE_PACKAGES} # Same persistent-state volume as the python-slim stage; pass # --build-arg RUNTIME_HOME=/root when building with RUNTIME_USER=root. VOLUME ${RUNTIME_HOME}/.headroom EXPOSE 8787 # This stage is distroless, so there is no shell to expand ${HEADROOM_PORT}. # Resolve the port inside Python instead, keeping exec form (issue #2432). # `or` rather than a get() default so an empty HEADROOM_PORT falls back the # same way `${HEADROOM_PORT:-8787}` does in the shell-form stage above. HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \ CMD ["python3", "-c", "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:' + (os.environ.get('HEADROOM_PORT') or '8787') + '/readyz', timeout=5)"] ENTRYPOINT ["python3", "-m", "headroom.cli", "proxy"] # Keep host and port defaults in the CLI so HEADROOM_HOST and HEADROOM_PORT # remain authoritative for the published image. # Default published image remains python-slim runtime FROM runtime-slim-base AS runtime - INSTRUCTIONS.md (3421 bytes)
--- name: headroom description: Use when a Python or TypeScript app sends large tool outputs, logs, JSON, search results or long chat history to an LLM and the task is to cut input tokens, reduce token cost, fit a context window, or trim tool results without losing errors. Adds the headroom-ai compress() call; in-process, no API key, no model download. --- # Headroom: compress LLM context before the call ## When this applies The user asks to reduce LLM token cost, trim or compress tool outputs or tool results, prune or summarize chat history, fit a prompt into a context window, or stop large logs or JSON from blowing up the prompt. Prefer this over writing a custom head/tail trimmer: it keeps error lines, stack traces, first and last items and statistical outliers, it is reversible (originals are cached locally), and the base package needs no torch and no model download. ## Python (in-process, no network) ```bash pip install headroom-ai # add to requirements.txt / pyproject ``` ```python from headroom import compress result = compress(messages, model=MODEL_NAME) # OpenAI or Anthropic message dicts # call the provider with result.messages instead of messages # log result.tokens_before, result.tokens_after, result.tokens_saved ``` Compression applies to `tool` (and assistant) messages. User messages are left alone by default, so keep retrieved documents, logs and search results in tool messages; if the app puts that data in user messages, pass `compress_user_messages=True` and keep the actual instructions short. Verify without a provider key: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` Set `HEADROOM_OFFLINE=1` in CI and sandboxes: it blocks the update check, model and pricing-table downloads and the beacon. `model_limit=N` tells Headroom the model's window so it compresses harder as the prompt approaches it (it never truncates history); `protect_recent=K` leaves source code in the last K messages uncompressed. ## TypeScript `npm install headroom-ai` gives `compress()` and `withHeadroom()` wrappers for the OpenAI, Anthropic, Gemini and Vercel AI SDK clients, but compression runs in the Python proxy: start it first with `pip install "headroom-ai[proxy]"` and `headroom proxy --port 8787`. If a Python sidecar is not acceptable, say so and stop. ## Coding agents To cut tokens for Claude Code, Codex, Cursor or Copilot themselves (not an app): `uv tool install --python 3.13 "headroom-ai[all]"` then `headroom wrap claude`. ## Do not - Do not claim a savings percentage; print `tokens_before` and `tokens_after`. - Do not use `headroom.integrations.asgi.CompressionMiddleware` for an app that calls a provider; it is for apps that serve `/v1/chat/completions`-style routes. - Do not install `[all]` or `[ml]` unless asked; the base package is enough. Docs index: https://docs.headroomlabs.ai/llms.txt · FastAPI example: https://docs.headroomlabs.ai/docs/fastapi · Repo: https://github.com/headroomlabs-ai/headroom - LICENSE (10770 bytes)
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to the Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2025 Headroom Contributors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - llms.txt (8898 bytes)
# Headroom > Headroom (`headroom-ai`) compresses tool outputs, logs, JSON, files, RAG chunks and chat history before each LLM call. Same answers, fewer input tokens. Measured: 21–57% fewer input tokens on four agent scenarios, 90%+ on repeated JSON arrays and log lines, little on prose. Library (no API key, in-process), proxy, and MCP server. Apache 2.0, local-first. Headroom is shipped as a Python package (`headroom-ai`), a TypeScript package (`headroom-ai`), an OpenAI + Anthropic-compatible HTTP proxy (`headroom proxy`), and an MCP server (`headroom_compress`, `headroom_retrieve`, `headroom_stats` tools). All four modes use the same compression pipeline: per-content-type compressors (JSON, code, logs, diffs, text) feed into a Compress-Cache-Retrieve (CCR) store so compression stays reversible — the LLM can ask for the original whenever it wants. The canonical, always-current documentation index lives at the docs site below. If you can fetch one URL, fetch that one; the entries here are a hand-curated subset. ## Canonical docs (start here) - [Live llms.txt (full doc index)](https://docs.headroomlabs.ai/llms.txt): Auto-generated index of every doc page with descriptions. - [Live llms-full.txt (every doc page concatenated)](https://docs.headroomlabs.ai/llms-full.txt): One Markdown blob containing every doc page. Use when you can spend the tokens for full context. - [Docs site](https://docs.headroomlabs.ai/docs): Human-browsable docs with search. - [GitHub repo](https://github.com/headroomlabs-ai/headroom): Source, issues, releases. - [PyPI package](https://pypi.org/project/headroom-ai/): Python install. - [npm package](https://www.npmjs.com/package/headroom-ai): TypeScript install. ## Install (copy-paste-runnable) - Python: `pip install headroom-ai` is enough for `compress()`; add `[proxy]` for the HTTP proxy and `headroom wrap`, `[all]` for the CLI/proxy feature set. PyTorch-backed `[ml]`, `[memory]`, `[evals]`, and `[voice]` are separate opt-ins. - TypeScript / Node: `npm install headroom-ai` (or `pnpm add headroom-ai`, `bun add headroom-ai`) - Docker local: `docker run --rm -p 127.0.0.1:8787:8787 ghcr.io/headroomlabs-ai/headroom:latest --host 0.0.0.0` - Run the proxy: `headroom proxy --port 8787` then point any client at `http://127.0.0.1:8787` - Wrap an agent in one command: `headroom wrap claude` (also: `codex`, `grok`, `copilot`, `cursor`, `aider`, `opencode`, `cline`, `continue`, `goose`, `openhands`, `openclaw`, `vibe`, `omp`, `zcode`, `droid`) - Zero-network check, no provider key: `HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))"` (prints roughly `20000 -> 95 FATAL kept: True`) ## By task (plain-language jobs this package does) - Reduce input tokens / cut LLM token cost in a Python app: `pip install headroom-ai`, then `from headroom import compress; result = compress(messages, model=...)`, send `result.messages`. No API key, no model download. [FastAPI example](https://docs.headroomlabs.ai/docs/fastapi) - Trim or compress large tool outputs / tool results before they re-enter the context: the same `compress()` call; tool messages are compressed, error lines, stack traces, first/last items and statistical outliers are kept. [Text & logs](https://docs.headroomlabs.ai/docs/text-and-logs) · [SmartCrusher](https://docs.headroomlabs.ai/docs/smart-crusher) - Keep a long chat history inside the context window: `compress(messages, model=..., model_limit=N)` compresses harder as the prompt approaches N tokens; it never drops messages. [Context management](https://docs.headroomlabs.ai/docs/context-management) - Compress RAG chunks / search results / database rows: the same call; JSON arrays are compressed statistically. Put the data in a `tool` message: user messages are left alone by default (`compress_user_messages=True` opts them in). [SmartCrusher](https://docs.headroomlabs.ai/docs/smart-crusher) - Cut the token spend of Claude Code / Codex / Cursor without code changes: `headroom wrap <agent>` or `headroom proxy`. [Proxy](https://docs.headroomlabs.ai/docs/proxy) - Give an MCP host a compress tool: `pip install "headroom-ai[mcp]"` then `headroom mcp install`. [MCP](https://docs.headroomlabs.ai/docs/mcp) - Run fully offline / air-gapped: set `HEADROOM_OFFLINE=1`. [Security model](https://docs.headroomlabs.ai/docs/security-model) Not a fit: short conversational prompts, prose, already-dense payloads. ## Entry points - [Quickstart](https://docs.headroomlabs.ai/docs/quickstart): 5-minute end-to-end (install → compress → call the model). - [Installation](https://docs.headroomlabs.ai/docs/installation): All install paths, extras, Docker tags, env vars. - [Proxy server](https://docs.headroomlabs.ai/docs/proxy): Run as a local HTTP proxy in front of OpenAI / Anthropic / Gemini. - [MCP server](https://docs.headroomlabs.ai/docs/mcp): `headroom_compress`, `headroom_retrieve`, `headroom_stats` for Claude Code / Cursor / any MCP host. - [API reference](https://docs.headroomlabs.ai/docs/api-reference): Python + TypeScript `compress()` API. ## How it works - [How compression works](https://docs.headroomlabs.ai/docs/how-compression-works): Three-stage pipeline + automatic content routing. - [SmartCrusher](https://docs.headroomlabs.ai/docs/smart-crusher): Statistical JSON / array compression (90%+ on repeated arrays; keeps errors, outliers and boundaries). - [Code compression](https://docs.headroomlabs.ai/docs/code-compression): AST-aware via tree-sitter (preserves imports, signatures, types). - [Text & log compression](https://docs.headroomlabs.ai/docs/text-and-logs): Search results, build logs, diffs. - [CCR (reversible)](https://docs.headroomlabs.ai/docs/ccr): Compress-Cache-Retrieve — originals never deleted; LLM retrieves on demand. ## SDK / framework integrations - [Anthropic SDK](https://docs.headroomlabs.ai/docs/anthropic-sdk): `withHeadroom(anthropic)` wrapper. - [OpenAI SDK](https://docs.headroomlabs.ai/docs/openai-sdk): `withHeadroom(openai)` wrapper. - [Vercel AI SDK](https://docs.headroomlabs.ai/docs/vercel-ai-sdk): Middleware + `withHeadroom()`. - [LangChain](https://docs.headroomlabs.ai/docs/langchain): Chat models, memory, retrievers, agents. - [Agno](https://docs.headroomlabs.ai/docs/agno): Model wrapping + observability hooks. - [Strands](https://docs.headroomlabs.ai/docs/strands): Model wrapping + hook-based tool output compression. - [LiteLLM](https://docs.headroomlabs.ai/docs/litellm): Single callback; works with all 100+ LiteLLM providers. ## Memory & cross-agent state - [Persistent memory](https://docs.headroomlabs.ai/docs/memory): Per-project SQLite + HNSW vector store. No cross-project bleed (GH #462). - [SharedContext](https://docs.headroomlabs.ai/docs/shared-context): Compressed inter-agent context handoffs. - [Failure learning](https://docs.headroomlabs.ai/docs/failure-learning): Offline analysis writes corrections to `CLAUDE.local.md` (default, gitignored) or `CLAUDE.md` (shared) / `AGENTS.md` / `GEMINI.md`. ## Operations - [Configuration](https://docs.headroomlabs.ai/docs/configuration): Env vars, config file, per-call overrides. - [Benchmarks](https://docs.headroomlabs.ai/docs/benchmarks): Token-savings numbers across content types. - [Troubleshooting](https://docs.headroomlabs.ai/docs/troubleshooting): Common failure modes and fixes. - [Limitations](https://docs.headroomlabs.ai/docs/limitations): What Headroom won't do well today. ## Headroom for Teams Headroom OSS is a framework built to show what's possible on one developer's laptop, so it saves less than Headroom Enterprise. Enterprise adds more cost savings, tokens secured, on premise deployment, deep attribution and many more features. Teams and enterprises: contact [email protected] or visit https://headroomlabs.ai. - [Headroom for Teams](https://docs.headroomlabs.ai/docs/teams): Open source vs Enterprise, and how to reach us. - [headroomlabs.ai](https://headroomlabs.ai): Company site. Email [email protected]. ## Licensing Apache 2.0. Use commercially, modify, redistribute. Data stays on the user's machine when running the library, proxy, or MCP server locally. An anonymous usage beacon is **on by default**: it reports compression ratios, counters, provider and model IDs, OS and architecture, and never prompts, completions, code or file paths. Turn it off with `HEADROOM_BEACON=off`, `DO_NOT_TRACK=1`, or `HEADROOM_OFFLINE=1`. See [Telemetry](https://github.com/headroomlabs-ai/headroom#telemetry). - Makefile (6962 bytes)
# Headroom Rust build targets. `just` is not installed on dev boxes; this # Makefile is the source of truth and is mirrored by .github/workflows/rust.yml. SHELL := /bin/bash CARGO ?= cargo MATURIN ?= maturin PYTHON ?= python3 FIXTURES ?= tests/parity/fixtures .PHONY: help test test-parity bench build-proxy build-wheel fmt fmt-check lint clippy clean ci-precheck ci-precheck-rust ci-precheck-python ci-precheck-commitlint install-git-hooks verify-rust-core help: @echo "Headroom Rust targets:" @echo " make test - cargo test --workspace" @echo " make test-parity - parity-run against recorded fixtures" @echo " make bench - cargo bench --workspace" @echo " make build-proxy - release build + strip headroom-proxy, print size" @echo " make build-wheel - release wheel for headroom-py" @echo " make verify-rust-core - build + install + import-verify headroom._core" @echo " make fmt - cargo fmt --all" @echo " make fmt-check - cargo fmt --all -- --check" @echo " make lint - cargo clippy --workspace -- -D warnings" @echo " make clean - cargo clean" @echo "" @echo "E2e targets:" @echo " make build-e2e-wrap - build the wrap-e2e Docker image" @echo " make run-e2e-wrap - build + run the wrap-e2e Docker container" @echo "" @echo "Pre-push verification (run BEFORE git push to catch CI failures locally):" @echo " make ci-precheck - run all CI gates (rust + python + commitlint)" @echo " make ci-precheck-rust - cargo fmt --check + clippy + test" @echo " make ci-precheck-python - smart_crusher-affected python tests" @echo " make ci-precheck-commitlint - lint commits since origin/main" @echo " make install-git-hooks - install pre-commit, commit-msg, and pre-push hooks" test: $(CARGO) test --workspace # headroom-parity has no pyo3 dependency — its comparators call headroom-core # directly, so this target needs neither a venv nor a built extension module. # (See crates/headroom-parity/Cargo.toml: "Phase 0 does not invoke Python from # Rust.") Dropping the `maturin develop` step keeps the harness runnable from a # bare checkout and takes the Python toolchain off the CI parity job. test-parity: $(CARGO) run -p headroom-parity -- run --fixtures $(FIXTURES) bench: $(CARGO) bench --workspace build-proxy: $(CARGO) build --release -p headroom-proxy @BIN=target/release/headroom-proxy; \ if command -v strip >/dev/null 2>&1; then strip "$$BIN" || true; fi; \ SIZE=$$(wc -c < "$$BIN"); \ printf 'headroom-proxy: %s bytes (%.1f MiB)\n' "$$SIZE" "$$(echo "$$SIZE / 1048576" | bc -l)" build-wheel: $(MATURIN) build --release -m crates/headroom-py/Cargo.toml # Hotfix-A0: maturin-develop + symlink + import-verify in one shot. Run this # any time you suspect the proxy is silently falling back to Python-only # mode (Finding #2 in HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md). The # proxy itself runs the same check at lifespan startup; this target # exposes it as a developer-facing one-liner. verify-rust-core: @if [ -z "$$VIRTUAL_ENV" ]; then \ echo "error: activate a venv first (e.g. source .venv/bin/activate)"; \ exit 1; \ fi bash scripts/build_rust_extension.sh fmt: $(CARGO) fmt --all fmt-check: $(CARGO) fmt --all -- --check clippy lint: $(CARGO) clippy --workspace -- -D warnings clean: $(CARGO) clean # ─── Pre-push CI gate ────────────────────────────────────────────────────── # # These targets run the same checks GitHub Actions runs, locally. The intent # is: if `make ci-precheck` is green, `git push` will not turn red. The # 2026-04-27 push surfaced five CI breaks (cargo fmt drift, x86_64-apple- # darwin wheel, headroom._core not built in test-extras + smoke-test, # commitlint footer-leading-blank). The first three are caught by the gates # below; the last two are caught by the workflow fixes themselves. # # Run before EVERY `git push`. Install the git hook (one-time) with: # make install-git-hooks ci-precheck: ci-precheck-rust ci-precheck-python ci-precheck-commitlint @echo "" @echo "✅ ci-precheck PASSED — safe to push." ci-precheck-rust: @echo "── ci-precheck-rust ────────────────────────────────────────────" $(CARGO) fmt --all -- --check $(CARGO) clippy --workspace -- -D warnings $(CARGO) test --workspace # Mirrors the smart_crusher-affected test files we expect green on every # push. Builds the Rust extension first because most of these tests # instantiate `SmartCrusher`, which hard-imports `headroom._core`. ci-precheck-python: @echo "── ci-precheck-python ─────────────────────────────────────────" @if [ -z "$$VIRTUAL_ENV" ]; then \ echo "error: activate a venv first (e.g. source .venv/bin/activate)"; \ exit 1; \ fi bash scripts/build_rust_extension.sh $(PYTHON) -m pytest -q \ tests/test_transforms/test_smart_crusher_bugs.py \ tests/test_transforms/test_smart_crusher_rust_parity.py \ tests/test_transforms/test_diff_compressor.py \ tests/test_transforms/test_diff_compressor_rust_parity.py \ tests/test_relevance.py \ tests/test_relevance_extra.py \ tests/test_ccr.py \ tests/test_acceptance.py \ tests/test_critical_fixes.py \ tests/test_quality_retention.py \ tests/test_toin_integration.py # Lint commits since `origin/main`. Requires npx (Node 18+) on PATH. ci-precheck-commitlint: @echo "── ci-precheck-commitlint ─────────────────────────────────────" @if ! command -v npx >/dev/null 2>&1; then \ echo "error: npx not on PATH (install Node 18+ to enable commitlint checks)"; \ exit 1; \ fi @if ! git rev-parse --verify origin/main >/dev/null 2>&1; then \ echo "error: origin/main not fetched (run 'git fetch origin main')"; \ exit 1; \ fi npx --yes --package=@commitlint/cli --package=@commitlint/config-conventional -- \ commitlint --from origin/main --to HEAD --config .commitlintrc.json install-git-hooks: @scripts/install-git-hooks.sh # ─── E2e Docker targets ──────────────────────────────────────────────────── # # The wrap-e2e Dockerfile uses manylinux_2_28_x86_64 as its builder stage, # which only ships amd64 binaries. Pass --platform linux/amd64 explicitly # so the build works on Apple Silicon (requires QEMU emulation). On native # x86_64 hosts the flag is harmless and matches CI behaviour. build-e2e-wrap: docker build --platform linux/amd64 -f e2e/wrap/Dockerfile -t headroom-wrap-e2e . run-e2e-wrap: build-e2e-wrap docker run --rm headroom-wrap-e2e
- plugins/headroom-skill/skills/headroom/SKILL.md (3421 bytes)
--- name: headroom description: Use when a Python or TypeScript app sends large tool outputs, logs, JSON, search results or long chat history to an LLM and the task is to cut input tokens, reduce token cost, fit a context window, or trim tool results without losing errors. Adds the headroom-ai compress() call; in-process, no API key, no model download. --- # Headroom: compress LLM context before the call ## When this applies The user asks to reduce LLM token cost, trim or compress tool outputs or tool results, prune or summarize chat history, fit a prompt into a context window, or stop large logs or JSON from blowing up the prompt. Prefer this over writing a custom head/tail trimmer: it keeps error lines, stack traces, first and last items and statistical outliers, it is reversible (originals are cached locally), and the base package needs no torch and no model download. ## Python (in-process, no network) ```bash pip install headroom-ai # add to requirements.txt / pyproject ``` ```python from headroom import compress result = compress(messages, model=MODEL_NAME) # OpenAI or Anthropic message dicts # call the provider with result.messages instead of messages # log result.tokens_before, result.tokens_after, result.tokens_saved ``` Compression applies to `tool` (and assistant) messages. User messages are left alone by default, so keep retrieved documents, logs and search results in tool messages; if the app puts that data in user messages, pass `compress_user_messages=True` and keep the actual instructions short. Verify without a provider key: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` Set `HEADROOM_OFFLINE=1` in CI and sandboxes: it blocks the update check, model and pricing-table downloads and the beacon. `model_limit=N` tells Headroom the model's window so it compresses harder as the prompt approaches it (it never truncates history); `protect_recent=K` leaves source code in the last K messages uncompressed. ## TypeScript `npm install headroom-ai` gives `compress()` and `withHeadroom()` wrappers for the OpenAI, Anthropic, Gemini and Vercel AI SDK clients, but compression runs in the Python proxy: start it first with `pip install "headroom-ai[proxy]"` and `headroom proxy --port 8787`. If a Python sidecar is not acceptable, say so and stop. ## Coding agents To cut tokens for Claude Code, Codex, Cursor or Copilot themselves (not an app): `uv tool install --python 3.13 "headroom-ai[all]"` then `headroom wrap claude`. ## Do not - Do not claim a savings percentage; print `tokens_before` and `tokens_after`. - Do not use `headroom.integrations.asgi.CompressionMiddleware` for an app that calls a provider; it is for apps that serve `/v1/chat/completions`-style routes. - Do not install `[all]` or `[ml]` unless asked; the base package is enough. Docs index: https://docs.headroomlabs.ai/llms.txt · FastAPI example: https://docs.headroomlabs.ai/docs/fastapi · Repo: https://github.com/headroomlabs-ai/headroom - pyproject.toml (28436 bytes)
[build-system] requires = ["maturin>=1.5,<2.0"] build-backend = "maturin" [project] name = "headroom-ai" version = "0.44.0" description = "Compress tool outputs, logs, JSON and chat history before each LLM call. Fewer input tokens, same answers; in-process, no API key. Library, proxy and MCP server." # PyPI renders this file; GitHub renders README.md. Keep the two in step on the # install commands and the savings claim (README.md "Proof" is the source). readme = "README-pypi.md" license = "Apache-2.0" requires-python = ">=3.11" authors = [ { name = "Headroom Contributors" } ] maintainers = [ { name = "Headroom Contributors" } ] keywords = [ "llm", "openai", "anthropic", "claude", "gpt", "gemini", "token-cost", "reduce-tokens", "input-tokens", "context-window", "token-budget", "tool-output", "tool-results", "chat-history", "trim-messages", "context-compression", "prompt-compression", "context-engineering", "agents", "coding-agents", "mcp", "proxy", "fastapi", "langchain", "litellm", ] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Topic :: Scientific/Engineering :: Artificial Intelligence", "Topic :: Software Development :: Libraries :: Python Modules", "Typing :: Typed", ] dependencies = [ # Core: lightweight compression (SmartCrusher, ContentRouter, CCR, TOIN) "tiktoken>=0.5.0", # Tokenizer for all compressors "pydantic>=2.0.0", # Config and data models # Model registry, pricing and non-core providers; all lazily imported and # ImportError-guarded. 1.96.2 is the first litellm with prebuilt wheels for # Python 3.11-3.14 on Linux, macOS and Windows: older releases either reject # Python 3.14 (GH #956) or fall back to a Rust source build. "litellm>=1.96.2,<2.0", "click>=8.5.0", # CLI framework; PYSEC-2026-2132 fix (command injection in click.edit()) "rich>=13.0.0", # Rich terminal output "opentelemetry-api>=1.45.2", # CVE-2026-114603 tracestate parsing; CVE-2026-126360 traceparent ReDoS # AST-aware code slicing (CodeCompressor); binary wheel. 0.44.1 is excluded: # that PyPI release was a compromised supply-chain build shipping an # info-stealer `sg.exe` (Trojan:Win64/Lazy!MTB) alongside the real binary # (GH #2332). The `!=` keeps every other release installable, including a # future patched one. "ast-grep-cli>=0.30.0,!=0.44.1", "pyyaml>=6.0", # omp wrap: parse/merge omp's models.yml registry "tomlkit>=0.15.2,<1.0", # Loss-minimizing Codex config recovery # Verify upstream TLS through the OS trust store (macOS Keychain, Windows # cert store), where IT installs corporate TLS-inspection roots (Zscaler, # Netskope). Pure Python, no deps; the same library pip uses by default. "truststore>=0.10.0", # Not imported directly: huggingface-hub (via litellm -> tokenizers, so in # every install) uses it for Hub downloads. Before 1.6.0 its worker threads # could re-initialize or re-enter Python while the interpreter shut down # after a download, crashing the process at exit (SIGSEGV in # Py_InitializeEx, or a fatal TstateNULL error). Declared here rather than # as a uv constraint so the floor reaches the published wheel's metadata # and pip upgrades an older copy. Same platform marker as huggingface-hub's # own hf-xet requirement, so it never installs where hf-xet would not. "hf-xet>=1.6.0; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'", # Not imported directly: litellm, huggingface-hub and torch use it. # CVE-2026-112907/112913/112940 are fixed in 4.2.1, but litellm 1.106 caps # filelock <4, so without this floor pip picks that litellm and 3.32.7; # with it pip takes litellm 1.103.7 (uv.lock has 1.101.0). Declared here so # the floor reaches pip installs. "filelock>=4.2.1", ] [project.optional-dependencies] # Proxy server (most common install: pip install headroom-ai[proxy]) proxy = [ "fastapi>=0.100.0", "uvicorn>=0.23.0,<1.0", # LiteLLM provider backends (e.g. openrouter) expect orjson at runtime but # litellm only declares it under its own [proxy] extra (GH #2056). "orjson>=3.9.14; platform_python_implementation != 'PyPy'", "httpx[http2]>=0.24.0", "openai>=2.14.0", # OpenAI API format support # SDK 2 constructor-callback API (GH #2658). 2.2.0 fixes CVE-2026-59951 # and CVE-2026-116684/116685/116686, which affect 2.0.0-2.1.x. "mcp>=2.2.0,<3.0.0", # MCP server (headroom_compress, retrieve, stats) "magika>=0.6.0; python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64'", # ML detection needs ORT wheels "zstandard>=0.20.0", # Decompress zstd request bodies (Codex, etc.) "websockets>=13.0", # WebSocket proxy for /v1/responses (Codex gpt-5.4+) "python-socks[asyncio]>=2.7.1", # SOCKS support when websockets discovers a system proxy # ORT <1.24 deadlocks in Rust fastembed instead of returning an init error. # Python 3.10 keeps Python-only ORT and bypasses native detection. Intel # macOS also needs older ORT because newer releases have no x86_64 wheels. # No ORT wheel exists for Intel macOS Python 3.14+; optional ML detection # falls back to heuristics there. Transitive ORT backends use the same gate. "onnxruntime>=1.24.0; python_version>='3.11' and (sys_platform != 'darwin' or platform_machine != 'x86_64')", "onnxruntime>=1.16.0,<1.24.0; python_version<'3.11' or (python_version<'3.14' and sys_platform == 'darwin' and platform_machine == 'x86_64')", "transformers>=5.5.0,<6.0", # Tokenizer only (for Kompress) # File watcher for live code graph reindexing (--code-graph). watchdog 6.0.0 # publishes no macOS wheel for Python 3.14, so installing it there needs a C # compiler. The watcher is optional and skips itself when watchdog is absent. "watchdog>=4.0.0; sys_platform != 'darwin' or python_version < '3.14'", "sqlite-vec>=0.1.9", # Vector index for memory (--memory). Lightweight, no torch. ] # Production ASGI/WSGI server — Unix-only (gunicorn does not support Windows). # Kept separate from [proxy] so that dev, CI, and Windows users are not forced # to install a non-functional package. Production deployments should use: # pip install headroom-ai[proxy,proxy-prod] proxy-prod = [ "headroom-ai[proxy]", "gunicorn>=21.0.0; sys_platform != 'win32'", ] # Google Vertex through the LiteLLM translation backend # (`--backend vertex` / `vertex_ai` / `litellm-vertex`), which imports `vertexai`. # Kept separate from [proxy] for the same reason as proxy-prod: this pulls # google-cloud-aiplatform and its transitive tree (~175MB installed), and the # proxy's *native* Vertex passthrough routes # (/v1/projects/.../locations/.../publishers/...) do not need it at all. Only # the LiteLLM backend does. Install with: # pip install headroom-ai[proxy,vertex] vertex = [ "google-cloud-aiplatform>=1.38", ] # AST-based code compression (tree-sitter) # tree-sitter-language-pack >=1.0 removed the bundled tree-sitter package and # switched to an incompatible internal node API (.kind vs .type, callable # root_node, etc.). Pin to <1.0.0 so that tree-sitter>=0.25.2 is pulled in # as a transitive dependency and the existing code_compressor.py node-walk # logic continues to work. See issue #1216. code = [ "tree-sitter-language-pack>=0.10.0,<1.0", "tree-sitter>=0.26.0,<0.27", ] # ML-based compression with Kompress (ModernBERT). # (The legacy [llmlingua] extra was removed in 0.9.x — no live code path used it. # Use [ml] for the supported ML compression dependencies.) ml = [ # PyTorch does not publish wheels for macOS 15 x86_64 at this floor, which # makes `headroom-ai[all]` unsatisfiable on Intel Macs (#1931). "torch>=2.12.1; sys_platform != 'darwin' or platform_machine != 'x86_64'", "transformers>=5.5.0,<6.0", # transformers >= 5.x requires huggingface-hub >= 1.5.0,< 3.0; pinning # the floor here prevents Kompress from silently falling back to # "unavailable" when a sibling install (e.g. `pip install # strands-agents`) drags huggingface-hub backwards. "huggingface-hub>=1.5.0,<3.0", ] # Memory system (hierarchical memory with vector search). # Uses the pure-Python sqlite-vec backend by default (VectorBackend.AUTO -> # SQLITE_VEC), so no C++ toolchain is required. The optional HNSW backend lives # in the [vector] extra below; installing it here would make `[all]` (which pulls # [memory]) fail on any machine without a compiler — see #1368. memory = [ "sqlite-vec>=0.1.9", "sentence-transformers>=2.2.0,<7.0; sys_platform != 'darwin' or platform_machine != 'x86_64'", ] # Optional HNSW vector backend. Needs a C++ toolchain to build hnswlib, so it is # kept out of [memory] and [all]; opt in with `pip install headroom-ai[vector]` # and select it via MemoryConfig(vector_backend=VectorBackend.HNSW). The default # sqlite-vec backend needs no compiler. vector = [ "hnswlib>=0.8.0", ] # Qdrant + Neo4j memory backend helpers memory-stack = [ "mem0ai>=2.0.18,<3.0", "qdrant-client>=1.9.0,<2.0", "neo4j>=5.20.0,<7.0", ] # Apple-Silicon GPU (MPS) offload for the memory embedder. Opt in at runtime with # HEADROOM_EMBEDDER_RUNTIME=pytorch_mps. macOS-only; intentionally excluded from [all]. pytorch-mps = [ "torch>=2.12.1; sys_platform == 'darwin'", "sentence-transformers>=2.2.0; sys_platform == 'darwin'", ] # Semantic relevance scoring with embeddings. # Uses `fastembed` (BAAI/bge-small-en-v1.5 by default — 33M params, # 384 dims, ~67 MB quantized ONNX). The Rust scorer (`fastembed` crate) # loads the same pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, so both # languages run the same ONNX file; tests/parity/fixtures/embedding/ # bounds the remaining ONNX Runtime drift. Replaced sentence-transformers in # Stage 3c.1 — fastembed is faster (~2-3x), smaller (no torch # dependency), and outranks all-MiniLM-L6-v2 on MTEB by ~6 points. # >=0.6.0 for `specific_model_path`, which the pinned-snapshot load needs. relevance = [ "fastembed>=0.6.0; python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64'", "numpy>=1.24.0", ] # Image compression (ML-based routing + OCR) # # OCR backend uses ONNX Runtime regardless of Python version. The # rapidocr ecosystem split into two flavors after 1.4.x: # * rapidocr-onnxruntime 1.4.x — bundled-ORT package, capped at # Python <3.13 by its requires-python metadata. Drop-in for our # existing v1 tuple-shaped API call. # * rapidocr 3.x — engine-agnostic core, supports Python 3.13+. # Returns a RapidOCROutput dataclass (txts, scores, boxes, ...). # Needs `onnxruntime` installed separately to use the ORT backend. # # `headroom/image/compressor.py` adapts both API shapes at runtime via # a try/except cascade. See issue #372 for context. image = [ "pillow>=12.3.0", # PYSEC-2026-2253..2257; uv.lock takes 13.0.0 (see override-dependencies) "sentencepiece>=0.1.99", # Required by SigLIP tokenizer (SiglipTokenizer) # Python 3.6–3.12: keep the proven ORT-bundled package directly. # ~15 MB ONNX models auto-downloaded on first use. "rapidocr-onnxruntime>=1.4.0,<2; python_version<'3.13'", # Python 3.13+: rapidocr-onnxruntime is unavailable (its wheels # declare requires-python<3.13). Use the successor `rapidocr` 3.x # core + `onnxruntime` engine; same ORT backend, just split into # two packages. Total install size and inference speed unchanged. "rapidocr>=3.0,<4; python_version>='3.13' and (python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64')", "onnxruntime>=1.7,<2; python_version>='3.13' and (python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64')", ] # Report generation reports = [ "jinja2>=3.0.0", ] # Binary spreadsheet ingestion (.xlsx / .xls -> tabular text) spreadsheet = [ "openpyxl>=3.1.0", # .xlsx "xlrd>=2.0.1", # legacy .xls ] # OpenTelemetry metrics export otel = [ "opentelemetry-sdk>=1.24.0", "opentelemetry-exporter-otlp-proto-http>=1.24.0", ] # any-llm multi-provider backend (requires Python 3.11+) anyllm = [ "any-llm-sdk>=1.0.0; python_version >= '3.11'", ] # LangChain integration langchain = [ "langchain-core>=1.3.3,<4.0", "langchain-openai>=1.1.14,<2.0", ] # LangGraph integration (headroom.integrations.langchain.langgraph) # LangGraph ships its own langchain-core pin, so this only adds langgraph itself # on top of the langchain extra. langgraph = [ "langchain-core>=1.3.3,<4.0", "langchain-openai>=1.1.14,<2.0", "langgraph>=1.2.14,<2.0", ] # Agno agent framework integration agno = [ "agno>=1.0.0", ] # AWS Strands Agents SDK integration strands = [ "strands-agents>=0.1.0", ] # CrewAI: no extra on purpose. The integration only wraps CrewAI tools, so it # uses whatever CrewAI the user already has (`pip install headroom-ai crewai`). # An extra here would put CrewAI's own dependency tree (including chromadb, # which has unpatched server-side CVEs) into our lockfile and security alerts. # AutoGen agent framework integration autogen = [ "autogen-agentchat>=0.7", ] # MCP server for Claude Code integration mcp = [ "mcp>=2.2.0,<3.0.0", "httpx>=0.24.0", "starlette>=0.27.0", "uvicorn>=0.23.0,<1.0", ] # Voice filler detection voice = [ "onnxruntime>=1.24.0; python_version>='3.11' and (sys_platform != 'darwin' or platform_machine != 'x86_64')", "onnxruntime>=1.16.0,<1.24.0; python_version<'3.11' or (python_version<'3.14' and sys_platform == 'darwin' and platform_machine == 'x86_64')", "transformers>=5.5.0,<6.0", "torch>=2.12.1; sys_platform != 'darwin' or platform_machine != 'x86_64'", ] # Voice training (includes voice deps + training extras) voice-train = [ "headroom-ai[voice]", "datasets>=5.0.1", "accelerate>=0.20.0", ] # Evaluation framework evals = [ "datasets>=5.0.1", "sentence-transformers>=2.2.0,<7.0; sys_platform != 'darwin' or platform_machine != 'x86_64'", "numpy>=1.24.0", "scikit-learn>=1.3.0", "anthropic>=0.18.0", "openai>=1.0.0", ] # AWS Bedrock backend bedrock = [ # `aws login` (IAM Identity Provider / console-login, DPoP) requires # boto3 >= 1.41.0 AND the AWS Common Runtime (CRT) per AWS docs # ("Boto3 1.41.0 or later with CRT"). C - README-pypi.md (3284 bytes)
# Headroom (`headroom-ai`) Compress tool outputs, logs, JSON and chat history before each LLM call, so the model sees fewer input tokens and the same facts. Runs in-process, in Python, with no API key and no model download. Originals are cached locally and retrievable. Measured: 21–57% fewer input tokens on four agent scenarios, 90%+ on repeated JSON arrays and log lines, little on prose. Numbers and how to reproduce them: https://docs.headroomlabs.ai/docs/benchmarks and the "Proof" section of https://github.com/headroomlabs-ai/headroom ## Use it from a Python app that calls an LLM ```bash pip install headroom-ai ``` ```python from headroom import compress result = compress(messages, model="gpt-4o") # OpenAI or Anthropic message dicts result.messages # same shape as the input, fewer tokens result.tokens_before, result.tokens_after, result.tokens_saved ``` Pass `result.messages` to your existing client call instead of `messages`. Nothing else changes. Error lines, stack traces, first and last items and statistical outliers are kept; repeated and low-information items are dropped. Zero-network check, no provider key needed: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` Prints on the order of `20000 -> 95 FATAL kept: True`. `HEADROOM_OFFLINE=1` blocks every connection Headroom would make on its own: the update check, model and pricing-table downloads, and the anonymous usage beacon the proxy and MCP server send by default (`HEADROOM_BEACON=off` turns off only the beacon). FastAPI example with a `POST /chat` endpoint: https://docs.headroomlabs.ai/docs/fastapi ## Use it around a coding agent (no code changes) ```bash uv tool install --python 3.13 "headroom-ai[all]" # CLI in its own env; pip install "headroom-ai[proxy]" also works headroom wrap claude # also: codex, grok, copilot, cursor, aider, opencode, cline, continue, goose, openhands, openclaw, vibe, omp, zcode headroom proxy --port 8787 # or run the proxy and point any client at it ``` ## Other entry points - OpenAI, Anthropic, LiteLLM, LangChain, Agno, Strands, CrewAI, AutoGen adapters: https://docs.headroomlabs.ai/docs (Integrations) - MCP server (`headroom_compress`, `headroom_retrieve`, `headroom_stats`): https://docs.headroomlabs.ai/docs/mcp - TypeScript SDK (`npm install headroom-ai`, needs the proxy running): https://www.npmjs.com/package/headroom-ai - Full docs index for agents: https://docs.headroomlabs.ai/llms.txt - Source and issues: https://github.com/headroomlabs-ai/headroom Extras: `[proxy]` adds the HTTP proxy and `headroom wrap`, `[mcp]` the MCP server, `[ml]` the Kompress text model (torch), `[all]` everything. The base install is enough for `compress()`. Requires Python 3.11+. Apache 2.0. The import and CLI name is `headroom`; the PyPI distribution named `headroom` is a different project. - README.md (37166 bytes)
<div align="center"> <img src="https://raw.githubusercontent.com/headroomlabs-ai/headroom/main/.github/assets/hero.svg" alt="Headroom — the context compression layer for AI agents. A 55,957 token agent prompt compresses to the 24,340 tokens actually sent to the model, and the FATAL line at item 67 survives byte for byte." width="880"> <a href="https://trendshift.io/repositories/20881" target="_blank"><img src="https://trendshift.io/api/badge/repositories/20881" alt="headroomlabs-ai/headroom | Trendshift — #1 Repository Of The Day" width="250" height="55"/></a> <p> <a href="https://github.com/headroomlabs-ai/headroom"><img src="https://img.shields.io/github/stars/headroomlabs-ai/headroom?style=flat&color=00F0B5&labelColor=0C1118&label=stars" alt="GitHub stars"></a> <a href="https://github.com/headroomlabs-ai/headroom/actions/workflows/ci.yml"><img src="https://github.com/headroomlabs-ai/headroom/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://pypi.org/project/headroom-ai/"><img src="https://img.shields.io/pypi/v/headroom-ai.svg?color=00F0B5&labelColor=0C1118&label=pypi" alt="PyPI"></a> <a href="https://www.npmjs.com/package/headroom-ai"><img src="https://img.shields.io/npm/v/headroom-ai.svg?color=00F0B5&labelColor=0C1118&label=npm" alt="npm"></a> <a href="https://huggingface.co/chopratejas/kompress-v2-base"><img src="https://img.shields.io/badge/model-kompress--v2--base-65D8FF?labelColor=0C1118" alt="Model"></a> <a href="https://docs.headroomlabs.ai/docs"><img src="https://img.shields.io/badge/docs-online-00F0B5?labelColor=0C1118" alt="Docs"></a> <a href="https://github.com/headroomlabs-ai/headroom/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-98A4B3?labelColor=0C1118" alt="License"></a> </p> <p> <b><a href="https://docs.headroomlabs.ai/docs/quickstart">Quickstart</a></b> · <b><a href="#headroom-for-teams">Headroom for Teams</a></b> · <a href="#get-started-60-seconds">Install</a> · <a href="#proof-measured-token-savings">Proof</a> · <a href="#agent-compatibility">Agents</a> · <a href="https://docs.headroomlabs.ai/docs">Docs</a> · <a href="https://discord.gg/yRmaUNpsPJ">Discord</a> · <a href="https://github.com/headroomlabs-ai/headroom/blob/main/llms.txt">llms.txt</a> </p> <sub><b>AI agents / LLMs:</b> read <a href="https://github.com/headroomlabs-ai/headroom/blob/main/llms.txt"><code>/llms.txt</code></a> here, or fetch <a href="https://docs.headroomlabs.ai/llms.txt">the live index</a> · <a href="https://docs.headroomlabs.ai/llms-full.txt">full docs blob</a>.</sub> </div> <!-- mcp-name: io.github.headroomlabs-ai/headroom --> **Headroom** (`headroom-ai` on PyPI and npm) compresses everything your AI agent reads — tool outputs, logs, RAG chunks, files, and conversation history — before it reaches the LLM. Same answers, fewer input tokens. Measured: 21–57% fewer input tokens on four agent scenarios, 90%+ on repeated JSON arrays and log lines, little on prose ([Proof](#proof-measured-token-savings)). Compression runs on your machine; no prompt or file content is sent anywhere to be compressed. ## Start here **My app calls an LLM** (Python, LangChain, LiteLLM, Agno, Strands): ```bash pip install headroom-ai ``` ```python from headroom import compress result = compress(messages, model="gpt-4o") # OpenAI or Anthropic message dicts client.chat.completions.create(model="gpt-4o", messages=result.messages) ``` No API key, no model download, in-process. Set `HEADROOM_OFFLINE=1` for a zero-network run. Tool and assistant messages are compressed; user messages are left alone unless you pass `compress_user_messages=True`. [FastAPI example](https://docs.headroomlabs.ai/docs/fastapi) · [API reference](https://docs.headroomlabs.ai/docs/api-reference). TypeScript apps: `npm install headroom-ai` plus the proxy (`pip install "headroom-ai[proxy]"`, `headroom proxy --port 8787`); the npm SDK sends messages to it. [TypeScript quickstart](https://docs.headroomlabs.ai/docs/quickstart) **I run coding agents** (Claude Code, Codex, Cursor, Copilot, OpenCode, Aider): ```bash uv tool install --python 3.13 "headroom-ai[all]" headroom wrap claude ``` Zero code changes. [Agent compatibility](#agent-compatibility) · [Proxy](https://docs.headroomlabs.ai/docs/proxy) <p align="center"> <a href="https://headroomlabs.ai"><img src="https://raw.githubusercontent.com/headroomlabs-ai/headroom/main/.github/assets/enterprise-perk.svg" alt="Headroom Enterprise for teams. Talk to us: [email protected]." width="880"></a> </p> <div align="center"> <img src="https://raw.githubusercontent.com/headroomlabs-ai/headroom/main/HeadroomDemo-Fast.gif" alt="Headroom compressing a 10,144 token log dump to 1,260 tokens while preserving the FATAL line" width="820"> <br><sub>10,144 → 1,260 tokens. The same <code>FATAL</code> found.</sub> </div> ## What it does - **Library** — `compress(messages)` in Python or TypeScript, inline in any app. - **Proxy** — `headroom proxy --port 8787`, zero code changes, any language. - **Agent wrap** — `headroom wrap claude|codex|grok|copilot|cursor|aider|opencode|cline|continue|goose|openhands|openclaw|vibe|omp|droid|zcode` in one command; undo with `headroom unwrap <tool>`. - **MCP server** — `headroom_compress`, `headroom_retrieve`, `headroom_stats` for any MCP client. - **Cross-agent memory** — one shared store across Claude, Codex, Gemini and Grok, with automatic dedup. - **`headroom learn`** — mines failed sessions and writes corrections to `CLAUDE.local.md` (default, gitignored), `CLAUDE.md`, `AGENTS.md`, `GEMINI.md` or `GROK.md`. - **Output token reduction** — trims what the model *writes back*, not only what you send. See [below](#output-token-reduction). - **Reversible (CCR)** — originals are cached locally and retrieved on demand. ## How it works ``` Your agent / app (Claude Code, Cursor, Codex, LangChain, Agno, Strands, your own code…) │ prompts · tool outputs · logs · RAG results · files ▼ ┌────────────────────────────────────────────────────┐ │ Headroom (runs locally — your data stays here) │ │ ──────────────────────────────────────────────── │ │ CacheAligner → ContentRouter → CCR │ │ ├─ SmartCrusher (JSON) │ │ ├─ CodeCompressor (AST) │ │ └─ Kompress-v2-base (text, HF) │ │ │ │ Cross-agent memory · headroom learn · MCP │ └────────────────────────────────────────────────────┘ │ compressed prompt + retrieval tool ▼ LLM provider (Anthropic · OpenAI · Bedrock · …) ``` - **ContentRouter** detects the content type and selects a compressor for it. - **SmartCrusher / CodeCompressor / Kompress-v2-base** handle JSON, source code and prose respectively. - **CacheAligner** flags volatile content that would bust a provider KV-cache prefix. It never rewrites prompts. - **CCR** stores originals locally so the model can call `headroom_retrieve` when it needs the full text. → [Architecture](https://docs.headroomlabs.ai/docs/architecture) · [CCR](https://docs.headroomlabs.ai/docs/ccr) · [Kompress-v2-base model card](https://huggingface.co/chopratejas/kompress-v2-base) ## Get started (60 seconds) ```bash # 1 — Install uv tool install --python 3.13 "headroom-ai[all]" # CLI in a self-contained env pip install "headroom-ai[all]" # Python — ships the `headroom` CLI npm install headroom-ai # TypeScript SDK only — no CLI # 2 — Pick a mode headroom deploy # turnkey local deployment + agent config headroom wrap claude # wrap a coding agent headroom proxy --port 8787 # drop-in proxy, zero code changes # or: from headroom import compress # inline library # 3 — Check it and watch the savings headroom doctor # health check — confirms routing works headroom perf headroom dashboard # live savings (proxy must be running) ``` ### Reduce input tokens in your own app ```python from headroom import compress from openai import OpenAI messages = [{"role": "user", "content": "Analyze these results"}] result = compress(messages, model="gpt-4o") client = OpenAI() response = client.chat.completions.create(model="gpt-4o", messages=result.messages) print(f"Saved {result.tokens_saved} tokens ({result.compression_ratio:.0%})") ``` Launch a wrapped agent session each time, so the setup runs. `headroom wrap` starts a local proxy and launches the agent configured to route through Headroom. Add `--code-memory serena` to also install **[Serena](https://github.com/oraios/serena)** for semantic code navigation (off by default). For Claude Code, Serena is registered for the wrapped project only (as a `local`-scope MCP server in `~/.claude.json`); use `--code-memory-scope user` to make it available in every project. `headroom unwrap` removes either registration. `headroom wrap ...` is session-scoped: its proxy exits with the wrapped agent. For a proxy that survives reboots, use the turnkey `headroom deploy` command. Use `headroom install apply` when you need to choose a specific persistent service, scheduled task, Docker runtime, scope, or provider target; see the [persistent installs guide](https://docs.headroomlabs.ai/docs/persistent-installs). The `headroom` CLI ships only in the PyPI package. The npm `headroom-ai` package is the TypeScript SDK — a library you import (`import { compress } from 'headroom-ai'`) — and provides no `headroom` command. ## Proof: measured token savings Four scenarios built from real MCP server output formats, measured with the provider tokenizer and the shipped `compress()` at its default configuration. Seeded and offline, so you get the same numbers we did: ```bash uv run python benchmarks/index_proof_table.py --seed 20260902 ``` | Scenario | Before | After | Saved | |---|---:|---:|---:| | Code search (100 results) | 17,199 | 13,597 | **21%** | | SRE incident debugging | 55,957 | 24,340 | **57%** | | Codebase exploration | 58,801 | 33,895 | **42%** | | GitHub issue triage | 46,067 | 32,429 | **30%** | Savings scale with how repetitive the payload is. Repeated JSON arrays and log lines clear 90% in `benchmarks/bench_latency.py`; prose and already-dense output compress very little. Run `headroom savings` against your own traffic for the number that applies to you. Compression costs **well under a millisecond** — 0.21 ms p50 on a 10K-token JSON search result, 1.4 ms at 100K tokens — so it does not show up in agent latency. **Accuracy.** `python -m headroom.evals suite --tier 1`: | Benchmark | Category | N | Baseline | Headroom | Delta | |---|---|---:|---:|---:|---| | GSM8K | Math | 100 | 0.870 | 0.870 | ±0.000 | | TruthfulQA | Factual | 100 | 0.530 | 0.560 | +0.030 | | SQuAD v2 | QA | 100 | — | 97% | at 19% compression | | BFCL | Tools | 100 | — | 97% | at 32% compression | At N=100 a delta of ±0.03 falls inside the confidence interval, so TruthfulQA shows no detectable difference rather than an improvement. [Methodology →](https://docs.headroomlabs.ai/docs/benchmarks) ## Output token reduction Everything above shrinks the prompt you **send**. You also pay for every token the model **writes back**, and on Opus-class models output costs 5× input. Much of that output is ceremony: "Great, let me…" preambles, code re-printed straight back at you, and deep reasoning spent on routine steps like reading a file. Headroom trims it from the proxy, with no change to your code: - **Verbosity steering** appends a short "be terse, don't restate context" note to the *end* of the system prompt, so your prompt cache still hits. It applies to Anthropic `/v1/messages` and to OpenAI-compatible `/v1/chat/completions` and `/v1/responses`, with the same `output_shaper:*` labels on all paths. ```bash export HEADROOM_OUTPUT_SHAPER=1 # off by default headroom proxy --port 8787 ``` > **Already running a proxy?** These switches are read live on every request, so > a proxy that `headroom wrap` *reused* rather than started would not see a value > you export afterwards — its environment was snapshotted at launch. `headroom > wrap` hot-syncs your current settings to the running proxy over a loopback > `POST /admin/runtime-env`, so they take effect with no restart and no dropped > requests. On a shared proxy these overrides are global; the last explicit > setting wins. **Terseness you didn't have to configure.** People rarely state how terse they want answers — they show it, by interrupting long replies or moving on before they could have read them. `headroom learn --verbosity` reads past sessions and picks the level: ```bash headroom learn --verbosity # dry run — preview what it found headroom learn --verbosity --apply # save it and apply it to the running proxy ``` A proxy in cache mode (the default) never re-reads the learned level, because a level that changes mid-conversation would bust the prompt cache. There `--apply` pins it with `HEADROOM_VERBOSITY_LEVEL`; set that variable yourself to keep it after a restart. **Measuring it.** Output savings are counterfactual — we never see what the model *would* have written — so Headroom reports an estimate with a confidence range and labels it as one: ```bash headroom output-savings # Reduction: 31.7% (95% CI 27.7% … 35.7%) [estimated] ``` For a measured number instead, hold out 10% of conversations as an unshaped control: `export HEADROOM_OUTPUT_HOLDOUT=0.1`. The dashboard's **Output Tokens Saved** card then reads `measured` rather than `estimated`, with the band. → [Output token reduction](https://docs.headroomlabs.ai/docs/savings) ## Agent compatibility | Agent | `headroom wrap` | Notes | |---|:---:|---| | Claude Code | ✅ | `--memory` · `--code-graph` · `--1m` · `--tool-search` | | Codex | ✅ | shares memory with Claude | | Grok CLI | ✅ | routes via `GROK_MODELS_BASE_URL` | | Cursor | Manual setup | starts the proxy and prints base URLs for Cursor settings | | Aider | ✅ | starts proxy + launches | | Copilot CLI | ✅ | starts proxy + launches | | VS Code Copilot | ✅ | transparent proxy; keeps the selected model | | OpenClaw | ✅ | installs as a ContextEngine plugin | | OpenCode | ✅ | injects config · starts proxy + launches [^third-party-upstream] | | Cline | ✅ | starts proxy + injects config | | Continue | ✅ | starts proxy + injects config | | Goose | ✅ | starts proxy + launches | | OpenHands | ✅ | starts proxy + launches | | Mistral Vibe | ✅ | starts proxy + launches | | Oh My Pi | ✅ | Anthropic + OpenAI/Codex · injects config · starts proxy + launches | | Factory Droid | ✅ | routes via `FACTORY_API_BASE_URL`; Anthropic-model turns compressed, other Factory traffic passed through | | Cortex Code | Library only | 60–65% savings in library mode; no `wrap` | | Kimi CLI | ✅ | OAuth bearer for
- REALIGNMENT/00-overview.md (7660 bytes)
# 00 — Overview & Wrong Mental Model ## Executive summary Headroom is built on the wrong mental model: **"compression means choosing what to drop from conversation history."** The flagship `IntelligentContextManager` (ICM) tokenizes the entire `messages` array, scores each message for importance, and removes old messages until the budget is hit. It has been wired into the Rust proxy on `/v1/messages` with `frozen_message_count: 0` hardcoded — so every compression event drops messages from index 0, busting the Anthropic prompt cache for every customer that triggers it. The correct mental model — confirmed by an authoritative engineering guide and ten parallel deep-audit subagents — is the opposite: **"passthrough is sacred; compress only the live zone, type-aware, hash-keyed, position-preserving, with side-channel metadata."** The cache hot zone (system prompt, tools, old turns, reasoning/thinking/redacted/compaction items) is **never** touched. The audit found: - **5 top-tier cache-killer bugs** all stemming from the wrong model - **~10 K LOC of architectural over-build** (ICM + scoring + relevance + rolling-window + progressive-summarizer + tool-crusher + cache-aligner rewrite path + most of `crates/headroom-core/src/{context,scoring,relevance}/`) - **Wire-format gaps** in the streaming SSE parser (missing `thinking_delta`, `signature_delta`, `citations_delta`; UTF-8-split corruption; single-`\n` SSE split bugs in fallback paths) - **Bedrock/Vertex parity is fake** — a lossy LiteLLM Anthropic-to-OpenAI conversion drops `thinking`, `redacted_thinking`, `document`, `search_result`, `image`, `server_tool_use`, `mcp_tool_use` blocks - **No tool-definition normalization** anywhere - **No auth-mode awareness** — PAYG, OAuth, and subscription CLIs all get the same policy and the same fingerprint-leaking re-serialization - **`X-Headroom-*` request headers leak upstream**, plus `anthropic-beta` mutation and `OpenAI-Beta` auto-injection — fingerprint-class subscription-revocation risks - **CCR markers** are computed but never injected into the outgoing request body in the Rust path; the `ccr_retrieve` tool flips on/off per request — busts the tools array on every state change ## What changes The realignment is structured in 9 phases, 40 PRs, ~13 weeks sequential or ~8 weeks with parallel work: - **Phase A — Lockdown (1 week):** stop the cache bleeding immediately. Make `/v1/messages` compression a passthrough; stop mutating the system prompt; switch Python forwarders from `httpx ... json=body` (re-serializes) to `httpx ... content=raw_bytes`; honor customer-set `cache_control` markers in Rust; strip `x-headroom-*` from upstream-bound headers; pin `anthropic-beta` order and make it session-sticky; add a SHA-256 byte-faithful round-trip test. - **Phase B — Live-zone engine (2 weeks):** delete ICM, scoring, relevance, rolling-window, progressive-summarizer, tool-crusher (~10 K LOC). Build a live-zone-only block dispatcher in Rust that runs SmartCrusher / LogCompressor / DiffCompressor / SearchCompressor / KompressCompressor on the latest user message content + latest tool_result + latest function_call_output + latest local_shell_call_output. Token-validate every compression with fallback. CCR hardens: persistent backend + always-on `ccr_retrieve` tool registration. - **Phase C — Rust proxy paths (3 weeks):** byte-level SSE parser with full state machine; `/v1/chat/completions`, `/v1/responses` (HTTP and streaming) handlers; per-item-type passthrough preservation (V4A patches, `local_shell_call.action.command` argv, Codex `phase` field, MCP items, `compaction`). - **Phase D — Bedrock/Vertex native (2 weeks):** delete the LiteLLM lossy converter; build native `/model/.../invoke` (AWS) and `/v1beta1/projects/.../publishers/anthropic/.../streamRawPredict` (GCP) routes with SigV4 + ADC signing. Cache fidelity restored on Bedrock/Vertex traffic. - **Phase E — Phase 3 cache stabilization (1 week):** sort tool array deterministically; sort JSON Schema keys recursively; auto-place up to 4 `cache_control` breakpoints (Anthropic); auto-inject `prompt_cache_key` (OpenAI); volatile-content detector with customer warning (no rewrite); cache-bust drift telemetry. - **Phase F — Auth-mode policy (1 week):** `classify_auth_mode(headers)` helper returning `payg | oauth | subscription`; per-mode compression policy gates; TOIN aggregation key extended to `(auth_mode, model_family, structure_hash)`; conditional `X-Forwarded-*` headers in Rust. - **Phase G — RTK + observability (1 week):** extend wrap CLIs (cline, continue, goose, openhands); wire the dead `tokens_saved_rtk` field; per-invocation RTK Prometheus metrics. - **Phase H — Python retirement (2 weeks):** delete `headroom/proxy/server.py`, all handlers, `responses_converter.py`, `memory_handler.py`, `memory_tool_adapter.py`, `batch.py`, `semantic_cache.py`, all of `headroom/transforms/*` Python (per Phase B); keep CLI wrappers, RTK installer, evals, learn, memory writers, tokenizers, TOIN. - **Phase I — Test infra (continuous, parallel):** SHA-256 round-trip tests; SSE corner-case fixtures (UTF-8 split, ping, all delta types, `[DONE]`, mid-stream error); property tests (no-panic SSE parser, tokens-non-increasing compression); cache-hit-rate continuous metric; promote `ccr` / `log_compressor` / `cache_aligner` parity comparators from `Skipped` stubs to real; make `make test-parity` a per-PR gate. ## Top 5 wrong assumptions 1. **"Compression means choosing what to drop from history."** Implemented as ICM + DropByScoreStrategy + MessageScorer + relevance + scoring + rolling-window + progressive-summarizer. Fix: retire entirely; compress live-zone content only. 2. **"TOIN can influence per-request compression decisions."** `headroom/telemetry/toin.py:853-927` mutates pattern state during a call and returns hints that bias the same-input-bytes decision. Fix: strict observation-only; recommendations published between deploys. 3. **"CCR can mutate the cache hot zone (tools array, system prompt) on demand."** `headroom/ccr/tool_injection.py:302-328` only adds `ccr_retrieve` when content was compressed — tools list flips between requests. `cache_aligner.py:160-262` and `headroom/proxy/server.py:1051` rewrite the system prompt. Fix: register `ccr_retrieve` on every request; route memory injection to the live zone tail; delete the cache_aligner rewrite path. 4. **"Summarizing past turns is a strategy."** `intelligent_context.py:316-353` SUMMARIZE replaces messages with a single summary at the same position — head modification. Fix: delete; offer compaction only as an explicit customer-initiated action. 5. **"ToolCrusher operates on every tool message in history without a frozen check."** `headroom/transforms/tool_crusher.py:106` iterates all tool messages. Fix: delete; ContentRouter covers the use case correctly. ## What's preserved Per your direction: - **TOIN** (Tool Output Intelligence Network) — observation-only refactor; per-tenant key - **CCR** (Compress-Cache-Retrieve) — persistent backend + always-on tool - **Kompress-base** — plain-text §8.6 compressor; stays in Python now, Rust port via `ort` crate later - **ContentRouter** — Python ~2150 LOC, the architecturally correct piece (NOTE: earlier project memory said 53 K lines — that was wrong by 25×; the file is fine) - All per-type compressors: SmartCrusher (Rust 25 files), CodeCompressor, LogCompressor, SearchCompressor, DiffCompressor ## What's deleted ~25 K LOC across two languages. See [01-bug-list.md](./01-bug-list.md) §6 for the full retirement list with file:line evidence. - REALIGNMENT/01-bug-list.md (26439 bytes)
# 01 — Comprehensive Bug & Gap List Ranked P0 (cache-killer) → P5 (long tail). Every entry has: title, file:line, evidence, guide §, fix, ROI estimate. Sources: 10 parallel deep-audit subagents (Rust proxy passthrough; Rust compression correctness; Python proxy + bridges; prefix cache safety; streaming + wire-format; RTK + tests/parity; over-engineering; OpenAI long-tail + Bedrock; Headroom-side injections; auth-mode handling). --- ## P0 — Cache-killer smoking guns (every customer affected) These bugs collapse Anthropic prompt-cache hit rate toward 0% for any traffic that triggers them. Fix in Phase A. ### P0-1. System prompt mutated by `.strip()` and memory-context append - **File:** `headroom/proxy/server.py:1050-1058`; `headroom/proxy/handlers/openai.py:1212` - **Evidence:** `body["system"] = (existing_system + "\n\n" + context).strip()` — strips whitespace and appends dynamic memory context to the cache hot zone on every memory-enabled call. - **Guide:** §1.11 (whitespace fidelity), §6.3 #10 (tiny system prompt edits invalidate cache), §10.1 (system = always cache hot). - **Fix:** Remove `_inject_system_context` path; route memory context to the **first block of the latest user message** (live zone). The existing `_append_context_to_latest_non_frozen_user_turn` already does this — make it the only path. - **ROI:** Restores cache hits for ~all memory-enabled traffic. - **Phase A → PR-A2.** ### P0-2. Every Python forwarder re-serializes JSON via `httpx ... json=body` - **File:** `headroom/proxy/server.py:1088, 1090`; `headroom/proxy/handlers/streaming.py:651`; `headroom/proxy/handlers/openai.py:2392-2397`; `headroom/proxy/handlers/batch.py:344` - **Evidence:** httpx default encoder calls `json.dumps(body, separators=(", ", ": "), ensure_ascii=True)`. Inbound bytes use `,`/`:` and raw UTF-8 in user content; outbound bytes use `, `/`: ` and `\uXXXX` escapes. Bytes never reach upstream byte-equal to bytes that arrived. - **Guide:** §1.9 (the single most expensive proxy mistake), §1.10 (numeric precision), §1.11 (whitespace fidelity). - **Fix:** Switch every forwarder to `httpx ... content=raw_bytes_modified_in_place`. Keep the original `await request.body()` bytes; if a transform mutated the body, re-serialize with `separators=(",", ":")` + `ensure_ascii=False`. Better: surgical byte-fragment replacement on `messages` only, leaving the envelope's bytes untouched. - **ROI:** Restores cache hits for **all** Python-forwarded traffic. - **Phase A → PR-A3.** ### P0-3. Rust proxy ignores customer `cache_control` markers - **File:** `crates/headroom-proxy/src/compression/anthropic.rs:151-156` - **Evidence:** `frozen_message_count: 0` hardcoded with `TODO: detect provider prefix-cached messages from the request. Until we wire that detection, we treat the whole list as droppable.` Combined with ICM, every compression event drops messages from index 0. - **Guide:** §2.19 (up to 4 cache_control markers), §6.2 (cache breakpoints define the prefix). - **Fix:** Walk `messages[*].content[*].cache_control`, `system[*].cache_control`, `tools[*].cache_control`; set `frozen_message_count` to the highest message index that contains a cache_control marker. - **ROI:** Restores cache hits for **all** clients using Anthropic prompt caching (which is virtually all production Anthropic traffic). - **Phase A → PR-A4.** ### P0-4. ICM compresses by dropping messages from cache hot zone (wrong scope) - **File:** `crates/headroom-proxy/src/compression/anthropic.rs:146-157`; `crates/headroom-core/src/context/strategy/drop_by_score.rs:64-80`; `crates/headroom-core/src/context/manager.rs`; `headroom/transforms/intelligent_context.py:354-450` - **Evidence:** ICM with default `keep_last_turns: 2` is allowed to drop any message older than the last two turns. Combined with P0-3, this is a 100%-likely cache-buster on any conversation with ≥3 user turns. - **Guide:** §6.5 (live zone vs hot zone), §10.1 ("Old conversation turns ... never compress"), §6.3 #11 ("Truncation/summarization at the head"), §7.2 (append-only compression). - **Fix:** Delete ICM. Replace with live-zone-only block-level compression. Phase B builds the replacement. - **ROI:** Eliminates the largest single class of cache-bust events. - **Phase A → PR-A1 (stop calling ICM); Phase B → PR-B1 (delete ICM).** ### P0-5. Numeric precision lost via `serde_json::Value` round-trip - **File:** `crates/headroom-proxy/src/compression/anthropic.rs:91, 172` - **Evidence:** Body parsed into `serde_json::Value` and re-serialized via `serde_json::to_vec(&parsed)`. `Value::Number` is `i64|u64|f64` so any `1.0` round-trips to `1`; large integers above 2^53 lose precision. `Cargo.toml:34` enables `preserve_order` only — no `arbitrary_precision`, no `RawValue`. - **Guide:** §1.10. - **Fix:** Add `arbitrary_precision` and `raw_value` features to `serde_json`. Use `&RawValue` for `messages[*]` so individual messages forward as exact byte copies. Strategy outputs only need to be "drop this index" or "replace this block's content." - **ROI:** Closes the second-largest re-serialization byte-drift class. - **Phase A → PR-A4 (jointly with P0-3).** ### P0-6. Memory tool injection toggles tools list and mutates `anthropic-beta` - **File:** `headroom/proxy/memory_handler.py:389-398`; `headroom/proxy/handlers/anthropic.py:1147-1171` - **Evidence:** Memory adds `memory_save`, `memory_search` tools to `body["tools"]` only when memory is enabled for the request. Mid-session config flicker → tool set changes → cache busts (§6.3 #2). Same code mutates `anthropic-beta` adding `context-management-2025-06-27` when injection happens (§6.3 #6). - **Fix:** Make memory tool injection **session-sticky**: once injected, always inject for the lifetime of the session. Pin `anthropic-beta` order; never reorder tokens within the comma-list. - **ROI:** Eliminates mid-session cache busts. - **Phase A → PR-A6, PR-A7.** ### P0-7. `responses_converter.py` drops Codex `phase` field and corrupts multi-text-part rebuild - **File:** `headroom/proxy/responses_converter.py:94, 221-256` - **Evidence:** `phase` field is dropped on the Chat-Completions trip (line 94 maps `role` only); only `copy.copy(original)` accidentally retains it on the rebuild path. Multi-text-part input messages get corrupted: `_extract_text_from_parts` joins with `\n`, `_reconstruct_item:254-256` puts the concatenated text into the first part only and leaves parts 1..N as-is, doubling content. - **Guide:** §4.5 (preserve `phase` exactly), §7.9 (position preservation). - **Fix:** Stash `phase` and restore in `_reconstruct_item`. Rebuild text parts by index, replacing each part's text in place. Better: in Phase C, port `/v1/responses` to Rust and never decompose item structure for compression. - **Phase A → PR-A8 (Python hotfix), Phase C → PR-C5 (full rebuild).** --- ## P1 — Wire-format / streaming corruption ### P1-8. SSE buffers decoded with `errors="ignore"` / `errors="replace"` - **File:** `headroom/proxy/handlers/streaming.py:58, 772`; `headroom/ccr/response_handler.py:672` - **Evidence:** `chunk.decode("utf-8", errors="ignore")` silently drops emoji/CJK bytes split across TCP reads. The wire passthrough at `streaming.py:788` is bytes (correct), but every `_parse_sse_usage_from_buffer` and `_parse_sse_to_response` decision is made on a string that may have lost bytes. - **Guide:** §1.4 (UTF-8 multi-byte split across chunks). - **Fix:** Bytes-level buffer; find `\n\n` boundary in bytes; decode each complete event after split. - **Phase C → PR-C1 (Rust SSE parser); Phase A → PR-A8 includes a Python hotfix.** ### P1-9. SSE parser misses `thinking_delta`, `signature_delta`, `citations_delta` - **File:** `headroom/proxy/handlers/streaming.py:213-298` - **Evidence:** Only `text_delta` and `input_json_delta` are switched on (lines 268-271). Thinking blocks reconstructed without text or signature; signature-protected blocks rejected on replay. - **Guide:** §2.5, §2.7, §5.1 transitions table. - **Fix:** Add all delta-type arms. In Rust SSE parser (Phase C), implement guide §5.1 fully. - **Phase A → PR-A8 (Python); Phase C → PR-C1 (Rust).** ### P1-10. Memory continuation re-emitter emits whole `partial_json` in one delta - **File:** `headroom/proxy/handlers/streaming.py:300-391` (`_response_to_sse`) - **Evidence:** `"partial_json": json.dumps(block["input"])` (line 366-374) emits the entire input JSON as a single delta — clients accumulating per-spec receive one giant fragment instead of an incremental stream. Tool IDs are fabricated as `f"toolu_{idx}"` (line 345). Thinking blocks dropped entirely. - **Guide:** §2.6. - **Fix:** Either delete this function (do memory continuation as non-streaming retry) or rewrite to spec. - **Phase B → PR-B6 (memory injection refactor likely deletes it).** ### P1-11. LiteLLM bridge fabricates `toolu_<uuid>` when upstream `tc.id` missing - **File:** `headroom/backends/litellm.py:860` - **Evidence:** `tool_id = tc.id or f"toolu_{uuid.uuid4().hex[:24]}"`. If upstream omits `id` on chunk 1, fake ID is generated and the upstream tool_call_id is lost forever; next turn `tool_result` references the fake ID and pairing breaks. - **Guide:** §3.5, §2.10. - **Fix:** Drop the fallback; surface an error if `tc.id` is None on first appearance. - **Phase D → PR-D1 deletes this whole file.** ### P1-12. OpenAI WS→HTTP fallback uses single-`\n` SSE split - **File:** `headroom/proxy/handlers/openai.py:2422-2447` - **Evidence:** `aiter_text()` decodes UTF-8 chunk-by-chunk → `buffer.split("\n", 1)` instead of `\n\n`. Multi-line `data:` payloads get wrong-split. - **Fix:** Switch to `aiter_bytes()` + bytes-level `\n\n` boundary. - **Phase C → PR-C3 ports this surface to Rust.** ### P1-13. Re-serialization in Rust path even when no body fields mutated - **File:** `crates/headroom-proxy/src/compression/anthropic.rs:90-188` - **Evidence:** When `should_apply` is true and ICM doesn't drop anything (`anthropic.rs:162-168`), the function correctly returns `NoCompression` and forwards original bytes. But the `Compressed` path always re-serializes via `serde_json::to_vec(&parsed)` — even if only one message changed, every retained message gets re-encoded through `Value`. - **Guide:** §1.12. - **Fix:** Use `RawValue` for retained `messages[*]` entries; only the modified message gets re-encoded. - **Phase A → PR-A4 / Phase B → PR-B2 (live-zone replacement).** ### P1-14. Mid-stream `error` events not handled (Anthropic + OpenAI) - **File:** `headroom/proxy/handlers/streaming.py:160-211, 213-298` - **Evidence:** No `event_type == "error"` arm. The wire passthrough is byte-faithful (good) but Headroom's bookkeeping (`stream_state.input_tokens` etc.) silently doesn't reflect the failure; `_finalize_stream_response` reports a clean PERF line for an errored stream. - **Guide:** §1.7, §2.21. - **Fix:** Add `error` handling to telemetry. - **Phase C → PR-C1 (Rust SSE).** ### P1-15. Connection drop without `message_stop`/`[DONE]` not surfaced - **File:** `headroom/proxy/handlers/streaming.py:899` (`finally: await self._finalize_stream_response`) - **Evidence:** The `finally` runs but there's no flag indicating the stream was truncated; logs report a PERF line as if it succeeded. - **Guide:** §1.8. - **Fix:** Track terminator-seen flag; emit truncation telemetry when missing. - **Phase C → PR-C1.** ### P1-16. OpenAI `refusal` field on Chat assistant message not handled - **File:** None (zero references) - **Evidence:** Memory and tool-call extraction look only at `message.content` / `tool_calls`; refusal turns silently look like content==null with output_tokens=0. - **Guide:** §3.7. - **Fix:** Inspect `refusal` field; surface in telemetry. - **Phase C → PR-C2 (Rust /v1/chat/completions).** ### P1-17. `current_block: Optional[dict]` instead of `blocks: HashMap<usize, BlockState>` - **File:** `headroom/proxy/handlers/streaming.py:227, 700` - **Evidence:** Anthropic emits one block at a time today, but the guide explicitly says "track blocks by `index`" — current code captures `index` and never uses it as a key. - **Guide:** §2.4, §5.1. - **Fix:** Index-keyed map. - **Phase C → PR-C1.** --- ## P2 — Architectural over-build ### P2-18. ICM-as-history-dropper (the structural mismatch) - **Files:** `headroom/transforms/intelligent_context.py`; `crates/headroom-core/src/context/manager.rs`; `crates/headroom-proxy/src/compression/icm.rs` - **Status:** Delete in Phase B (PR-B1). ### P2-19. `RollingWindow`, `ProgressiveSummarizer` (head-truncation strategies) - **Files:** `headroom/transforms/rolling_window.py` (395 LOC); `headroom/transforms/progressive_summarizer.py` (508 LOC) - **Guide:** §6.3 #11, §6.4 (compaction is the explicit exception, intended to break cache once). - **Status:** Delete in Phase B (PR-B1). ### P2-20. `MessageScorer`, `scoring/`, `relevance/` machinery - **Files:** `crates/headroom-core/src/scoring/{scorer,score,weights,traits,mod}.rs` (~1500 LOC); `crates/headroom-core/src/relevance/{embedding,bm25,hybrid,base,mod}.rs` (~1600 LOC); `headroom/transforms/scoring.py` (459 LOC) - **Evidence:** Sole consumer is `DropByScoreStrategy::try_fit`. Without ICM, no consumer. - **Status:** Delete in Phase B (PR-B1). MessageScorer Rust port (PR #338, #343) becomes wasted work. ### P2-21. `crates/headroom-core/src/context/` — except `safety.rs` - **Files:** `crates/headroom-core/src/context/{config,workspace,candidate,ccr_drop,manager,strategy/}.rs` (~1500 LOC) - **Status:** Delete in Phase B (PR-B1). `safety.rs` (tool-pair atomicity) is moved to `crates/headroom-core/src/transforms/safety.rs` and kept. ### P2-22. `ToolCrusher` operates without `frozen_message_count` - **File:** `headroom/transforms/tool_crusher.py:106` - **Evidence:** Iterates all result_messages, no frozen check. Crushes any tool message above token threshold regardless of position. - **Guide:** §10.1 (old tool results are cache-hot). - **Status:** Delete in Phase B (PR-B1). ContentRouter covers the use case correctly. ### P2-23. `CacheAligner` rewrite path violates the very thing it claims to stabilize - **File:** `headroom/transforms/cache_aligner.py:160-262` - **Evidence:** Strips dynamic content from system prompt and re-inserts as a context block — mutates the cache hot zone. Currently `enabled=False` in `server.py:299`. - **Guide:** §9.3. - **Fix:** Delete the rewrite path (~400 LOC); keep detector + customer warning (~140 LOC). - **Phase A → PR-A2 includes the deletion.** ### P2-24. Memory-handler injection at request lifecycle entry - **File:** `headroom/proxy/memory_handler.py:498-510`; `headroom/proxy/handlers/openai.py:535-540` - **Evidence:** Prepends a system message with retrieved memories on every turn. Retrieval is non-deterministic (vector store grows turn-to-turn). - **Fix:** Move retrieval out of the request lifecycle; treat as an explicit customer-invoked tool. - **Phase B → PR-B6 (memory refactor).** ### P2-25. CCR `ccr_retrieve` tool injected only when content w - REALIGNMENT/02-architecture.md (17997 bytes)
# 02 — Realigned Target Architecture The Rust-only proxy after Phase H. Each subsystem documented with its scope, invariants, file layout, and what it explicitly does NOT do. --- ## 2.1 Request lifecycle (Rust, post-Phase-C) ``` Client request │ ▼ ┌──────────────────────────────────────────────┐ │ headroom-proxy (axum) │ │ │ │ 1. classify_auth_mode(headers) │ ← Phase F │ → "payg" | "oauth" | "subscription" │ │ │ │ 2. strip x-headroom-* from upstream-bound │ ← Phase A (PR-A5) │ │ │ 3. byte-buffer body via RawValue │ ← Phase A (PR-A4) │ (numeric precision preserved) │ │ │ │ 4. honor cache_control markers │ ← Phase A (PR-A4) │ → frozen_message_count │ │ │ │ 5. live_zone_compress(body, frozen_count, │ ← Phase B │ auth_mode) │ │ ├─ identify live-zone blocks │ │ ├─ per-block content-type detection │ │ ├─ dispatch to type-aware compressor │ │ ├─ token-validate; fallback to original │ │ ├─ CCR: hash-key, store, marker │ │ └─ replace block bytes in-place │ │ │ │ 6. tool_def_normalize(body) │ ← Phase E (PR-E1, E2) │ ├─ alpha-sort tools[] │ │ └─ recursive-sort JSON Schema keys │ │ │ │ 7. cache_control_auto_place(body) │ ← Phase E (PR-E3) │ (Anthropic; up to 4 ephemeral) │ │ │ │ 8. prompt_cache_key_inject(body) │ ← Phase E (PR-E4) │ (OpenAI; only if not customer-set) │ │ │ │ 9. forward via reqwest with original bytes │ │ for unmodified envelope (RawValue diff) │ │ │ │ 10. SSE response: byte-level state machine │ ← Phase C (PR-C1) │ ├─ track blocks/items by id │ │ ├─ all delta types handled │ │ ├─ mid-stream error/ping/drop surfaced │ │ └─ pure passthrough to client │ │ │ │ 11. usage telemetry (cache_read, │ ← Phase G │ cache_creation, output_tokens, etc.) │ └──────────────────────────────────────────────┘ │ ▼ Upstream provider ``` --- ## 2.2 The cache-safety invariants (every PR enforces) ### Invariant I1 — Byte-faithful passthrough on unmutated bytes For every request, the bytes sent to upstream are byte-equal (SHA-256) to the bytes received from the client, **modulo only the byte ranges that a transform explicitly modified**. No re-serialization through a `Value` type. No JSON-prettifier whitespace insertion. No `\uXXXX` ASCII escaping of UTF-8 user content. **Implementation:** `serde_json::value::RawValue` for `messages[*]` entries; modified messages get fresh serialization, retained messages forward as exact byte copies. Workspace `Cargo.toml` adds `arbitrary_precision` + `raw_value` features. **Test gate:** `proxy_byte_faithful_anthropic_sha256` — record a real Anthropic `/v1/messages` payload, send it through the proxy with compression off, assert SHA-256 byte-equal at the upstream mock. ### Invariant I2 — Cache hot zone never modified The following are never mutated by Headroom: - `system` (string or block list) - `tools[*]` (other than alpha-sorting and JSON Schema key sorting in Phase E — both deterministic) - Any message at index < `frozen_message_count` - Reasoning items with `encrypted_content` - Thinking blocks with `signature` - `redacted_thinking.data` - Compaction items (`{"type": "compaction", "encrypted_content": ...}`) **Implementation:** `live_zone_compress` walks `messages` from the tail, identifies live-zone blocks (latest user message, latest tool_result, latest function_call_output, latest local_shell_call_output, latest apply_patch_call_output), and ONLY modifies bytes within those blocks. **Test gate:** `cache_hot_zone_unchanged_under_compression` — fixture with system + tools + 5 historical turns + new tool_result; assert system + tools + first 5 turns bytes equal at upstream. ### Invariant I3 — Append-only Once a message has appeared in any prior request to upstream, its bytes are frozen. Compression operates on the live zone (latest turn) only. **Implementation:** `frozen_message_count` is the floor; any compressor that touches index < `frozen_message_count` is rejected at compile time (Rust trait constraint) or runtime (Python assertion). **Test gate:** `append_only_invariant_under_recompression` — same input bytes through the compressor twice produces byte-equal output; retained messages are byte-equal across the two runs. ### Invariant I4 — Determinism For the same `(input bytes, frozen_count, auth_mode)`, the compressor produces byte-equal output. No timestamps, no random seeds, no time-dependent decisions. **Implementation:** - TOIN is observation-only (Phase B PR-B5); it never alters request-time decisions. - All hashing is BLAKE3 / SHA-256 with stable input ordering. - Sort orders are explicit (`BTreeMap` for output, never `HashMap`). - No `Instant::now()` in any compression code path. **Test gate:** Property test — for arbitrary valid input, `compress(input) == compress(compress(input).original)` (idempotence on already-compressed); `compress(input) == compress(input)` (run-to-run determinism). ### Invariant I5 — Token-aware, not byte-aware Every compression is validated post-compression with a tokenizer. If `compressed.tokens >= original.tokens`, the original is forwarded. **Implementation:** Phase B PR-B4. Per-content-type byte thresholds: code>2KB, JSON>1KB, logs>500B, plain text>5KB. Below threshold = no compression attempted (overhead exceeds savings). **Test gate:** `proptest_compression_token_count_non_increasing` — for arbitrary valid inputs from a strategy, `tokens(output) ≤ tokens(input)`. ### Invariant I6 — Position-preserving Compression never reorders blocks within a content array, never splits one block into multiple, never adds inline metadata fields to existing blocks. **Implementation:** Compressor signature is `fn(block: &mut Block) -> Result<()>` — operates in place. Block type, `tool_use_id` / `call_id`, `is_error`, all sibling fields preserved. **Side-channel metadata:** A separate marker block (text-type, sibling) carries CCR retrieval directives. Never an extra field on the original block. ### Invariant I7 — Tool definitions normalized, not compressed Tools are sorted alphabetically by name; JSON Schema keys are sorted recursively; description whitespace is normalized. The bytes of each tool definition's `input_schema.properties[*].description` are otherwise preserved. **Implementation:** Phase E PR-E1, PR-E2. ### Invariant I8 — `signature`, `encrypted_content`, `redacted_thinking.data` are sacrosanct These are passthrough only. Never inspected, never decoded, never transformed. **Implementation:** Compressor block-type dispatch has explicit no-op arms for these types. The Bedrock/Vertex native paths (Phase D) preserve them unlike the LiteLLM converter. ### Invariant I9 — TOIN observes, never mutates request bytes TOIN's pattern stats grow across requests. Recommendations are published to disk between deploys. The compressor reads recommendations at startup, not per-request. **Implementation:** Phase B PR-B5. TOIN's in-memory state writes are append-only; reads never block compression. ### Invariant I10 — Auth mode gates compression policy PAYG: aggressive (full live-zone compression, CCR, tool injection, Phase 3 stabilization). OAuth: passthrough-prefer (live-zone lossless only, no auto-`cache_control`, no auto-`prompt_cache_key`, no `X-Forwarded-*`). Subscription: stealth-prefer (everything OAuth does PLUS preserve `accept-encoding`, never inject `X-Headroom-*` upstream, never mutate `User-Agent`). **Implementation:** Phase F PR-F1, PR-F2. --- ## 2.3 The compressor module layout (post-Phase-B) ``` crates/headroom-core/src/ ├── lib.rs # public surface ├── tokenizer/ # KEEP (HF + tiktoken impls) │ ├── mod.rs │ ├── hf_impl.rs │ ├── tiktoken_impl.rs │ ├── estimator.rs │ └── registry.rs ├── ccr.rs # KEEP, hardened (persistent backend) ├── signals/ # KEEP — drives live-zone consumers │ ├── mod.rs │ ├── line_importance.rs │ ├── keyword_detector.rs │ └── tiered.rs ├── transforms/ # the compressors │ ├── mod.rs │ ├── safety.rs # MOVED from context/safety.rs (Phase B) │ ├── live_zone.rs # NEW — live-zone block dispatcher (Phase B) │ ├── content_detector.rs # KEEP │ ├── detection.rs # KEEP │ ├── magika_detector.rs # KEEP │ ├── unidiff_detector.rs # KEEP │ ├── adaptive_sizer.rs # KEEP │ ├── anchor_selector.rs # KEEP │ ├── tag_protector.rs # KEEP │ ├── log_compressor.rs # KEEP │ ├── search_compressor.rs # KEEP │ ├── diff_compressor.rs # KEEP │ ├── kompress_compressor.rs # NEW — Phase H Rust port via `ort` crate │ ├── smart_crusher/ # KEEP (25 files, correctly scoped) │ └── pipeline/ # SHRUNK — only the live-zone orchestrator │ ├── mod.rs │ ├── orchestrator.rs # rewrite to live-zone-only │ ├── traits.rs # LosslessTransform / LossyTransform │ └── offloads/ # KEEP — JSON, log, search, diff offloads └── auth_mode.rs # NEW — Phase F (classify_auth_mode helper) # DELETED in Phase B: # context/ ← except safety.rs which moved # scoring/ # relevance/ ``` ``` crates/headroom-proxy/src/ ├── lib.rs ├── main.rs ├── config.rs ├── error.rs ├── proxy.rs # Phase A: pure passthrough on /v1/messages # Phase C: + /v1/chat/completions, /v1/responses ├── headers.rs # Phase F: conditional X-Forwarded-* ├── websocket.rs # Phase C: WS Codex flow ├── sse/ # NEW — Phase C │ ├── mod.rs │ ├── parser.rs # byte-level state machine │ ├── anthropic.rs # 4-event dance + delta types │ ├── openai_chat.rs # tool_call accumulation │ └── openai_responses.rs # output items + reasoning summary ├── compression/ │ ├── mod.rs # routing by path × auth_mode │ ├── live_zone_anthropic.rs # NEW (Phase B) │ ├── live_zone_openai.rs # NEW (Phase C) │ ├── tool_def_normalize.rs # NEW (Phase E) │ ├── cache_control.rs # NEW (Phase E) │ └── model_limits.rs # KEEP ├── bedrock/ # NEW — Phase D │ ├── mod.rs │ ├── sigv4.rs │ ├── invoke.rs │ └── eventstream.rs ├── vertex/ # NEW — Phase D │ ├── mod.rs │ ├── adc.rs │ └── stream_raw_predict.rs └── observability/ # NEW — Phase G ├── mod.rs ├── prometheus.rs ├── cache_hit_rate.rs └── compression_ratio.rs # DELETED: # compression/icm.rs ← Phase A PR-A1 # compression/anthropic.rs ← Phase A PR-A1 (replaced with live_zone_anthropic.rs in Phase B) ``` --- ## 2.4 The auth-mode policy matrix (Phase F) | Policy aspect | PAYG | OAuth | Subscription | |---|---|---|---| | Live-zone compression | aggressive | lossless-only | lossless-only | | CCR enabled | yes | yes | yes (long-session) | | Tool def alpha-sort | yes | yes | yes | | JSON Schema key sort | yes | yes | yes | | Auto `cache_control` placement | yes | NO (could void scope) | NO | | Auto `prompt_cache_key` injection | yes (OpenAI) | NO | NO | | `anthropic-beta` mutation | NO | NO | NO | | `X-Headroom-*` upstream | NO | NO | NO | | `X-Forwarded-*` upstream | yes | yes | NO | | `User-Agent` rewrite | NO | NO | NO | | `accept-encoding` strip | OK | OK | NO (preserve) | | Lossy compressors (LLMLingua) | OK | NO | NO | | Memory injection | live-zone tail | live-zone tail (gated) | live-zone tail (gated) | | TOIN aggregation key | (mode, model) | (mode, model) | (mode, model) | | `Authorization` log redaction | first 12 chars | first 12 chars | first 12 chars | --- ## 2.5 Preserved primitives detail ### TOIN (post-Phase-B-PR-B5) ```rust // Strict observation-only. pub trait Telemetry { fn record_compression( &self, auth_mode: AuthMode, model: ModelFamily, structure_hash: StructureHash, outcome: CompressionOutcome, ); // No request-time hint API. Period. } // Recommendations published between deploys via: // $ cargo run -p headroom-toin-publish -- --auth-mode payg --model claude-3-7-sonnet // Output: recommendations.toml committed to repo, loaded by compressor at startup. ``` ### CCR (post-Phase-B-PR-B7) ```rust pub trait CcrStore: Send + Sync { fn put(&self, hash: ContentHash, original: Bytes, ttl: Duration) -> Result<()>; fn get(&self, hash: ContentHash) -> Result<Option<Bytes>>; fn purge_expired(&self) -> usize; } pub struct SqliteCcrStore { ... } // primary backend pub struct Re - REALIGNMENT/03-phase-A-lockdown.md (21258 bytes)
# Phase A — Cache-Safety Lockdown **Goal:** Stop the cache-killer bleeding tonight. Each PR is small, low-risk, independently reversible. Zero new architecture; minimum viable fixes only. **Calendar:** 1 week. PR-A1 lands today; A2–A8 over the week. **Shape:** 8 PRs, each on its own branch, each in its own worktree. Sequential dependency only between A1→A4; the rest are parallelizable. --- ## PR-A1 — Make `/v1/messages` compression a passthrough **Branch:** `realign-A1-icm-passthrough` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A1-icm-passthrough` **Risk:** **LOW** (deletion + tests; no new logic) **LOC:** -180 / +30 ### Scope Stop calling ICM from the Rust proxy on `/v1/messages`. The proxy becomes a pure byte-faithful passthrough on this endpoint. Zero compression value temporarily, but eliminates the C1+C2+C3+C4 cache-killer cluster (P0-3, P0-4, P0-5, P1-13). Compression returns in Phase B. ### Files **Delete:** - `crates/headroom-proxy/src/compression/icm.rs` **Modify:** - `crates/headroom-proxy/src/compression/mod.rs` — remove `pub mod icm;`, remove ICM dispatch in `maybe_compress`. The `is_compressible_path` check still matches `/v1/messages` but `compress_anthropic_request` becomes a no-op stub returning `Outcome::NoCompression`. - `crates/headroom-proxy/src/compression/anthropic.rs` — replace function body with `Ok(Outcome::NoCompression)`. Keep the function signature so callers compile; subsequent PRs in Phase B replace this with the live-zone block dispatcher. - `crates/headroom-proxy/src/proxy.rs` — confirm the `Outcome::NoCompression` branch forwards original bytes (already does at line 296-298; just verify with the new test). **Tests added:** - `crates/headroom-proxy/tests/integration_compression.rs::compression_on_message_passes_body_unchanged_sha256` — record a real Anthropic request body to a fixture; send through proxy; assert SHA-256 of upstream-received body equals SHA-256 of inbound body. **Tests deleted/updated:** - Update `compression_on_short_body_passes_through` to assert SHA-256 byte-equality (not just `len()`). - Update `compression_on_long_body_drops_messages` — rename to `compression_on_long_body_passes_through_in_phase_A`. The old assertion (fewer messages arrived) becomes the opposite (same messages arrive). ### Acceptance criteria - `cargo test -p headroom-proxy` green. - New SHA-256 round-trip test passes. - The proxy still starts and serves `/healthz`. - `make ci-precheck` green. ### Blocked by None. Land first. ### Blocks PR-A4 (cache_control honoring needs the ICM call site removed first to avoid conflict). All Phase B PRs (which delete the surrounding code). ### Rollback `git revert` the merge commit. ICM is restored. (Note: this also restores P0-3 and P0-4. Acceptable for ~hours during emergency rollback.) ### Notes - The `compress_anthropic_request` function stays as a stub so Phase B has a single rewrite target. - This PR does NOT delete ICM the module yet — `crates/headroom-core/src/context/manager.rs` still compiles. PR-B1 deletes the modules. Splitting keeps the diff scoped. --- ## PR-A2 — Stop mutating the system prompt; route memory context to live zone **Branch:** `realign-A2-system-prompt-immutable` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A2-system-prompt-immutable` **Risk:** **MEDIUM** (touches memory feature behavior) **LOC:** -150 / +80 ### Scope Eliminate P0-1 and P2-23. The system prompt is never mutated; memory context is appended to the latest user message tail (live zone). Delete the cache_aligner rewrite path; keep the volatile-content detector for warnings only. ### Files **Modify:** - `headroom/proxy/server.py:1026-1071` — delete `_inject_system_context`. Memory context handling routes exclusively through `_append_context_to_latest_non_frozen_user_turn` (already exists at `handlers/anthropic.py:1117-1135` for cache mode; promote to default). - `headroom/proxy/handlers/openai.py:1212` — same: delete `body["instructions"] = f"{existing_instructions}\n\n{memory_context}"`. Replace with append-to-latest-user-message-tail. - `headroom/transforms/cache_aligner.py` — delete the rewrite path (lines 160-262). Keep the volatile-content detector and the `cache_aligner_warnings` callback that surfaces detected dynamic content (UUIDs, dates, tokens) to a customer-visible log line. - `headroom/proxy/server.py:299` — `cache_aligner.enabled` flag stays default-False; document that turning it on now only affects warnings. **Tests added:** - `tests/test_proxy_system_prompt_immutable.py::test_memory_enabled_does_not_mutate_system` — request with memory enabled; assert outbound system bytes equal inbound system bytes. - `tests/test_proxy_system_prompt_immutable.py::test_memory_context_appears_in_user_tail` — same request; assert memory context appears in the last user message's tail. - `tests/test_cache_aligner_detector_only.py::test_volatile_content_detected_warned_not_rewritten` — system prompt with UUID; assert detector emits warning log; assert system bytes unchanged. **Tests deleted/updated:** - `tests/test_cache_aligner_rewrite_*.py` — delete; rewrite path is gone. ### Acceptance criteria - All new tests pass. - Existing memory tests still pass (the live-zone-tail append should produce equivalent semantics). - No regression in `tests/test_proxy_anthropic_cache_stability.py`. ### Blocked by None. Parallel with A1. ### Blocks PR-B6 (memory subsystem refactor, which builds on this). ### Rollback `git revert` the merge commit. Memory injection returns to system prompt. P0-1 returns. --- ## PR-A3 — Switch Python forwarders to byte-faithful body forwarding **Branch:** `realign-A3-byte-faithful-forwarders` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A3-byte-faithful-forwarders` **Risk:** **HIGH** (touches every outbound HTTP call in Python) **LOC:** -200 / +250 ### Scope Eliminate P0-2 universally. Every Python forwarder switches from `httpx ... json=body` to `httpx ... content=raw_bytes`. When body was mutated by a transform, re-serialize once with `separators=(",", ":")`, `ensure_ascii=False`, and the original encoding. When unmutated, forward the original `await request.body()` verbatim. ### Files **Modify:** - `headroom/proxy/server.py:1073-1124` — `_retry_request`: track whether body was mutated; if not, forward `original_body_bytes`; if yes, re-serialize with the canonical settings. Switch `await self.http_client.post(url, json=body)` to `await self.http_client.post(url, content=outbound_bytes, headers={**headers, "content-type": "application/json"})`. - `headroom/proxy/handlers/streaming.py:617-660` — same pattern in `_send_streaming_request`. - `headroom/proxy/handlers/openai.py:2392-2410` — WS→HTTP fallback: same pattern. - `headroom/proxy/handlers/batch.py:340-360` — batch endpoint: same pattern. - `headroom/proxy/helpers.py` — add `serialize_body_canonical(body: dict) -> bytes` helper using `json.dumps(body, separators=(",", ":"), ensure_ascii=False).encode("utf-8")`. **Tests added:** - `tests/test_proxy_byte_faithful_forwarding.py::test_passthrough_no_mutation_byte_equal` — request with no compression / memory / transforms; assert SHA-256 of upstream-received body equals SHA-256 of client-sent body. - `tests/test_proxy_byte_faithful_forwarding.py::test_compression_off_unicode_preserved` — request with `🔥` and CJK chars in user message; assert no `\uXXXX` escaping at upstream. - `tests/test_proxy_byte_faithful_forwarding.py::test_compression_off_numeric_precision_preserved` — request with `temperature: 1.0` and `seed: 12345678901234567`; assert exact bytes. ### Acceptance criteria - New byte-faithful tests pass. - Existing test suite green. - Manual smoke test: send a real request through the proxy with `tcpdump` or a recording mock; verify the bytes hitting upstream match a direct-to-Anthropic baseline. ### Blocked by None. Parallel with A1, A2. ### Blocks PR-A6 (memory tool injection refactor relies on the new mutation-tracking helper). ### Rollback `git revert`. The httpx `json=` defaults return. ### Notes - This is the highest-impact single PR for cache hit rate. Test coverage carefully. - `httpx.AsyncClient` defaults set Content-Length from the bytes; verify no Transfer-Encoding chunked drift. - The `accept-encoding` header strip stays for now; Phase F PR-F2 makes it conditional on auth mode. --- ## PR-A4 — Honor customer `cache_control` markers in Rust; enable `arbitrary_precision`+`raw_value` **Branch:** `realign-A4-honor-cache-control` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A4-honor-cache-control` **Risk:** **MEDIUM** (Rust-only; tightly scoped) **LOC:** -30 / +200 ### Scope Eliminate P0-3 and P0-5 directly. In Rust, walk customer-set `cache_control` markers in `system`, `tools`, and `messages`; compute the effective `frozen_message_count`. Switch `serde_json` to `arbitrary_precision` + `raw_value` features. Use `&RawValue` for `messages[*]` so unmodified messages forward as exact byte copies. The `compress_anthropic_request` function is currently a no-op stub (per A1) — this PR adds the cache_control parser as preparation for Phase B. ### Files **Modify:** - `Cargo.toml:34` — add features: `serde_json = { version = "1", features = ["preserve_order", "arbitrary_precision", "raw_value"] }`. Run `cargo update -p serde_json`. - `crates/headroom-proxy/src/compression/anthropic.rs` — add `pub fn compute_frozen_count(parsed: &serde_json::Value) -> usize` that walks `messages[*].content[*].cache_control`, `system[*].cache_control`, `tools[*].cache_control` and returns the highest message index whose content contains a marker. (Used by Phase B; currently called only by tests.) - `crates/headroom-core/src/lib.rs` — re-export `compute_frozen_count` for use in Phase B. **Add:** - `crates/headroom-proxy/tests/integration_cache_control.rs::cache_control_marker_at_message_3_yields_frozen_count_3` - `crates/headroom-proxy/tests/integration_cache_control.rs::cache_control_in_system_blocks_yields_frozen_count_full_history` - `crates/headroom-proxy/tests/integration_cache_control.rs::cache_control_ttl_1h_before_5m_passes` - `crates/headroom-proxy/tests/integration_cache_control.rs::cache_control_ttl_5m_before_1h_warns_and_passes` (we don't reject, but log per §2.19 ordering rule) ### Acceptance criteria - `cargo build -p headroom-proxy` works with new features. - `cargo test -p headroom-proxy` green. - The `compute_frozen_count` returns 0 for a request with zero markers; returns N for a request with a marker on `messages[N]`. ### Blocked by PR-A1 (the call site needs to be removed before this can land cleanly). ### Blocks PR-B2 (live-zone block dispatcher uses `compute_frozen_count`). ### Rollback `git revert`. The Cargo features stay (harmless). ### Notes - `RawValue` is enabled but not yet consumed in this PR. Phase B PR-B2 wires it. - Per guide §2.19, `1h` markers must precede `5m` markers; we log a warning when the ordering is reversed but don't reject (the customer's request, not ours to validate). --- ## PR-A5 — Strip `x-headroom-*` from upstream-bound headers **Branch:** `realign-A5-strip-headroom-headers` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A5-strip-headroom-headers` **Risk:** **LOW** **LOC:** -10 / +50 ### Scope Eliminate P5-49. `dict(request.headers.items())` is captured unmodified and forwarded; this PR adds an explicit strip step before any upstream call. Reduces fingerprint surface for subscription detection. ### Files **Modify:** - `headroom/proxy/handlers/anthropic.py:526` — wrap `dict(request.headers.items())` with `_strip_internal_headers(headers)` (new helper). - `headroom/proxy/handlers/openai.py:232-264` — same. - `headroom/proxy/handlers/streaming.py:617` — same. - `headroom/proxy/handlers/batch.py:340` — same. - `headroom/proxy/handlers/gemini.py:31` — same. - `headroom/proxy/helpers.py` — add `_strip_internal_headers` helper. Default strip list: `x-headroom-*` (case-insensitive prefix), plus a hardcoded set of internal flags. - `crates/headroom-proxy/src/headers.rs` — add `strip_internal_headers` to the request-side filter. Document that response-side `X-Headroom-*` injection (which is fine) is unrelated. **Tests added:** - `tests/test_header_isolation.py::test_x_headroom_bypass_not_forwarded` - `tests/test_header_isolation.py::test_x_headroom_mode_not_forwarded` - `tests/test_header_isolation.py::test_x_headroom_user_id_not_forwarded` - `crates/headroom-proxy/tests/integration_headers.rs::x_headroom_request_headers_stripped` ### Acceptance criteria - New tests pass. - Existing client-driven `x-headroom-bypass: true` flow still works (proxy reads it; just doesn't forward). - No legitimate header is stripped (whitelist `x-request-id`, `x-trace-id`, etc. by default — though they aren't `x-headroom-*` so they're untouched). ### Blocked by None. Parallel. ### Blocks PR-F2 (auth-mode policy uses this helper). ### Rollback `git revert`. Headers leak again. Low operational risk. --- ## PR-A6 — Pin `anthropic-beta` order; session-stickiness skeleton **Branch:** `realign-A6-anthropic-beta-stable` **Worktree:** `~/claude-projects/headroom-worktrees/realign-A6-anthropic-beta-stable` **Risk:** **MEDIUM** (touches memory injection beta-mutation) **LOC:** -40 / +180 ### Scope Eliminate P5-50 and start P5-51. When the proxy mutates `anthropic-beta` (memory injection), the new comma-list is computed deterministically (sort tokens or preserve insertion order with new tokens appended). Add a per-session "betas seen so far" tracker so any beta seen in turn N is included in turn N+1 even if the client drops it. ### Files **Modify:** - `headroom/proxy/handlers/anthropic.py:1162-1168` — replace the ad-hoc concat with a helper `merge_anthropic_beta(client: str, headroom: list[str]) -> str` that splits client's value on `,`, lowercases each token, deduplicates, appends Headroom-required tokens (sorted within the appended group), and rejoins. - `headroom/proxy/server.py` — extend `session_state` (already exists for memory) to track `betas_seen: set[str]` per session. Update on every request; merge into outbound `anthropic-beta` for follow-up requests. - `headroom/proxy/helpers.py` — add `betas_seen_lock` and `update_session_betas` helpers. **Tests added:** - `tests/test_anthropic_beta_session_sticky.py::test_beta_seen_turn_1_present_in_turn_2_even_if_client_drops` - `tests/test_anthropic_beta_session_sticky.py::test_memory_injection_appends_deterministic_order` - `tests/test_anthropic_beta_session_sticky.py::test_client_value_preserved_when_no_injection` ### Acceptance criteria - New tests pass. - Session ID is keyed off the existing session detection (per `headroom/proxy/handlers/anthropic.py:1417`). - The "betas seen" set is bounded (LRU eviction at 1000 sessions). ### Blocked by PR-A3 (relies on byte-faithful forwarder for header-bytes correctness; if A3 is rolled back, this still - REALIGNMENT/04-phase-B-live-zone.md (23894 bytes)
# Phase B — Live-Zone-Only Compression Engine **Goal:** Delete ~10 K LOC of architectural over-build (ICM, scoring, relevance, rolling-window, progressive-summarizer, tool-crusher); build the correct architecture: per-block compression on the live zone only, with type-aware dispatch, token validation, and CCR hardening. **Calendar:** 2 weeks. PR-B1 is the big delete (high-LOC, lower-risk-than-it-looks because the code was unreachable after Phase A). PR-B2..B7 build the replacement. **Shape:** 7 PRs. B1 is independent; B2..B5 depend on B1; B6 + B7 layer on B2. --- ## PR-B1 — The big delete: retire ICM and its dependencies **Branch:** `realign-B1-delete-icm-and-deps` **Worktree:** `~/claude-projects/headroom-worktrees/realign-B1-delete-icm-and-deps` **Risk:** **MEDIUM** (large diff, but most code became unreachable after Phase A PR-A1) **LOC:** **-10,000 / +50** (the big retirement) ### Scope Delete the wrong-mental-model machinery wholesale. After Phase A PR-A1 made the proxy a passthrough on `/v1/messages`, none of this code is reached at runtime; this PR removes the source so future contributors can't re-wire it. ### Files **Delete (Python):** - `headroom/transforms/intelligent_context.py` (1077 LOC) - `headroom/transforms/rolling_window.py` (395 LOC) - `headroom/transforms/progressive_summarizer.py` (508 LOC) - `headroom/transforms/scoring.py` (459 LOC) - `headroom/transforms/tool_crusher.py` (338 LOC) **Delete (Rust):** - `crates/headroom-core/src/context/manager.rs` - `crates/headroom-core/src/context/config.rs` - `crates/headroom-core/src/context/workspace.rs` - `crates/headroom-core/src/context/candidate.rs` - `crates/headroom-core/src/context/ccr_drop.rs` - `crates/headroom-core/src/context/strategy/mod.rs` - `crates/headroom-core/src/context/strategy/drop_by_score.rs` - All of `crates/headroom-core/src/scoring/*.rs` (~1500 LOC) - All of `crates/headroom-core/src/relevance/*.rs` (~1600 LOC) - `crates/headroom-core/.fastembed_cache/` directory and its `bge-small-en-v1.5` ONNX artifacts (~50 MB) **Move:** - `crates/headroom-core/src/context/safety.rs` → `crates/headroom-core/src/transforms/safety.rs`. Update all callers' `use` paths. The tool-pair atomicity logic is preserved verbatim (it's correct and live-zone code needs it). **Modify:** - `crates/headroom-core/src/lib.rs` — remove `pub mod context;`, `pub mod scoring;`, `pub mod relevance;`. Add `pub use transforms::safety;`. - `crates/headroom-core/src/context/mod.rs` — delete (empty after move). - `crates/headroom-proxy/src/lib.rs` — no changes (already doesn't reach into deleted modules after PR-A1). - `crates/headroom-py/src/lib.rs` — remove any PyO3 exports of `MessageScorer`, `IntelligentContextManager`, etc. (per agent reports, MessageScorer was exposed in PR #338/#343). - `headroom/transforms/__init__.py` — remove imports of deleted modules. - `headroom/proxy/handlers/anthropic.py` — remove all imports / call sites of `IntelligentContextManager`. (Agent C found these at multiple locations; track via `grep -n IntelligentContextManager headroom/`.) - `headroom/proxy/server.py` — remove ICM import and instantiation. - `Cargo.toml` workspace dependencies — drop `fastembed`, `tantivy`, `ort` (if only used by relevance), and any other deps that become orphaned. **Tests deleted:** - All `tests/test_intelligent_context*.py` - All `tests/test_rolling_window*.py` - All `tests/test_progressive_summarizer*.py` - All `tests/test_scoring*.py` - All `tests/test_tool_crusher*.py` - `crates/headroom-core/tests/scoring_*.rs`, `relevance_*.rs`, `context_*.rs` (other than safety) - Parity comparator for `message_scorer` (PR #338/#343 work) — delete the comparator and the fixtures it consumed. - Parity fixtures `tests/parity/fixtures/message_scorer/` (13 fixtures per Agent F report). ### Acceptance criteria - `cargo build --workspace` green. - `cargo test --workspace` green (after test deletions). - `make ci-precheck` green. - `pytest -x` green. - Workspace builds **without** the fastembed cache dir. - `git grep -i "IntelligentContextManager\|MessageScorer\|RollingWindow\|ProgressiveSummarizer\|ToolCrusher\|DropByScoreStrategy"` returns nothing in `crates/`, `headroom/`, `tests/` except comments referencing the deletion. ### Blocked by PR-A1 (ICM call site must be removed first). ### Blocks PR-B2 (live-zone block dispatcher fills the void). ### Rollback `git revert`. ~10K LOC returns. Cache-killer bugs DO NOT return because Phase A PR-A1 already removed the call site (the deleted code is unreachable). Safe to revert. ### Notes - **MessageScorer Rust port retirement:** PR #338 and #343 (April 2026) ported MessageScorer to Rust. That work becomes deletable here. Sunk cost stays sunk. The fixtures and the parity-harness scaffolding learnings carry forward to live-zone work. - **`bge-small-en-v1.5` ONNX cache:** ~50 MB. Removing it is reversible (re-fetched on next fastembed init if anyone re-adds the dep). Document in CHANGELOG. - **`anchor_selector.py`** — Agent G suspected it might be ICM-only. Check: `git grep AnchorSelector headroom/ crates/`. If only consumed by ICM/scoring/SmartCrusher, delete; if consumed by SmartCrusher's anchor logic, keep. Current Rust has `crates/headroom-core/src/transforms/anchor_selector.rs` which is consumed by SmartCrusher — keep that one, delete the Python one if ICM was its only caller. --- ## PR-B2 — Live-zone block dispatcher in Rust **Branch:** `realign-B2-live-zone-dispatcher` **Worktree:** `~/claude-projects/headroom-worktrees/realign-B2-live-zone-dispatcher` **Risk:** **MEDIUM-HIGH** (new architecture; the central piece) **LOC:** +800 ### Scope Build the new compressor: a function that takes an Anthropic `/v1/messages` body, identifies the live-zone blocks (latest user message tool_results, latest user message text, latest assistant tool_use is hot zone — exclude), and dispatches each to a type-aware compressor. Does NOT yet wire the type-aware compressors (PR-B3); does NOT yet validate tokens (PR-B4); does NOT yet inject CCR (PR-B7). PR-B2 lays the dispatching skeleton with no-op compressors; subsequent PRs fill them in. ### Files **Add:** - `crates/headroom-core/src/transforms/live_zone.rs` — the dispatcher. Public API: ```rust pub fn compress_live_zone( body_raw: &serde_json::value::RawValue, frozen_message_count: usize, auth_mode: AuthMode, ) -> Result<LiveZoneOutcome>; pub enum LiveZoneOutcome { NoChange, Modified { new_body: Box<serde_json::value::RawValue>, manifest: CompressionManifest }, } ``` Implementation skeleton: 1. Parse `body` minimally (only `messages` field; leave the rest as `RawValue`). 2. For each message at index `>= frozen_message_count`: - Identify if it's the latest user message (live zone candidate). - For each block in its content: - If block type is `tool_result`, dispatch to a no-op compressor (filled in PR-B3). - If block type is `text`, dispatch to text compressor (no-op for now). - Otherwise (image, etc.), no-op. 3. Reassemble the modified `messages` array, preserving unmodified messages as `RawValue` byte-copies. 4. Reassemble the body, preserving the original envelope as `RawValue` byte-copies; the only modified bytes are within the messages array. 5. Return `Modified` only if any block was actually mutated; otherwise `NoChange` and the caller forwards original bytes. **Modify:** - `crates/headroom-proxy/src/compression/mod.rs` — add `pub mod live_zone_anthropic;` and route `/v1/messages` to it (replacing the no-op stub from PR-A1). - `crates/headroom-proxy/src/compression/live_zone_anthropic.rs` (new) — calls `compress_live_zone` with `frozen_count = compute_frozen_count(parsed)` (from PR-A4). - `crates/headroom-proxy/src/compression/anthropic.rs` — delete (replaced by `live_zone_anthropic.rs`). **Tests added:** - `crates/headroom-core/tests/live_zone_skeleton.rs::dispatches_only_to_latest_user_message` - `crates/headroom-core/tests/live_zone_skeleton.rs::respects_frozen_message_count` - `crates/headroom-core/tests/live_zone_skeleton.rs::no_change_when_no_block_mutated_returns_original` - `crates/headroom-core/tests/live_zone_skeleton.rs::modified_messages_byte_equal_outside_block` - `crates/headroom-core/tests/live_zone_skeleton.rs::system_and_tools_byte_equal_always` - `crates/headroom-proxy/tests/integration_live_zone.rs::end_to_end_live_zone_passthrough` ### Acceptance criteria - `cargo test -p headroom-core` green. - `cargo test -p headroom-proxy` green. - All Phase A SHA-256 tests still pass (live-zone with no-op compressors is byte-identical). ### Blocked by PR-B1 (deletion); PR-A4 (cache_control / RawValue features). ### Blocks PR-B3, PR-B4, PR-B7. ### Rollback `git revert`. Compression returns to passthrough (the Phase A state); no functional regression. ### Notes - The `RawValue`-based approach is the correctness mechanism: bytes outside modified blocks are byte-copies, not parse-then-reserialize. - `AuthMode` parameter is unused in B2 (always `Payg` from B2's perspective); Phase F PR-F2 wires the gate. --- ## PR-B3 — Wire type-aware compressors into live-zone dispatcher **Branch:** `realign-B3-wire-type-aware-compressors` **Worktree:** `~/claude-projects/headroom-worktrees/realign-B3-wire-type-aware-compressors` **Risk:** **MEDIUM** (existing compressors are battle-tested) **LOC:** +600 ### Scope Wire `SmartCrusher`, `LogCompressor`, `SearchCompressor`, `DiffCompressor`, `CodeCompressor` into the dispatcher. Per-block content-type detection drives dispatch. No token validation yet (PR-B4); no CCR hardening yet (PR-B7). ### Files **Modify:** - `crates/headroom-core/src/transforms/live_zone.rs` — replace no-op compressors with real dispatch: ```rust fn compress_block(block: &mut Block, content_type: ContentType) -> Result<Option<CompressionResult>> { match content_type { ContentType::JsonArrayOfDicts => smart_crusher::crush(block), ContentType::Logs => log_compressor::compress(block), ContentType::SearchResults => search_compressor::compress(block), ContentType::Diff => diff_compressor::compress(block), ContentType::SourceCode => code_compressor::compress(block), ContentType::PlainText => Ok(None), // PR-B4 adds Kompress; for now, leave untouched ContentType::Image | ContentType::Unknown => Ok(None), } } ``` - `crates/headroom-core/src/transforms/content_detector.rs` — extend `ContentType` enum with the variants above. Use existing `Magika` + `unidiff-rs` + heuristic detectors. **Tests added:** - `crates/headroom-core/tests/live_zone_dispatch.rs::json_tool_result_routes_to_smart_crusher` - `crates/headroom-core/tests/live_zone_dispatch.rs::log_tool_result_routes_to_log_compressor` - `crates/headroom-core/tests/live_zone_dispatch.rs::diff_tool_result_routes_to_diff_compressor` - `crates/headroom-core/tests/live_zone_dispatch.rs::source_code_tool_result_routes_to_code_compressor` - `crates/headroom-core/tests/live_zone_dispatch.rs::unknown_content_type_no_op` ### Acceptance criteria - All new tests pass. - Existing SmartCrusher / LogCompressor / DiffCompressor / SearchCompressor tests still pass (their code is unchanged; only the caller is new). - A representative `/v1/messages` request with a 50KB JSON tool_result through the proxy results in measurable compression (>2× size reduction) and SHA-256-equal envelope outside the compressed block. ### Blocked by PR-B2. ### Blocks PR-B4 (token validation gate), PR-B7 (CCR injection). ### Rollback `git revert`. Live-zone goes back to no-op compressors. --- ## PR-B4 — Token validation gate with fallback; per-content-type byte thresholds **Branch:** `realign-B4-token-validation-gate` **Worktree:** `~/claude-projects/headroom-worktrees/realign-B4-token-validation-gate` **Risk:** **LOW** **LOC:** +250 ### Scope Eliminate P3-33 and P3-34. After every per-block compression, run the tokenizer over `original` and `compressed`. If `compressed.tokens >= original.tokens`, fall back to original. Add per-content-type byte thresholds: code>2KB, JSON>1KB, logs>500B, plain text>5KB. Below threshold → no compression attempted. ### Files **Modify:** - `crates/headroom-core/src/transforms/live_zone.rs` — wrap each compressor call with: ```rust let original_tokens = tokenizer.count(&original_bytes)?; let compressed_tokens = tokenizer.count(&compressed_bytes)?; if compressed_tokens >= original_tokens { metrics::compression_rejected_by_token_check(compressor_name); return Ok(None); // fall back to original } ``` - `crates/headroom-core/src/transforms/live_zone.rs::compress_block` — gate on byte threshold per content type: ```rust const THRESHOLDS: &[(ContentType, usize)] = &[ (ContentType::SourceCode, 2048), (ContentType::JsonArrayOfDicts, 1024), (ContentType::Logs, 512), (ContentType::PlainText, 5120), (ContentType::Diff, 1024), (ContentType::SearchResults, 1024), ]; if block.bytes_len() < threshold_for(content_type) { return Ok(None); } ``` **Tests added:** - `crates/headroom-core/tests/live_zone_thresholds.rs::below_threshold_no_compression_attempted` - `crates/headroom-core/tests/live_zone_thresholds.rs::above_threshold_compression_attempted` - `crates/headroom-core/tests/live_zone_token_validation.rs::compressed_more_tokens_falls_back` - `crates/headroom-core/tests/live_zone_token_validation.rs::compressed_fewer_tokens_accepted` - Property test: `proptest! { fn live_zone_compression_token_count_non_increasing(blocks in arb_blocks_strategy()) { ... } }` ### Acceptance criteria - All tests pass. - A pathological input (already-minified JSON, dense base64) falls back to original instead of inflating tokens. - Prometheus emits `compression_rejected_by_token_check_total{strategy=...}` counter. ### Blocked by PR-B3. ### Blocks PR-B6, PR-B7. ### Rollback `git revert`. Token validation removed; bytes-only gate returns. Slight regression risk on pathological inputs. --- ## PR-B5 — TOIN observation-only refactor **Branch:** `realign-B5-toin-observation-only` **Worktree:** `~/claude-projects/headroom-worktrees/realign-B5-toin-observation-only` **Risk:** **MEDIUM** (TOIN is a preserved primitive; refactor must maintain its learning value) **LOC:** -300 / +400 ### Scope Eliminate P2-27 and P5-56. Strip TOIN's request-time hint API; keep the recording API. Recommendations published between deploys via a CLI tool that aggregates and writes a TOML file the compressor loads at startup. Per-tenant aggregation key extended to `(auth_mode, model_family, structure_hash)`. ### Files **Modify:** - `headroom/telemetry/toin.py:853-927` — remove `get_recommendation()` and `CompressionHint`. Replace with a no-op stub that returns `None`; deprecation warning in docstring. - `headroom/telemetry/toin.py:103` — `Pattern` adds `auth_mode: str`, `model_family: str` fie - REALIGNMENT/05-phase-C-rust-proxy.md (16540 bytes)
# Phase C — Rust Proxy Paths **Goal:** Port the remaining proxy surfaces to Rust. After Phase C, the Rust proxy handles `/v1/messages`, `/v1/chat/completions`, `/v1/responses` (HTTP + streaming), with a byte-level SSE state machine that handles every wire-format quirk the guide enumerates. **Calendar:** 3 weeks. Mostly sequential (each PR builds on the SSE parser). **Shape:** 5 PRs. --- ## PR-C1 — Byte-level SSE parser with full state machine **Branch:** `realign-C1-rust-sse-parser` **Worktree:** `~/claude-projects/headroom-worktrees/realign-C1-rust-sse-parser` **Risk:** **HIGH** (foundational; many wire-format quirks; UTF-8 split-byte handling) **LOC:** +1500 ### Scope Eliminate P1-8, P1-9, P1-14, P1-15, P1-17, P4-48. Build the byte-level SSE parser in `crates/headroom-proxy/src/sse/`. Three parsers (one per provider × API), all sharing a common framing layer. Per-stream state (no module-level state). Models the streaming state machines from guide §5 exactly. ### Files **Add:** - `crates/headroom-proxy/src/sse/mod.rs` — module re-exports. - `crates/headroom-proxy/src/sse/framing.rs` — byte-level framing. Reads `bytes::Bytes` chunks, accumulates into a `BytesMut` buffer, finds `\n\n` event terminators in bytes (not strings), yields complete events as `(event_name: Option<String>, data: Bytes)`. Decodes UTF-8 per complete event, never per chunk. Handles `: ping` keepalives (skip silently). Handles `[DONE]` literal. - `crates/headroom-proxy/src/sse/anthropic.rs` — Anthropic stream state machine per guide §5.1: ```rust pub struct AnthropicStreamState { pub message_id: Option<String>, pub model: Option<String>, pub blocks: HashMap<usize, BlockState>, // keyed by index pub current_block_index: Option<usize>, pub stop_reason: Option<String>, pub usage: UsageBuilder, pub status: StreamStatus, } pub struct BlockState { pub block_type: String, pub text_buffer: String, pub partial_json: String, pub signature: Option<String>, pub citations: Vec<Citation>, pub metadata: serde_json::Value, pub complete: bool, } impl AnthropicStreamState { pub fn apply(&mut self, event: SseEvent) -> Result<()>; } ``` Handlers for `message_start`, `content_block_start`, `content_block_delta` (switching on `delta.type`: `text_delta` / `thinking_delta` / `input_json_delta` / `citations_delta` / `signature_delta`), `content_block_stop`, `message_delta`, `message_stop`, `error`, `ping`. - `crates/headroom-proxy/src/sse/openai_chat.rs` — OpenAI Chat Completions state machine per guide §5.2. `ChunkState`, `ChoiceState`, `ToolCallState`. Handles `[DONE]` and `stream_options.include_usage` final chunk. - `crates/headroom-proxy/src/sse/openai_responses.rs` — OpenAI Responses state machine per guide §5.3. `ResponseState`, `ItemState` keyed by `id` (not position) for out-of-order completion. Handlers for `response.created`, `output_item.added/done`, `content_part.added/done`, `output_text.delta/done`, `function_call_arguments.delta/done`, `reasoning_summary.delta/done`, `response.completed/failed/incomplete`. **Modify:** - `crates/headroom-proxy/src/proxy.rs` — when forwarding a streaming response, the state machine runs in parallel with the byte-passthrough (so client gets raw bytes immediately; state machine populates telemetry without blocking the stream). **Tests added:** - `crates/headroom-proxy/tests/sse_framing.rs::utf8_split_emoji_across_chunks_preserved` - `crates/headroom-proxy/tests/sse_framing.rs::single_newline_does_not_emit_event` - `crates/headroom-proxy/tests/sse_framing.rs::double_newline_emits_event` - `crates/headroom-proxy/tests/sse_framing.rs::ping_keepalive_skipped` - `crates/headroom-proxy/tests/sse_framing.rs::done_sentinel_detected` - `crates/headroom-proxy/tests/sse_framing.rs::trailing_data_after_done_tolerated` - `crates/headroom-proxy/tests/sse_anthropic.rs::four_event_dance_text_block` - `crates/headroom-proxy/tests/sse_anthropic.rs::thinking_delta_accumulated` - `crates/headroom-proxy/tests/sse_anthropic.rs::signature_delta_preserved_byte_equal` - `crates/headroom-proxy/tests/sse_anthropic.rs::input_json_delta_concatenated_parsed_at_stop` - `crates/headroom-proxy/tests/sse_anthropic.rs::citations_delta_accumulated` - `crates/headroom-proxy/tests/sse_anthropic.rs::message_delta_finalizes_stop_reason_and_output_tokens` - `crates/headroom-proxy/tests/sse_anthropic.rs::mid_stream_error_event_handled` - `crates/headroom-proxy/tests/sse_anthropic.rs::interleaved_blocks_by_index` - `crates/headroom-proxy/tests/sse_openai_chat.rs::tool_call_id_and_name_only_first_chunk` - `crates/headroom-proxy/tests/sse_openai_chat.rs::tool_call_arguments_concatenated` - `crates/headroom-proxy/tests/sse_openai_chat.rs::usage_in_final_chunk_when_include_usage_set` - `crates/headroom-proxy/tests/sse_openai_chat.rs::refusal_field_handled` - `crates/headroom-proxy/tests/sse_openai_responses.rs::out_of_order_item_completion_by_id` - `crates/headroom-proxy/tests/sse_openai_responses.rs::reasoning_summary_accumulated` - `crates/headroom-proxy/tests/sse_openai_responses.rs::function_call_arguments_string_preserved` - Property test: `proptest! { fn sse_parser_no_panic_on_arbitrary_bytes(bytes in any::<Vec<u8>>()) { let _ = parse(bytes); } }` ### Acceptance criteria - All new tests pass. - Property test: 100K random byte sequences never panic the parser. - Real-traffic shadow test: feed a recorded production Anthropic stream through both the Rust parser and the Python parser; assert telemetry agrees on `usage` totals. ### Blocked by PR-A1. ### Blocks PR-C2, PR-C3, PR-C4. ### Rollback `git revert`. SSE parsing returns to byte-passthrough (Phase A state). Telemetry less rich but no functional regression. --- ## PR-C2 — `/v1/chat/completions` handler in Rust **Branch:** `realign-C2-rust-chat-completions` **Worktree:** `~/claude-projects/headroom-worktrees/realign-C2-rust-chat-completions` **Risk:** **HIGH** (new endpoint surface) **LOC:** +1200 ### Scope Add `/v1/chat/completions` to `crates/headroom-proxy`. Handles request-body shape, live-zone compression dispatch (assistant `tool_calls` and `tool` role messages — equivalents of Anthropic's tool_result), and the streaming state machine from PR-C1. Adds Phase E PR-E1/E2 tool-def normalization gate (no-op until Phase E). ### Files **Add:** - `crates/headroom-proxy/src/handlers/chat_completions.rs` — POST handler. ```rust async fn handle_chat_completions( State(state): State<AppState>, headers: HeaderMap, body: Bytes, ) -> Result<Response>; ``` - `crates/headroom-proxy/src/compression/live_zone_openai.rs` — OpenAI live-zone dispatcher. Live zone for Chat Completions: latest `tool` role message's `content`; latest `user` message's text content. Compress per type-aware dispatch (same compressors as Anthropic; reused). **Modify:** - `crates/headroom-proxy/src/lib.rs` — route `/v1/chat/completions` (POST) to the new handler. - `crates/headroom-proxy/src/compression/mod.rs` — add OpenAI Chat dispatch path. **Tests added:** - `crates/headroom-proxy/tests/integration_chat_completions.rs::passthrough_no_compression_byte_equal` - `crates/headroom-proxy/tests/integration_chat_completions.rs::tool_message_compressed` - `crates/headroom-proxy/tests/integration_chat_completions.rs::n_greater_than_one_passthrough` - `crates/headroom-proxy/tests/integration_chat_completions.rs::stream_options_include_usage_preserved` - `crates/headroom-proxy/tests/integration_chat_completions.rs::tool_choice_change_passthrough_no_mutation` - `crates/headroom-proxy/tests/integration_chat_completions.rs::refusal_field_in_response_handled` - `crates/headroom-proxy/tests/integration_chat_completions.rs::streaming_tool_call_argument_accumulation` ### Acceptance criteria - All new tests pass. - A real Chat Completions request through the Rust proxy produces byte-equal upstream bytes when compression is off. - Streaming tool_call accumulation works for the `delta.tool_calls[].function.arguments` pattern. ### Blocked by PR-C1, PR-B3, PR-B4. ### Blocks PR-C3, PR-H1. ### Rollback `git revert`. `/v1/chat/completions` still flows through the Python proxy (Phase H hasn't deleted it yet). --- ## PR-C3 — `/v1/responses` handler in Rust (HTTP) **Branch:** `realign-C3-rust-responses-http` **Worktree:** `~/claude-projects/headroom-worktrees/realign-C3-rust-responses-http` **Risk:** **HIGH** **LOC:** +1500 ### Scope Add `/v1/responses` HTTP handler. Item-shape passthrough preservation for every Responses item type (V4A patches, `local_shell_call.action.command` argv, Codex `phase`, `compaction`, MCP items, computer_use, `image_generation_call`, server-side tool results). Live-zone compression for `function_call_output`, `local_shell_call_output`, `apply_patch_call_output` (only when >2KB). ### Files **Add:** - `crates/headroom-proxy/src/handlers/responses.rs` — POST handler. - `crates/headroom-proxy/src/compression/live_zone_responses.rs` — Responses live-zone dispatcher. Live zone: latest `function_call_output.output`, latest `local_shell_call_output.output`, latest `apply_patch_call_output.output`, latest `user` message text content. - `crates/headroom-proxy/src/responses_items.rs` — explicit per-item-type enum and passthrough rules: ```rust pub enum ResponseItem { Message { phase: Option<String>, .. }, Reasoning { encrypted_content: Option<String>, .. }, // passthrough only FunctionCall { call_id: String, arguments: String, .. }, // arguments stays as string LocalShellCall { command: Vec<String>, .. }, // argv array preserved ApplyPatchCall { operation: ApplyPatchOperation, .. }, // V4A diff verbatim Compaction { encrypted_content: String, .. }, // passthrough only McpCall { .. } | McpListTools { .. } | McpApprovalRequest { .. }, // passthrough ComputerCall { .. } | ComputerCallOutput { .. }, WebSearchCall { .. } | FileSearchCall { .. } | CodeInterpreterCall { .. }, ImageGenerationCall { .. }, ToolSearchCall { .. }, CustomToolCall { .. }, Unknown { type_: String, raw: Box<RawValue> }, // log warning; preserve verbatim } ``` **Modify:** - `crates/headroom-proxy/src/lib.rs` — route `/v1/responses` (POST) to the new handler. **Tests added:** - `crates/headroom-proxy/tests/integration_responses.rs::v4a_patch_byte_equal_through_proxy` - `crates/headroom-proxy/tests/integration_responses.rs::local_shell_call_command_argv_array_preserved` - `crates/headroom-proxy/tests/integration_responses.rs::codex_phase_commentary_preserved` - `crates/headroom-proxy/tests/integration_responses.rs::codex_phase_final_answer_preserved` - `crates/headroom-proxy/tests/integration_responses.rs::compaction_item_byte_equal` - `crates/headroom-proxy/tests/integration_responses.rs::reasoning_encrypted_content_byte_equal` - `crates/headroom-proxy/tests/integration_responses.rs::function_call_arguments_string_preserved` - `crates/headroom-proxy/tests/integration_responses.rs::call_id_referenced_not_id` - `crates/headroom-proxy/tests/integration_responses.rs::apply_patch_output_below_2kb_no_compression` - `crates/headroom-proxy/tests/integration_responses.rs::apply_patch_output_above_2kb_compressed` - `crates/headroom-proxy/tests/integration_responses.rs::local_shell_output_compressed` - `crates/headroom-proxy/tests/integration_responses.rs::mcp_tool_call_byte_equal` - `crates/headroom-proxy/tests/integration_responses.rs::computer_call_byte_equal` - `crates/headroom-proxy/tests/integration_responses.rs::image_generation_call_no_log_redaction_in_test_mode` - `crates/headroom-proxy/tests/integration_responses.rs::unknown_item_type_logged_warning_byte_equal` ### Acceptance criteria - All new tests pass. - A representative Responses request with reasoning + function_call + local_shell + apply_patch + custom items round-trips byte-equal modulo compressed live-zone outputs. ### Blocked by PR-C1, PR-C2. ### Blocks PR-C4. ### Rollback `git revert`. `/v1/responses` still flows through Python. --- ## PR-C4 — `/v1/responses` streaming + Conversations API awareness **Branch:** `realign-C4-rust-responses-streaming` **Worktree:** `~/claude-projects/headroom-worktrees/realign-C4-rust-responses-streaming` **Risk:** **MEDIUM-HIGH** **LOC:** +800 ### Scope Streaming for `/v1/responses` using the SSE state machine from PR-C1. Plus first-class awareness of the Conversations API (P4-40) — when `conversation: {"id": "conv_..."}` is in the body, the local view is incomplete; tokenizer must adjust or skip compression decisions. ### Files **Modify:** - `crates/headroom-proxy/src/handlers/responses.rs` — when `Accept: text/event-stream`, route to streaming handler. The streaming handler runs the `OpenAIResponsesStreamState` machine in parallel with byte-passthrough. - `crates/headroom-proxy/src/sse/openai_responses.rs` — add usage extraction from `response.completed`. **Add:** - `crates/headroom-proxy/src/conversations.rs` — detect `conversation: {"id": "conv_..."}` in request body. When present, log a warning and disable live-zone compression for that request (until Phase 4 cross-request shared cache lands). Telemetry: `proxy_conversations_api_request_count_total`. **Tests added:** - `crates/headroom-proxy/tests/integration_responses_streaming.rs::reasoning_summary_streamed_correctly` - `crates/headroom-proxy/tests/integration_responses_streaming.rs::function_call_arguments_streamed_byte_equal` - `crates/headroom-proxy/tests/integration_responses_streaming.rs::out_of_order_items_handled_by_id` - `crates/headroom-proxy/tests/integration_responses_streaming.rs::response_completed_usage_captured` - `crates/headroom-proxy/tests/integration_responses_streaming.rs::response_failed_handled` - `crates/headroom-proxy/tests/integration_responses_streaming.rs::response_incomplete_with_max_output_tokens_reason` - `crates/headroom-proxy/tests/integration_conversations.rs::conversation_id_present_skips_compression_warns` ### Acceptance criteria - All new tests pass. - The Conversations API warning appears in logs at `INFO` level with a `conversation_id` field. ### Blocked by PR-C3. ### Blocks PR-H1. ### Rollback `git revert`. Streaming Responses routes through Python. --- ## PR-C5 — `responses_converter.py` retirement (Rust handles it natively) **Branch:** `realign-C5-retire-responses-converter` **Worktree:** `~/claude-projects/headroom-worktrees/realign-C5-retire-responses-converter` **Risk:** **LOW** (cleanup; Rust handler from PR-C3/C4 covers this surface) **LOC:** -267 / +20 ### Scope Delete `headroom/proxy/responses_converter.py` (the Anthropic↔OpenAI Responses↔Chat Completions converter that mishandled `phase`, multi-text-part rebuild, etc.). The Rust handler from PR-C3 handles `/v1/responses` natively without converting between shapes. After this PR lands, no Python code is on the `/v1/responses` request path. ### Files **D - REALIGNMENT/06-phase-D-bedrock-vertex.md (10585 bytes)
# Phase D — Bedrock & Vertex Native Envelopes **Goal:** Replace the fake LiteLLM-based Bedrock/Vertex paths (which lossy-convert Anthropic↔OpenAI shapes) with native handlers in the Rust proxy. After Phase D, Anthropic-on-Bedrock and Anthropic-on-Vertex preserve `thinking`, `redacted_thinking`, `document`, `search_result`, `image`, `server_tool_use`, `mcp_tool_use` blocks AND benefit from the live-zone compression engine. **Calendar:** 2 weeks. SigV4 + EventStream are the bulk of the work. **Shape:** 4 PRs. D1+D2+D3 are AWS Bedrock; D4 is GCP Vertex. --- ## PR-D1 — Native Bedrock InvokeModel route (non-streaming) **Branch:** `realign-D1-bedrock-native-invoke` **Worktree:** `~/claude-projects/headroom-worktrees/realign-D1-bedrock-native-invoke` **Risk:** **HIGH** (new auth + envelope surface) **LOC:** +1500 ### Scope Eliminate part of P4-37 and P4-39. Add `POST /model/{model}/invoke` route to the Rust proxy. Recognizes the Bedrock envelope (`anthropic_version` body field, no `model` field, AWS SigV4 auth). Forwards the (possibly compressed) request to the Bedrock endpoint with re-signed SigV4. Live-zone compression runs the same as for direct Anthropic. ### Files **Add:** - `crates/headroom-proxy/src/bedrock/mod.rs` — module re-exports. - `crates/headroom-proxy/src/bedrock/sigv4.rs` — AWS SigV4 signing. Use the `aws-sigv4` crate. Sign over the (possibly modified) request body bytes. Critical: sign **after** Headroom finishes mutating the body, so the signature matches what Bedrock receives. - `crates/headroom-proxy/src/bedrock/invoke.rs` — POST handler for `/model/{model}/invoke`. Detects `anthropic.claude-*` model IDs; routes to live-zone compression for Anthropic shape; signs and forwards. - `crates/headroom-proxy/src/bedrock/envelope.rs` — `BedrockEnvelope` struct: parses `{"anthropic_version": "...", ...rest_of_anthropic_body}`. Re-emits in Bedrock shape with `anthropic_version` preserved as the first key. **Modify:** - `crates/headroom-proxy/src/lib.rs` — route `/model/{model}/invoke` and `/model/{model}/converse` (POST) to the new handler. - `crates/headroom-proxy/src/config.rs` — add `--bedrock-region` flag (default `us-east-1`) and AWS credential config (uses `aws-config` crate's default chain). - `Cargo.toml` workspace — add `aws-sigv4`, `aws-config`, `aws-credential-types`. **Tests added:** - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::native_envelope_round_trip_byte_equal` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::sigv4_signed_correctly_after_compression` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::thinking_block_preserved_through_bedrock` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::redacted_thinking_preserved` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::document_block_preserved` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::tool_result_array_with_image_preserved` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::stop_sequence_null_only_when_present` - `crates/headroom-proxy/tests/integration_bedrock_invoke.rs::tool_use_input_byte_equal_preserves_key_order` ### Acceptance criteria - All new tests pass. - Manual test against a real Bedrock endpoint (developer's AWS account) succeeds. - Existing fake Bedrock path (`headroom/backends/litellm.py`) still works in Python; this PR adds the Rust path alongside. ### Blocked by PR-C1. ### Blocks PR-D2, PR-D3, PR-H2. ### Rollback `git revert`. Bedrock requests fall back to Python LiteLLM converter (the fake path). No regression for users who weren't using Rust Bedrock. ### Notes - The SigV4 signing scope: `host`, `x-amz-date`, `x-amz-content-sha256` headers + canonical request body. Compute the body hash AFTER any compression mutations. - `accept-encoding` is preserved end-to-end (PAYG/OAuth/subscription all preserve it for Bedrock — there's no legacy CLI to mimic; the Bedrock SDK negotiates compression natively). --- ## PR-D2 — Bedrock streaming via binary EventStream **Branch:** `realign-D2-bedrock-event-stream` **Worktree:** `~/claude-projects/headroom-worktrees/realign-D2-bedrock-event-stream` **Risk:** **HIGH** (binary protocol, not SSE) **LOC:** +1100 ### Scope Add `POST /model/{model}/invoke-with-response-stream` route. Bedrock's streaming uses **binary EventStream** (vnd.amazon.eventstream content type), not SSE. Build a parser/forwarder for it. Translate to Anthropic SSE for Anthropic-shape responses (so the existing `AnthropicStreamState` from PR-C1 can run telemetry). ### Files **Add:** - `crates/headroom-proxy/src/bedrock/eventstream.rs` — EventStream binary parser. Format: 12-byte prelude (length + headers length + CRC32 of prelude), N bytes of headers, payload, 4-byte CRC32 of message. Parse incrementally; yield `EventStreamMessage { headers: HashMap, payload: Bytes }`. - `crates/headroom-proxy/src/bedrock/eventstream_to_sse.rs` — for Anthropic-shape Bedrock responses, each `EventStreamMessage` whose `:event-type` header is `chunk` carries an Anthropic SSE event in its payload. Re-emit as SSE to the client. (Or pass through as EventStream — choose based on the `Accept` header from the client.) - `crates/headroom-proxy/src/bedrock/invoke_streaming.rs` — POST handler. **Modify:** - `crates/headroom-proxy/src/lib.rs` — route `/model/{model}/invoke-with-response-stream`. - `crates/headroom-proxy/src/sse/anthropic.rs` — accept events from EventStream-translated source. **Tests added:** - `crates/headroom-proxy/tests/integration_bedrock_streaming.rs::eventstream_parses_correctly` - `crates/headroom-proxy/tests/integration_bedrock_streaming.rs::eventstream_translated_to_sse` - `crates/headroom-proxy/tests/integration_bedrock_streaming.rs::usage_extracted_from_translated_stream` - `crates/headroom-proxy/tests/integration_bedrock_streaming.rs::client_can_choose_eventstream_or_sse` - Property test: `proptest! { fn eventstream_parser_no_panic(bytes in any::<Vec<u8>>()) { let _ = parse(bytes); } }` ### Acceptance criteria - All tests pass. - Manual test against real Bedrock streaming endpoint succeeds. ### Blocked by PR-D1. ### Blocks PR-H2. ### Rollback `git revert`. Streaming Bedrock falls back to Python LiteLLM. --- ## PR-D3 — Bedrock-side observability + auth-mode integration **Branch:** `realign-D3-bedrock-observability` **Worktree:** `~/claude-projects/headroom-worktrees/realign-D3-bedrock-observability` **Risk:** **LOW** **LOC:** +400 ### Scope Per-Bedrock-model metrics, region tagging, IAM role attribution, and integration with auth-mode policy (Bedrock IAM = "oauth" mode by default; passthrough-prefer compression). ### Files **Modify:** - `crates/headroom-proxy/src/bedrock/invoke.rs` — auth_mode classification: when an inbound request hits `/model/.../invoke`, classify as `AuthMode::OAuth` for compression policy. - `crates/headroom-proxy/src/observability/prometheus.rs` — add `bedrock_invoke_count_total{model, region}`, `bedrock_invoke_latency_seconds`, `bedrock_eventstream_message_count_total`. **Add:** - `docs/bedrock.md` — operator docs: how to configure AWS credentials, what models are supported (any `anthropic.claude-*`), what compression behavior to expect (live-zone-only, lossless preferred). **Tests added:** - `crates/headroom-proxy/tests/integration_bedrock_authmode.rs::bedrock_classified_as_oauth` - `crates/headroom-proxy/tests/integration_bedrock_authmode.rs::oauth_policy_passthrough_prefer` ### Acceptance criteria - Tests pass. - Prometheus scrape includes Bedrock metrics. ### Blocked by PR-D2, PR-F1 (auth-mode helper). ### Blocks PR-H2. ### Rollback `git revert`. Loses Bedrock observability; functional path unchanged. --- ## PR-D4 — Native Vertex publisher path **Branch:** `realign-D4-vertex-native` **Worktree:** `~/claude-projects/headroom-worktrees/realign-D4-vertex-native` **Risk:** **HIGH** (new auth + envelope surface) **LOC:** +1300 ### Scope Eliminate P4-38, P4-39 (Vertex parts). Add `POST /v1beta1/projects/{project}/locations/{loc}/publishers/anthropic/models/{model}:rawPredict` and `:streamRawPredict` routes. Vertex auth is GCP ADC (Application Default Credentials) → bearer token. Envelope: `anthropic_version` body field, no `model` field, GCP auth header. ### Files **Add:** - `crates/headroom-proxy/src/vertex/mod.rs` — module re-exports. - `crates/headroom-proxy/src/vertex/adc.rs` — GCP ADC bearer token resolution. Use `gcp_auth` crate. - `crates/headroom-proxy/src/vertex/raw_predict.rs` — POST handler. - `crates/headroom-proxy/src/vertex/stream_raw_predict.rs` — streaming handler. Vertex uses SSE for streaming (unlike Bedrock); the existing `AnthropicStreamState` from PR-C1 works directly. **Modify:** - `crates/headroom-proxy/src/lib.rs` — route Vertex paths. - `Cargo.toml` workspace — add `gcp_auth`. **Tests added:** - `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs::native_envelope_round_trip_byte_equal` - `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs::adc_bearer_token_signed_correctly` - `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs::thinking_block_preserved` - `crates/headroom-proxy/tests/integration_vertex_raw_predict.rs::stream_raw_predict_sse_handled` ### Acceptance criteria - All tests pass. - Manual test against a real Vertex endpoint succeeds. ### Blocked by PR-C1, PR-D1 (envelope pattern). ### Blocks PR-H2. ### Rollback `git revert`. Vertex requests fall back to LiteLLM Python. No regression for non-Rust-Vertex users. --- ## Phase D acceptance summary After all 4 PRs land: - ✅ Native Bedrock `/model/{model}/invoke` route in Rust - ✅ Native Bedrock `/model/{model}/invoke-with-response-stream` (binary EventStream parsed and translated) - ✅ SigV4 signing post-compression - ✅ All Anthropic block types preserved through Bedrock (thinking, redacted_thinking, document, search_result, image, server_tool_use, mcp_tool_use) - ✅ `stop_sequence: null` no longer hardcoded - ✅ `tool_calls.function.arguments` preserved as string - ✅ Native Vertex `:rawPredict` and `:streamRawPredict` routes - ✅ ADC bearer token resolution - ✅ Bedrock/Vertex classified as `AuthMode::OAuth` (passthrough-prefer compression) - ✅ Per-Bedrock-model and per-Vertex-model Prometheus metrics **Phase D retires P4-37, P4-38, P4-39, P4-43.** Marketplace BYOC pitch (per project memory) becomes real. After Phase D, the LiteLLM Python converter is no longer on the request path for Bedrock/Vertex — Phase H deletes it. - REALIGNMENT/07-phase-E-cache-stabilization.md (12347 bytes)
# Phase E — Phase 3 Cache Stabilization **Goal:** Add the cache-stabilization surface that today is **completely missing**: tool array deterministic sort, recursive JSON Schema key sort, auto `cache_control` placement (Anthropic), `prompt_cache_key` auto-injection (OpenAI), volatile-content detector with customer warning (no rewrite), cache-bust drift telemetry. These are guide §8.5, §9.11, §6.2, §4.17 implementations and the "Phase 3" of the guide's implementation checklist. **Calendar:** 1 week. **Shape:** 6 PRs. Mostly parallel; E3 + E4 should land paired (per-mode policy). --- ## PR-E1 — Tool array deterministic sort (Rust) **Branch:** `realign-E1-tool-array-sort` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E1-tool-array-sort` **Risk:** **LOW** **LOC:** +200 ### Scope Eliminate P3-28. Sort `tools[]` alphabetically by name on the way out. Idempotent: re-sorting an already-sorted array is a no-op. Implementation matches Python's existing `_sort_tools_deterministically` (`headroom/proxy/handlers/anthropic.py:34-58`) — same sort key, same output bytes (modulo serialization). ### Files **Add:** - `crates/headroom-proxy/src/compression/tool_def_normalize.rs`: ```rust pub fn sort_tools_deterministically(tools: &mut Vec<&RawValue>) -> Result<()> { // Sort key: tool["name"] string, fallback to MD5(serialized) for unnamed tools. tools.sort_by_key(|t| { let parsed: serde_json::Value = serde_json::from_str(t.get()).unwrap_or_default(); parsed.get("name").and_then(|v| v.as_str()).unwrap_or("").to_string() }); Ok(()) } ``` **Modify:** - `crates/headroom-proxy/src/compression/live_zone_anthropic.rs` — call `sort_tools_deterministically` on the request body's `tools` array before forwarding (only on PAYG; gated by Phase F PR-F2). - `crates/headroom-proxy/src/compression/live_zone_openai.rs` — same. - `crates/headroom-proxy/src/compression/live_zone_responses.rs` — same. **Tests added:** - `crates/headroom-proxy/tests/integration_tool_sort.rs::sort_alphabetic_by_name` - `crates/headroom-proxy/tests/integration_tool_sort.rs::idempotent_resort_no_change` - `crates/headroom-proxy/tests/integration_tool_sort.rs::byte_stable_across_runs` ### Acceptance criteria - Tests pass. - Output `tools[]` byte-equal between Rust and Python sort implementations. ### Blocked by PR-B2. ### Blocks PR-E2. ### Rollback `git revert`. Rust path matches client's tool order (subject to the cache-bust risk). --- ## PR-E2 — Recursive JSON Schema key sort **Branch:** `realign-E2-schema-key-sort` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E2-schema-key-sort` **Risk:** **MEDIUM** (recursive sort over tool schemas; risk of breaking schema semantics if there's an `if`/`then`/`else` or `oneOf` ordering invariant — there isn't, but verify) **LOC:** +300 ### Scope Eliminate P3-29. Recursively sort JSON Schema object keys in every tool's `input_schema`. This includes nested `properties`, `definitions`, `oneOf`, `anyOf`, `allOf`, `if/then/else`, `additionalProperties`, etc. ### Files **Modify:** - `crates/headroom-proxy/src/compression/tool_def_normalize.rs` — add `sort_schema_keys_recursive`. Walks every Object node; replaces with `IndexMap` rebuilt in alphabetic key order. Preserves Array order (JSON Schema arrays are ordered: `prefixItems`, `oneOf` alternatives, etc.). **Tests added:** - `crates/headroom-proxy/tests/integration_schema_sort.rs::flat_schema_keys_sorted` - `crates/headroom-proxy/tests/integration_schema_sort.rs::nested_properties_sorted` - `crates/headroom-proxy/tests/integration_schema_sort.rs::oneof_array_order_preserved` - `crates/headroom-proxy/tests/integration_schema_sort.rs::definitions_keys_sorted` - `crates/headroom-proxy/tests/integration_schema_sort.rs::idempotent_resort` ### Acceptance criteria - Tests pass. - Snapshot test on a real production tool schema (e.g., Claude Code's `Read` tool) — pin the sorted bytes. ### Blocked by PR-E1. ### Blocks PR-E5. ### Rollback `git revert`. Schema keys reflect customer ordering. --- ## PR-E3 — Auto `cache_control` breakpoint placement (Anthropic) **Branch:** `realign-E3-cache-control-auto-place` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E3-cache-control-auto-place` **Risk:** **MEDIUM-HIGH** (auto-adds bytes to client request — only on PAYG) **LOC:** +400 ### Scope Eliminate P3-31. When PAYG mode is detected (Phase F PR-F1) and the customer has not set any `cache_control` markers, auto-place up to 4 ephemeral markers at: 1. End of system prompt (1 marker) 2. End of `tools[]` (1 marker) 3. After the last stable conversation history boundary (1 marker; configurable threshold for "stable") 4. Before the latest user message (1 marker) OAuth and subscription modes: never auto-place (could void scope). ### Files **Add:** - `crates/headroom-proxy/src/compression/cache_control.rs` — `pub fn auto_place_breakpoints(body: &mut serde_json::Value, auth_mode: AuthMode)`. Walks the structure; appends `cache_control: {type: "ephemeral"}` to the trailing block of system, tools, history, and current user message. **Modify:** - `crates/headroom-proxy/src/compression/live_zone_anthropic.rs` — call `auto_place_breakpoints` on PAYG only. **Tests added:** - `crates/headroom-proxy/tests/integration_cache_control_auto.rs::payg_auto_places_4_markers` - `crates/headroom-proxy/tests/integration_cache_control_auto.rs::oauth_no_auto_placement` - `crates/headroom-proxy/tests/integration_cache_control_auto.rs::subscription_no_auto_placement` - `crates/headroom-proxy/tests/integration_cache_control_auto.rs::customer_set_markers_respected_no_addition` - `crates/headroom-proxy/tests/integration_cache_control_auto.rs::ttl_ordering_correct_1h_before_5m` ### Acceptance criteria - Tests pass. - Customer requests with existing markers are unmodified. - A representative PAYG request gets 4 markers in the right positions. ### Blocked by PR-A4, PR-F1, PR-F2. ### Blocks None. ### Rollback `git revert`. Customers without their own `cache_control` markers don't get auto-placement; cache benefit smaller but correct. --- ## PR-E4 — `prompt_cache_key` auto-injection (OpenAI) **Branch:** `realign-E4-prompt-cache-key-inject` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E4-prompt-cache-key-inject` **Risk:** **MEDIUM** **LOC:** +250 ### Scope Eliminate P3-30. For OpenAI Chat Completions and Responses requests on PAYG mode where the customer has not set `prompt_cache_key`, auto-inject one derived from a stable session hash. OAuth/subscription modes: never inject (the CLI may already populate this and Headroom must preserve byte-for-byte). ### Files **Modify:** - `crates/headroom-proxy/src/compression/live_zone_openai.rs` — add `inject_prompt_cache_key` step. - `crates/headroom-proxy/src/compression/live_zone_responses.rs` — same. **Add:** - `crates/headroom-proxy/src/session.rs` — `pub fn derive_prompt_cache_key(session_id: &str, model: &str) -> String` — returns `{session_id}_{model_family}` (deterministic per session+model). **Tests added:** - `crates/headroom-proxy/tests/integration_prompt_cache_key.rs::payg_auto_injects_when_absent` - `crates/headroom-proxy/tests/integration_prompt_cache_key.rs::customer_value_preserved` - `crates/headroom-proxy/tests/integration_prompt_cache_key.rs::oauth_no_injection` - `crates/headroom-proxy/tests/integration_prompt_cache_key.rs::subscription_no_injection` - `crates/headroom-proxy/tests/integration_prompt_cache_key.rs::same_session_same_key_deterministic` ### Acceptance criteria - Tests pass. ### Blocked by PR-F1, PR-F2. ### Blocks None. ### Rollback `git revert`. OpenAI cache routing less sticky on PAYG; functional. --- ## PR-E5 — Volatile-content detector with customer warning (no rewrite) **Branch:** `realign-E5-volatile-detector` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E5-volatile-detector` **Risk:** **LOW** **LOC:** +400 ### Scope Eliminate P3-32. Detect dynamic content in the early prompt (timestamps, UUIDs, JWT tokens, build hashes, randomized IDs) and surface to the customer via a log line and a Prometheus metric. **Do not rewrite.** This is what Python's `cache_aligner.py` was *trying* to do correctly (the rewrite path is gone in Phase A PR-A2). ### Files **Add:** - `crates/headroom-core/src/transforms/volatile_detector.rs`: ```rust pub struct VolatileDetector { /* compiled regex set */ } impl VolatileDetector { pub fn scan(&self, content: &str) -> Vec<VolatileFinding>; } pub struct VolatileFinding { pub kind: VolatileKind, // Timestamp | Uuid | Jwt | BuildHash | ... pub byte_offset: usize, pub matched: String, pub recommendation: String, // "Move to metadata" } ``` Patterns: - ISO 8601 timestamp regex - UUID v4 regex - JWT shape (`eyJ...`) - Long hex strings >=32 chars (build hashes) - Unix epoch timestamps within an order of magnitude of `now` **Modify:** - `crates/headroom-proxy/src/compression/live_zone_anthropic.rs` — run scanner on system prompt; if findings, log warning and increment metric. - `crates/headroom-proxy/src/compression/live_zone_openai.rs` — same on `instructions` field. - `crates/headroom-proxy/src/compression/live_zone_responses.rs` — same. **Tests added:** - `crates/headroom-core/tests/volatile_detector.rs::detects_iso_8601_timestamp` - `crates/headroom-core/tests/volatile_detector.rs::detects_uuid_v4` - `crates/headroom-core/tests/volatile_detector.rs::detects_jwt_shape` - `crates/headroom-core/tests/volatile_detector.rs::detects_build_hash` - `crates/headroom-core/tests/volatile_detector.rs::no_false_positives_on_normal_prose` - `crates/headroom-proxy/tests/integration_volatile.rs::warning_logged_on_volatile_system_prompt` - `crates/headroom-proxy/tests/integration_volatile.rs::system_prompt_bytes_unchanged` ### Acceptance criteria - Tests pass. - Detector runs in <1ms on a 4KB system prompt (don't slow the request path). ### Blocked by PR-E2. ### Blocks None. ### Rollback `git revert`. Detector loses; no functional regression. --- ## PR-E6 — Cache-bust drift detector telemetry **Branch:** `realign-E6-cache-bust-detector` **Worktree:** `~/claude-projects/headroom-worktrees/realign-E6-cache-bust-detector` **Risk:** **LOW** **LOC:** +500 ### Scope Eliminate P3-35. Hash the prefix (system + tools + first N stable messages) of every request. Track per-session prefix hashes. When the prefix hash changes between turns, increment a counter and log a `prefix_drift` event with which subsystem mutated (likely none after Phase A — but if it happens, we want to know). ### Files **Add:** - `crates/headroom-proxy/src/observability/prefix_drift.rs`: ```rust pub struct PrefixDriftDetector { // Keyed by session_id; stores last-seen prefix hash + timestamp. cache: Cache<SessionId, (PrefixHash, Instant)>, } impl PrefixDriftDetector { pub fn check(&self, session_id: &str, body: &serde_json::Value) -> DriftCheck; } pub enum DriftCheck { FirstSeen, Stable, Drifted { previous_hash: PrefixHash, new_hash: PrefixHash, age: Duration }, } ``` **Modify:** - `crates/headroom-proxy/src/observability/prometheus.rs` — add `prefix_drift_detected_total{provider, model}` counter. **Tests added:** - `crates/headroom-proxy/tests/integration_prefix_drift.rs::stable_prefix_no_drift` - `crates/headroom-proxy/tests/integration_prefix_drift.rs::system_change_detected_as_drift` - `crates/headroom-proxy/tests/integration_prefix_drift.rs::tools_reorder_detected` ### Acceptance criteria - Tests pass. - A canary request that mutates the system prompt mid-session triggers the counter. ### Blocked by PR-B2. ### Blocks None. ### Rollback `git revert`. Loses observability; no functional regression. --- ## Phase E acceptance summary After all 6 PRs land: - ✅ Tools alphabetically sorted (deterministic, idempotent) - ✅ JSON Schema keys recursively sorted - ✅ `cache_control` auto-placement on PAYG (4 markers) - ✅ `prompt_cache_key` auto-injection on PAYG (OpenAI) - ✅ Volatile-content detector + customer warning (no rewrite) - ✅ Cache-bust drift telemetry per session **Phase E retires P3-28 through P3-32, P3-35.** - REALIGNMENT/08-phase-F-auth-mode.md (10331 bytes)
# Phase F — Auth-Mode Policy Gates **Goal:** Make auth mode a first-class policy axis. Detect (PAYG / OAuth / subscription) at request entry; gate compression behavior, header injection, and TOIN aggregation per mode. Stealth mode for subscription CLIs. **Calendar:** 1 week. **Shape:** 4 PRs. F1 first; F2 + F3 + F4 parallel after. Reference: `~/.claude/projects/-Users-tchopra-claude-projects-headroom/memory/project_auth_mode_compression_nuances.md`. --- ## PR-F1 — `classify_auth_mode` helper **Branch:** `realign-F1-classify-auth-mode` **Worktree:** `~/claude-projects/headroom-worktrees/realign-F1-classify-auth-mode` **Risk:** **LOW** **LOC:** +500 ### Scope Single helper called at request entry returning `AuthMode = Payg | OAuth | Subscription`. Pure-function classification from headers (Authorization shape, OpenAI-Beta, anthropic-beta) and User-Agent prefix. ### Files **Add:** - `crates/headroom-core/src/auth_mode.rs`: ```rust #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum AuthMode { Payg, OAuth, Subscription } pub fn classify(headers: &http::HeaderMap) -> AuthMode { let ua = headers.get("user-agent").and_then(|h| h.to_str().ok()).unwrap_or("").to_lowercase(); const SUBSCRIPTION_UA_PREFIXES: &[&str] = &[ "claude-cli/", "claude-code/", "codex-cli/", "cursor/", "claude-vscode/", "github-copilot/", "anthropic-cli/", "antigravity/", ]; if SUBSCRIPTION_UA_PREFIXES.iter().any(|p| ua.contains(p)) { return AuthMode::Subscription; } let auth = headers.get("authorization").and_then(|h| h.to_str().ok()).unwrap_or(""); if auth.starts_with("Bearer ") { let token = &auth[7..]; if token.starts_with("sk-ant-api") || token.starts_with("sk-") { return AuthMode::Payg; } if token.starts_with("sk-ant-oat-") || token.split('.').count() >= 3 { // sk-ant-oat-* (Claude Pro OAuth) or JWT (Codex/Cursor OAuth) return AuthMode::OAuth; } } // Bedrock / Vertex (no Authorization header from client; signed downstream) if !auth.is_empty() == false && headers.get("x-api-key").is_none() && headers.get("x-goog-api-key").is_none() { return AuthMode::OAuth; } // x-api-key present (Anthropic API key style) if headers.contains_key("x-api-key") { return AuthMode::Payg; } AuthMode::Payg // default } ``` **Add (Python):** - `headroom/proxy/auth_mode.py` — Python port of the same logic. Used in Python paths until Phase H deletes them. **Modify:** - `crates/headroom-core/src/lib.rs` — `pub mod auth_mode;`. - `crates/headroom-proxy/src/proxy.rs` — call `classify` at request entry; store in request extensions for downstream handlers. - `headroom/proxy/handlers/anthropic.py` — call Python `classify_auth_mode(headers)` at request entry. - `headroom/proxy/handlers/openai.py` — same. **Tests added:** - `crates/headroom-core/tests/auth_mode.rs::api_key_classified_payg` - `crates/headroom-core/tests/auth_mode.rs::oauth_jwt_classified_oauth` - `crates/headroom-core/tests/auth_mode.rs::oauth_sk_ant_oat_classified_oauth` - `crates/headroom-core/tests/auth_mode.rs::claude_code_ua_classified_subscription` - `crates/headroom-core/tests/auth_mode.rs::cursor_ua_classified_subscription` - `crates/headroom-core/tests/auth_mode.rs::no_auth_no_user_agent_default_payg` - `crates/headroom-core/tests/auth_mode.rs::bedrock_no_auth_classified_oauth` - Python equivalents in `tests/test_auth_mode.py`. ### Acceptance criteria - Tests pass. - Detection runs in <10us per call. - Documented in `docs/auth-modes.md` with the detection rules and how to extend. ### Blocked by None. ### Blocks PR-F2, PR-F3, PR-F4. ### Rollback `git revert`. All requests treated as PAYG (current behavior). --- ## PR-F2 — Per-mode compression policy gates **Branch:** `realign-F2-per-mode-policy` **Worktree:** `~/claude-projects/headroom-worktrees/realign-F2-per-mode-policy` **Risk:** **MEDIUM-HIGH** (changes compression behavior per request) **LOC:** +600 ### Scope Wire `AuthMode` into every compression decision per the policy matrix in `02-architecture.md §2.4`. PAYG = aggressive (current default). OAuth = passthrough-prefer (no auto-`cache_control`, no auto-`prompt_cache_key`, no lossy compressors). Subscription = stealth (everything OAuth does + preserve `accept-encoding`, never strip; never inject `X-Headroom-*`; never mutate User-Agent). ### Files **Modify:** - `crates/headroom-proxy/src/compression/live_zone_anthropic.rs` — gate `auto_place_breakpoints` on `auth_mode == Payg`. - `crates/headroom-proxy/src/compression/live_zone_openai.rs` — gate `inject_prompt_cache_key` on `auth_mode == Payg`. - `crates/headroom-proxy/src/compression/live_zone.rs` — gate lossy compressors (Kompress text) on `auth_mode == Payg`. OAuth and Subscription get lossless-only compression. - `crates/headroom-proxy/src/headers.rs:103-117` — `add_x_forwarded_headers` becomes `add_x_forwarded_headers_if(auth_mode)`. Skip on Subscription. - `crates/headroom-proxy/src/proxy.rs` — `accept-encoding` strip becomes conditional on `auth_mode != Subscription`. - `headroom/proxy/handlers/anthropic.py` — gate Python compression decisions identically. - `headroom/proxy/handlers/openai.py` — same. **Tests added:** - `crates/headroom-proxy/tests/integration_authmode_policy.rs::payg_aggressive_compression` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::oauth_no_auto_cache_control` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::oauth_no_auto_prompt_cache_key` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::oauth_lossless_only` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::subscription_no_x_forwarded` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::subscription_preserves_accept_encoding` - `crates/headroom-proxy/tests/integration_authmode_policy.rs::subscription_lossless_only` ### Acceptance criteria - Tests pass. - Manual smoke test: a real Claude Code session through the proxy produces no `X-Forwarded-*` upstream and preserves `accept-encoding`. ### Blocked by PR-F1, PR-E3, PR-E4. ### Blocks None. ### Rollback `git revert`. All requests treated as PAYG. No functional regression for PAYG users; OAuth/Subscription users may see scope-rejection or revocation increase. --- ## PR-F3 — TOIN per-tenant aggregation key **Branch:** `realign-F3-toin-per-tenant` **Worktree:** `~/claude-projects/headroom-worktrees/realign-F3-toin-per-tenant` **Risk:** **MEDIUM** **LOC:** -200 / +400 ### Scope Eliminate P5-56. Extend TOIN's aggregation key from `structure_hash` to `(auth_mode, model_family, structure_hash)`. Storage key prefix updated; in-memory dicts re-keyed. Existing observations from before this PR are preserved under a `legacy/` prefix and deprecated over the next 30 days. ### Files **Modify:** - `headroom/telemetry/toin.py:103` — `Pattern` adds `auth_mode: str = "unknown"`, `model_family: str = "unknown"`. (Already covered by Phase B PR-B5; this PR ensures the wiring lands.) - `headroom/telemetry/toin.py:477, 496, 727, 729, 1248, 1256` — change aggregation key to tuple. - `headroom/telemetry/toin.py` — migration helper that walks the legacy `structure_hash`-only store and copies entries under `("unknown", "unknown", structure_hash)` for graceful degrade. - `headroom/telemetry/toin.py` — bumping aggregation key invalidates earlier recommendations; re-publish via the deploy CLI. - `headroom/subscription/tracker.py:166` — replace `_current_token: str` (raw OAuth bearer storage) with `_current_token_id: str` (a one-way hash + last-4 chars for debugging). Polling code adapts to use the actual `Authorization` header per request rather than the stored copy. **Tests added:** - `tests/test_toin_per_tenant.py::test_aggregation_key_includes_auth_mode_model` - `tests/test_toin_per_tenant.py::test_legacy_observations_preserved_under_unknown` - `tests/test_toin_per_tenant.py::test_publish_per_auth_mode_writes_separate_recommendations` - `tests/test_subscription_tracker_token_hardening.py::test_raw_token_not_stored_in_memory` ### Acceptance criteria - Tests pass. - Recommendations file becomes structured `recommendations.toml` with sections per `(auth_mode, model_family)`. - The subscription tracker token-leak risk closed. ### Blocked by PR-B5, PR-F1. ### Blocks None. ### Rollback `git revert`. TOIN reverts to global aggregation. No functional break. --- ## PR-F4 — `X-Forwarded-*` conditional in Rust path **Branch:** `realign-F4-x-forwarded-conditional` **Worktree:** `~/claude-projects/headroom-worktrees/realign-F4-x-forwarded-conditional` **Risk:** **LOW** **LOC:** +100 ### Scope Eliminate P5-53. The Rust proxy currently always adds `X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Host`, `X-Request-Id` to upstream-bound requests. Make this conditional: PAYG → add; OAuth → add; Subscription → skip (fingerprint risk). ### Files **Modify:** - `crates/headroom-proxy/src/headers.rs:103-117` — `add_x_forwarded_headers` takes an `AuthMode` parameter; no-ops on Subscription. **Tests added:** - `crates/headroom-proxy/tests/integration_x_forwarded_authmode.rs::payg_adds_xfwd` - `crates/headroom-proxy/tests/integration_x_forwarded_authmode.rs::oauth_adds_xfwd` - `crates/headroom-proxy/tests/integration_x_forwarded_authmode.rs::subscription_no_xfwd` ### Acceptance criteria - Tests pass. ### Blocked by PR-F1. ### Blocks None. ### Rollback `git revert`. Headers always added. Mild fingerprint regression for Subscription users. --- ## Phase F acceptance summary After all 4 PRs land: - ✅ `classify_auth_mode` helper detects PAYG / OAuth / Subscription - ✅ Per-mode compression policy gates (auto-cache_control, prompt_cache_key, lossy compressors) - ✅ TOIN aggregation key per `(auth_mode, model_family, structure_hash)` - ✅ Subscription tracker doesn't store raw OAuth bearer - ✅ `X-Forwarded-*` skipped on Subscription mode - ✅ `accept-encoding` preserved on Subscription mode **Phase F retires P5-52, P5-53, P5-54, P5-55, P5-56.** After Phase F, fingerprint risk for Subscription CLI users is dramatically reduced. - REALIGNMENT/09-phase-G-rtk-observability.md (9945 bytes)
# Phase G — RTK Breadth + Observability > **SUPERSEDED.** RTK and lean-ctx were removed from Headroom entirely: the > `headroom/rtk/` and `headroom/lean_ctx/` packages, all `--rtk` / `--context-tool` > flags, the wrap-side hooks and hint-file injection, and the proxy-side `rtk gain` > polling are all gone, and `headroom/context_tool_cleanup.py` uninstalls what > earlier versions left on disk. The RTK-specific plan below is historical; the > non-RTK observability items (cache-hit rate, compression ratio, token > validation) were kept. `docs/rtk-architecture.md`, referenced throughout this > document, was deleted with the feature. **Goal:** Extend RTK coverage to more wrap-CLI agents; close the dead `tokens_saved_rtk` data plane; add per-invocation RTK metrics; add the cache-hit-rate, compression-ratio, token-validation observability surface that's missing today. **Calendar:** 1 week. **Shape:** 3 PRs. **Decision context:** Per Agent F audit and 2026-05-01 user direction, **RTK stays on the wrap-CLI side, NOT the proxy side**. Proxy-side invocation is rejected because (a) cache hot zone risk on tool_result content compression, (b) parallel implementation with `crates/headroom-core/src/transforms/log_compressor.rs`, (c) RTK rewrites *commands* not *outputs* — different value proposition. "Integrate RTK with everything" reads as "extend wrap-CLI breadth + close the data plane + observability." --- ## PR-G1 — Wrap CLI breadth: cline, continue, goose, openhands **Branch:** `realign-G1-wrap-more-agents` **Worktree:** `~/claude-projects/headroom-worktrees/realign-G1-wrap-more-agents` **Risk:** **LOW** **LOC:** +800 ### Scope Eliminate P5-62. Add `headroom wrap cline`, `headroom wrap continue`, `headroom wrap goose`, `headroom wrap openhands` (extending the existing pattern from `wrap claude` / `wrap codex` / `wrap aider` / `wrap copilot` / `wrap cursor`). Each wrap subcommand: 1. Ensures the RTK binary is installed (`_ensure_rtk_binary()`). 2. Injects the `<!-- headroom:rtk-instructions -->` block into the agent's instruction file (AGENTS.md / .cursorrules / etc.). 3. Spawns the proxy (or attaches to a running one). 4. Launches the agent CLI with proxy env-var overrides. ### Files **Add:** - `headroom/cli/wrap/cline.py` — wrap implementation for Cline (agent that lives in VS Code; instruction file is `.clinerules`). - `headroom/cli/wrap/continue_dev.py` — Continue agent (`.continue/config.json` configuration; system message injection). - `headroom/cli/wrap/goose.py` — Goose agent (Block's CLI; `.goose/config.yaml`). - `headroom/cli/wrap/openhands.py` — OpenHands (instruction injection via `OPENHANDS_INSTRUCTIONS` env var). **Modify:** - `headroom/cli/wrap/__init__.py` — register new subcommands. - `headroom/cli/main.py` — `headroom wrap --help` lists new agents. - `e2e/wrap/run.py` — extend the e2e runner to exercise the new wrappers (each wrapper has a smoke test that asserts: binary installed, instruction injected, proxy started, dummy LLM call works). **Tests added:** - `tests/test_cli/test_wrap_cline.py::test_wrap_cline_smoke` - `tests/test_cli/test_wrap_continue.py::test_wrap_continue_smoke` - `tests/test_cli/test_wrap_goose.py::test_wrap_goose_smoke` - `tests/test_cli/test_wrap_openhands.py::test_wrap_openhands_smoke` - `tests/test_cli/test_wrap_idempotent_inject.py::test_double_injection_no_duplicate_block` (for each new wrapper) ### Acceptance criteria - Tests pass. - Manual test: `headroom wrap cline -- claude-3-7-sonnet` launches a Cline session with the proxy in-front and RTK instructions in `.clinerules`. ### Blocked by None. ### Blocks None. ### Rollback `git revert`. Existing wrappers continue working; new ones absent. ### Notes - **Future agents to add later (not in this PR):** Roo Code, Devin-style CLIs, raw `gh copilot` standalone, gpt-engineer, sweep, smol-developer. Add as separate PRs as adoption justifies. --- ## PR-G2 — Wire `tokens_saved_rtk` data plane **Branch:** `realign-G2-tokens-saved-rtk` **Worktree:** `~/claude-projects/headroom-worktrees/realign-G2-tokens-saved-rtk` **Risk:** **LOW** **LOC:** +200 ### Scope Eliminate P5-60. The `tokens_saved_rtk` field on `SubscriptionContribution` (`headroom/subscription/models.py:260`) exists but is never populated. Wire it: poll `rtk gain --format json` periodically (already done by `_get_rtk_stats` in `helpers.py:132`), diff the cumulative `tokens_saved` since last snapshot, and feed into `tracker.update_session_savings(tokens_saved_rtk=delta)`. ### Files **Modify:** - `headroom/subscription/tracker.py` — add `_last_rtk_tokens_saved: int = 0` state; on every `update_session_savings` call, fetch `_get_rtk_stats()`, compute `delta = current.tokens_saved - self._last_rtk_tokens_saved`, set `tokens_saved_rtk=delta`, update state. - `headroom/proxy/helpers.py:132` — `_get_rtk_stats` returns `RtkStats { invocations: int, tokens_saved: int, last_run_at: datetime }`. Memoization stays at 5s. **Tests added:** - `tests/test_subscription_tracker_rtk_wired.py::test_tokens_saved_rtk_populated_from_rtk_stats` - `tests/test_subscription_tracker_rtk_wired.py::test_delta_computed_correctly_across_polls` - `tests/test_subscription_tracker_rtk_wired.py::test_rtk_failure_zero_delta_no_throw` ### Acceptance criteria - Tests pass. - A wrap session with RTK invocations produces `tokens_saved_rtk > 0` after the session ends. ### Blocked by None. ### Blocks None. ### Rollback `git revert`. `tokens_saved_rtk` returns to silent zero. --- ## PR-G3 — Per-invocation RTK metrics + observability gaps **Branch:** `realign-G3-rtk-metrics-and-obs` **Worktree:** `~/claude-projects/headroom-worktrees/realign-G3-rtk-metrics-and-obs` **Risk:** **LOW** **LOC:** +600 ### Scope Eliminate P6-68, P6-69, P5-58, P4-41, P4-42, P4-45, and P5-61 (documentation). Add Prometheus metrics: - `wrap_rtk_invocations_total{tool}` — derived from `rtk gain --format json` polling (`tool` label is the `git`, `ls`, `cargo`, etc. command). - `wrap_rtk_tokens_saved_per_session` — histogram, populated at session end. - `proxy_cache_hit_rate_per_session` — histogram, computed from `usage.cache_read_input_tokens / total_input_tokens` per session. - `proxy_compression_ratio_by_strategy{strategy, content_type}` — histogram. - `proxy_compression_rejected_by_token_check_total{strategy}` — counter (already in PR-B4; ensure it's exported here). - `proxy_passthrough_bytes_modified_total{path}` — gauge that **must stay 0** outside compression-on path. Alarm if non-zero. - `proxy_rate_limit_remaining_*` — extracted from upstream response headers. - `proxy_service_tier_count_total{tier}` — counter for `service_tier` distribution. - `proxy_response_status_count_total{status}` — `incomplete | failed | cancelled | completed | in_progress`. - `proxy_image_generation_call_log_redacted_total` — counter for log redactions of multi-MB base64. Plus image base64 log redaction (P4-45) lands here. ### Files **Modify:** - `crates/headroom-proxy/src/observability/prometheus.rs` — add all new metrics. - `crates/headroom-proxy/src/sse/anthropic.rs` — emit `proxy_cache_hit_rate_per_session` from `usage.cache_read_input_tokens / total_input_tokens` on `message_delta`. - `crates/headroom-proxy/src/sse/openai_responses.rs` — emit on `response.completed`. - `crates/headroom-proxy/src/sse/openai_chat.rs` — emit on final usage chunk. - `crates/headroom-proxy/src/handlers/responses.rs` — extract and log `service_tier`. - `crates/headroom-proxy/src/handlers/responses.rs` — log `incomplete_details.reason` when `status == incomplete`. - `headroom/proxy/request_logger.py` — redact base64 strings >1024 bytes; replace with `<base64 truncated, X bytes>`. - `crates/headroom-proxy/src/observability/cache_hit_rate.rs` — new module. - `crates/headroom-proxy/src/observability/compression_ratio.rs` — new module. **Add:** - `docs/observability.md` — documents every metric, what it means, what an operator should do when it drifts. - `docs/rtk-architecture.md` — explicitly documents the decision: RTK is wrap-CLI-only; proxy-side invocation is rejected. Includes the rationale (cache hot zone, parallel-impl with log_compressor, command-rewrite-vs-output-rewrite). Future contributors hit this doc before considering a proxy-side RTK call. **Tests added:** - `crates/headroom-proxy/tests/integration_metrics.rs::cache_hit_rate_emitted_per_session` - `crates/headroom-proxy/tests/integration_metrics.rs::compression_ratio_emitted_per_strategy` - `crates/headroom-proxy/tests/integration_metrics.rs::passthrough_bytes_modified_zero_when_no_compression` - `crates/headroom-proxy/tests/integration_metrics.rs::service_tier_logged` - `crates/headroom-proxy/tests/integration_metrics.rs::incomplete_status_logged_with_reason` - `tests/test_image_log_redaction.py::test_large_base64_truncated` ### Acceptance criteria - All tests pass. - Manual scrape of `/metrics` shows the new metric families. - `docs/rtk-architecture.md` reviewed and approved. ### Blocked by None. ### Blocks None. ### Rollback `git revert`. Loses observability; no functional regression. --- ## Phase G acceptance summary After all 3 PRs land: - ✅ Wrap CLI coverage extends to cline, continue, goose, openhands - ✅ `tokens_saved_rtk` field populated end-to-end - ✅ Per-invocation RTK Prometheus metrics - ✅ Per-session cache-hit-rate metric - ✅ Per-block compression-ratio histogram - ✅ Token-validation rejection counter - ✅ Passthrough-bytes-modified gauge (alarm-able) - ✅ Rate-limit headers observed and exported - ✅ `service_tier` distribution metric - ✅ Response status (`incomplete | failed | cancelled`) logged with reason - ✅ Image base64 log redaction - ✅ `docs/rtk-architecture.md` documents the keep-RTK-on-wrap-side decision **Phase G retires P4-41, P4-42, P4-45, P5-58, P5-60, P5-61, P5-62, P6-68, P6-69, P6-72.** - REALIGNMENT/10-phase-H-python-retirement.md (9738 bytes)
# Phase H — Python Proxy Retirement **Goal:** With Rust at full parity (Phases A–G), delete the Python proxy server, handlers, transforms, and supporting modules. Keep Python only where it's the right tool: CLI wrappers, RTK installer, evals, learn, memory writers, tokenizers (parity backstop). **Calendar:** 2 weeks. **Shape:** 3 PRs. H1 retires the request-path Python; H2 retires Bedrock/Vertex backend; H3 cleans up. **Pre-requisites:** - Phase A–G complete. - Real-traffic shadow test (Phase I) shows Rust ≥99.9% byte-equality vs Python on representative traffic. - Cache-hit-rate parity with direct upstream confirmed (Phase G observability). - All Bedrock/Vertex paths covered by native Rust handlers (Phase D). --- ## PR-H1 — Retire Python proxy request path **Branch:** `realign-H1-retire-python-proxy-request-path` **Worktree:** `~/claude-projects/headroom-worktrees/realign-H1-retire-python-proxy-request-path` **Risk:** **HIGH** (production-facing change; canary deploy mandatory) **LOC:** **-15,000 / +500** ### Scope Delete the Python FastAPI server, all handlers, the responses converter (already gone in Phase C PR-C5), memory subsystem (replaced by live-zone tail injection from Phase A PR-A2 and Phase B PR-B6), semantic cache, batch handler, etc. The Rust proxy becomes the canonical request-path implementation. Operators run only `headroom-proxy` (Rust binary). ### Files **Delete:** - `headroom/proxy/server.py` (2864 LOC) - `headroom/proxy/handlers/anthropic.py` (2423 LOC) - `headroom/proxy/handlers/openai.py` (2742 LOC) - `headroom/proxy/handlers/streaming.py` (1131 LOC) - `headroom/proxy/handlers/gemini.py` (839 LOC) - `headroom/proxy/handlers/batch.py` (1010 LOC) - `headroom/proxy/responses_converter.py` — already deleted in PR-C5; verify gone. - `headroom/proxy/memory_handler.py` (1756 LOC) - `headroom/proxy/memory_tool_adapter.py` (1273 LOC) - `headroom/proxy/semantic_cache.py` (142 LOC) - `headroom/proxy/savings_tracker.py` (934 LOC) — re-implemented in Rust as part of `observability/`. - `headroom/proxy/loopback_guard.py` (92 LOC) — Rust equivalent in `crates/headroom-proxy/src/handlers/debug.rs`. - `headroom/proxy/ws_session_registry.py` (226 LOC) — Rust equivalent. - `headroom/proxy/interceptors/` — all of this directory. - `headroom/proxy/cost.py`, `helpers.py`, `rate_limiter.py`, `request_logger.py` — re-implemented in Rust as part of compression dispatch / observability. - `headroom/proxy/prometheus_metrics.py` — re-implemented in Rust as `observability/prometheus.rs`. - `headroom/proxy/extensions.py`, `models.py`, `modes.py`, `stage_timer.py`, `warmup.py`, `responses_converter.py`, `debug_introspection.py`. - `headroom/transforms/cache_aligner.py` — already gutted in Phase A PR-A2; delete the remaining stub. **Move:** - `headroom/proxy/loopback_guard.py` test logic → `crates/headroom-proxy/tests/integration_loopback_guard.rs`. **Modify:** - `headroom/cli/proxy.py` — `headroom proxy start` now spawns the Rust binary (`./target/release/headroom-proxy`) instead of `uvicorn headroom.proxy.server:app`. - `headroom/cli/wrap/*.py` — same: env-var setup remains, but `proxy_url` points at the Rust binary's listen address. - `pyproject.toml` — remove `fastapi`, `uvicorn`, `pydantic`, etc. from runtime deps; keep them in dev/test deps for parity harness only. - `Dockerfile` — drop the Python proxy server stage; the Rust binary is the only proxy. - `docker-compose.yml` — same. - `RUST_DEV.md` — promote the Rust proxy from "Phase 1 transparent reverse proxy" to "the proxy." - All operator runbooks in `wiki/` and `docs/` — update to reference Rust binary. **Tests deleted:** - `tests/test_proxy_*.py` — most of these (which test the Python proxy directly). Keep tests that exercise CLI wrappers, RTK, evals, learn, memory writers, tokenizers. - Roughly 150 test files; keep ~40 that don't depend on the Python proxy. **Tests added:** - `e2e/proxy_full/test_e2e_canary.py` — deploys the Rust binary in a container; runs a full conversation suite; asserts cache hit rate, compression value, no 5xx errors. Run pre-merge in CI. ### Acceptance criteria - `pytest -x` green (after deletions). - `cargo test --workspace` green. - `make ci-precheck` green. - E2E canary in CI passes. - Manual test: `headroom proxy start` boots the Rust binary; `curl -s http://127.0.0.1:8787/healthz` returns OK. - Operator deploys the new image to staging; cache hit rate ≥ Python baseline; no 5xx regressions in 24h. ### Blocked by PR-A1 through PR-G3. ### Blocks PR-H2. ### Rollback `git revert` of just this PR restores the Python proxy. **Critical**: keep the previous container image around for at least 30 days so operators can pin to the pre-H1 image. Document the rollback path in `docs/operations/rollback.md`. ### Notes - This is the largest single PR in the realignment. Coordinate with operations team. - Do NOT delete in one giant commit; split into a series of smaller commits within the PR (one per module deletion) for git-blame friendliness. --- ## PR-H2 — Retire LiteLLM Bedrock/Vertex backend **Branch:** `realign-H2-retire-litellm-backends` **Worktree:** `~/claude-projects/headroom-worktrees/realign-H2-retire-litellm-backends` **Risk:** **MEDIUM** **LOC:** **-3,000 / +100** ### Scope After Phase D PR-D1..D4 added native Bedrock/Vertex Rust paths, the LiteLLM Python converter is no longer on any request path. Delete it. ### Files **Delete:** - `headroom/backends/litellm.py` (~1500 LOC, the lossy converter) - `headroom/backends/__init__.py` if the only contents was the LiteLLM backend. **Modify:** - `pyproject.toml` — remove `litellm` from dependencies. (Saves ~50 MB of installed-deps size.) - `headroom/providers/registry.py` — delete `litellm-bedrock`, `litellm-vertex` provider entries. - `headroom/cli/wrap/*` — verify no wrap CLIs route to LiteLLM (they shouldn't; they go through the proxy). **Tests deleted:** - `tests/test_backends_litellm*.py` ### Acceptance criteria - `pytest -x` green. - A real Bedrock request through the Rust proxy succeeds (already validated in Phase D PR-D1 manual test). ### Blocked by PR-H1, PR-D1, PR-D2, PR-D4. ### Blocks PR-H3. ### Rollback `git revert`. Restores LiteLLM. The Rust native paths from Phase D stay in place; both run side-by-side temporarily. --- ## PR-H3 — Final cleanup: orphaned modules, deps, docs **Branch:** `realign-H3-final-cleanup` **Worktree:** `~/claude-projects/headroom-worktrees/realign-H3-final-cleanup` **Risk:** **LOW** **LOC:** -2,000 / +500 ### Scope Sweep up everything orphaned by H1+H2: unused imports, dead test fixtures, stale docs, legacy CLI commands. Update all README / wiki / docs to reflect the Rust-only proxy. ### Files **Modify:** - `README.md` — operator-facing docs reflect the Rust binary. - `wiki/` — refresh. - `docs/` — refresh. - `RUST_DEV.md` — final form (renamed to `DEV.md` since there's no longer a Python/Rust split). - `headroom/__init__.py` — drop unused module imports. - `pyproject.toml` — final dependency cleanup. - `Cargo.toml` — final workspace cleanup. **Delete:** - `tests/parity/fixtures/` — most of these are used by Python parity comparators that no longer have a Python side. Keep only the fixtures that still gate Rust-vs-Rust parity (which is: none, after Phase H — though future ML compressor variants may want them back). - `crates/headroom-parity/` — the parity harness itself becomes irrelevant after Python side is gone. **Decision needed** (see `12-decisions-needed.md` Q3): keep parity-run as a "prior-version-vs-current-version" harness, or delete? **Add:** - `CHANGELOG.md` entry: "**Breaking**: Python proxy retired. Operators must use the Rust binary `headroom-proxy`. See migration guide at `docs/operations/python-to-rust-migration.md`." - `docs/operations/python-to-rust-migration.md` — operator migration guide. ### Acceptance criteria - `git grep -i "uvicorn\|fastapi" headroom/` returns nothing in non-test code. - `pyproject.toml` runtime deps are minimal. - All docs build. ### Blocked by PR-H1, PR-H2. ### Blocks None. ### Rollback `git revert`. Restores cleanup; previous PRs stay. --- ## What survives in Python after Phase H | Module | Role | Reason | |---|---|---| | `headroom/cli/wrap/*.py` | Agent launchers | Off-path; orchestrates filesystem + subprocess. Python is the right tool. | | `headroom/cli/{evals,init,install,learn,memory,perf,proxy,tools}.py` | CLI admin commands | Click-based; off-path. | | `headroom/rtk/installer.py` | RTK binary downloader | Off-path; filesystem operations. | | `headroom/providers/codex/install.py`, `claude/install.py` | Client config installation | Off-path; filesystem. | | `headroom/evals/`, `learn/`, `memory/` writers | Research / batch tooling | Off-path; long-running batch. | | `headroom/tokenizers/` | Parity backstop | Used only by parity harness if H3 keeps it. | | `headroom/telemetry/toin.py` | TOIN learning loop | Off-path; observation-only after Phase B PR-B5. | | `headroom/subscription/tracker.py`, `client.py` | Subscription usage poller | Off-path. | | `headroom/copilot_auth.py` | Copilot OAuth refresh | Off-path; specific to Copilot integration. | ## Phase H acceptance summary After all 3 PRs land: - ✅ Python proxy server retired - ✅ All Python proxy handlers deleted - ✅ Memory subsystem refactored or deleted - ✅ LiteLLM backend retired - ✅ Operators run only the Rust `headroom-proxy` binary - ✅ ~20 K LOC of Python deleted - ✅ Migration guide for operators **Phase H is the deletion. The OSS surface area shrinks dramatically; maintenance debt drops; behavior becomes consistent across deployments.** - REALIGNMENT/11-phase-I-test-infra.md (13636 bytes)
# Phase I — Test Infrastructure (Continuous, Parallel) **Goal:** Build the test/CI surface that makes the realignment safe to land and stays safe afterward. This phase runs **in parallel** with all other phases — its PRs land alongside the corresponding feature work. **Calendar:** Continuous. Each test PR pairs with the feature PR it gates. **Shape:** ~10 PRs, mostly small, parallelizable. --- ## PR-I1 — SHA-256 byte-faithful round-trip test on recorded production payload **Branch:** `realign-I1-sha256-round-trip` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I1-sha256-round-trip` **Risk:** **LOW** **LOC:** +400 ### Scope Eliminate P6-63. The single most important regression test for cache safety. Records a real Anthropic `/v1/messages` payload (sanitized of secrets), sends it through the proxy with compression off, asserts SHA-256 byte-equality at the upstream mock. ### Files **Add:** - `tests/fixtures/anthropic_messages_request_real.json` — sanitized real payload. Includes: - `system` as a list of blocks with `cache_control` markers - `tools[]` with non-trivial JSON Schema (nested properties, oneOf, definitions) - `messages[]` with mixed block types: text, image, thinking + signature, tool_use with non-trivial input, tool_result with array content + image - Non-ASCII characters (`🔥`, CJK) - Numeric values: `temperature: 1.0`, large integers, scientific notation - `cache_control` markers on `messages[*].content[*]` - `null` and absent fields side-by-side - `tests/fixtures/openai_chat_completions_real.json` — same shape for OpenAI Chat. - `tests/fixtures/openai_responses_real.json` — same shape for Responses, includes V4A patch, local_shell_call, reasoning, compaction items. - `crates/headroom-proxy/tests/integration_byte_faithful.rs::sha256_round_trip_anthropic_messages_passthrough` - `crates/headroom-proxy/tests/integration_byte_faithful.rs::sha256_round_trip_anthropic_messages_compression_off_via_auth_mode` - `crates/headroom-proxy/tests/integration_byte_faithful.rs::sha256_round_trip_openai_chat` - `crates/headroom-proxy/tests/integration_byte_faithful.rs::sha256_round_trip_openai_responses` - `tests/test_python_byte_faithful.py::test_sha256_round_trip_anthropic_passthrough` — Python side, gates Phase H readiness. **Modify:** - `Makefile` — `make test-byte-faithful` target that runs all of the above. - `.github/workflows/rust.yml` — make `make test-byte-faithful` a per-PR gate. ### Acceptance criteria - All tests pass after Phase A PR-A3, PR-A4 land. - Test runs in <5 seconds. ### Blocked by PR-A1. ### Blocks PR-H1 (Phase H gating). --- ## PR-I2 — SSE corner-case fixtures + fuzz tests **Branch:** `realign-I2-sse-corner-cases` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I2-sse-corner-cases` **Risk:** **LOW** **LOC:** +800 ### Scope Eliminate P6-66, P6-71. Record fixtures for every SSE corner case the audit identified. ### Files **Add:** - `crates/headroom-proxy/tests/fixtures/sse/anthropic_thinking_with_signature.sse` - `crates/headroom-proxy/tests/fixtures/sse/anthropic_interleaved_blocks.sse` (synthetic; locks the index-keyed model) - `crates/headroom-proxy/tests/fixtures/sse/anthropic_input_json_delta_split_utf8.sse` (4-byte emoji split across chunks) - `crates/headroom-proxy/tests/fixtures/sse/anthropic_ping_mid_stream.sse` - `crates/headroom-proxy/tests/fixtures/sse/anthropic_error_mid_stream.sse` - `crates/headroom-proxy/tests/fixtures/sse/openai_chat_tool_call_split.sse` - `crates/headroom-proxy/tests/fixtures/sse/openai_chat_done_with_trailing_whitespace.sse` - `crates/headroom-proxy/tests/fixtures/sse/openai_responses_out_of_order_done.sse` - `crates/headroom-proxy/tests/fixtures/sse/openai_429_as_application_json.http` (HTTP error, not SSE) - `crates/headroom-proxy/tests/fixtures/sse/anthropic_tcp_drop_before_message_stop.sse` - `crates/headroom-proxy/tests/integration_sse_fixtures.rs` — runs every fixture against the parser; asserts expected state. - Property test in `crates/headroom-proxy/tests/proptest_sse.rs::sse_parser_no_panic_on_arbitrary_bytes`. ### Acceptance criteria - All fixtures parse correctly. - Property test runs 10K random byte sequences without panic. ### Blocked by PR-C1. ### Blocks None. --- ## PR-I3 — Property tests for compression invariants **Branch:** `realign-I3-compression-proptest` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I3-compression-proptest` **Risk:** **LOW** **LOC:** +500 ### Scope Property tests that exercise the realigned compressor invariants: 1. **Determinism**: `compress(input) == compress(input)` for any valid input. 2. **Idempotence**: `compress(compress(input).output) == compress(input).output` (compressing already-compressed content is a no-op). 3. **Token-non-increasing**: `tokens(output) <= tokens(input)` for any valid input — fallback ensures this. 4. **Position preservation**: For all valid block arrays, `len(compressed) == len(original)`; block types match per index; `tool_use_id` / `call_id` preserved. 5. **Frozen-prefix integrity**: For any `frozen_count`, messages `0..frozen_count` are byte-equal in input and output. ### Files **Add:** - `crates/headroom-core/tests/proptest_compression.rs` — proptest strategies for `Block`, `Message`, `RequestBody`. Five property tests above. - `crates/headroom-core/tests/proptest_ccr.rs` — round-trip property test: `decompress(compress(content)) == content` for any content. ### Acceptance criteria - All property tests pass with `cases = 1000`. ### Blocked by PR-B4. ### Blocks None. --- ## PR-I4 — Real-traffic shadow test (Python vs Rust) **Branch:** `realign-I4-shadow-test` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I4-shadow-test` **Risk:** **MEDIUM** **LOC:** +1000 ### Scope Eliminate P6-67. A canary deployment runs the Python proxy and the Rust proxy side-by-side; for every request, both produce upstream-bound bytes; a comparator hashes both and reports SHA-256 mismatch percentage. Goal: 99.9% byte-equality before Phase H deletes Python. ### Files **Add:** - `e2e/shadow/runner.py` — splits incoming requests into "primary" (Python, response goes to client) and "shadow" (Rust, response discarded). Hashes upstream-bound bytes from both; reports per-request, per-endpoint, per-auth-mode mismatch rates. - `e2e/shadow/dashboard.py` — Grafana dashboard JSON that visualizes the shadow comparison. - `docs/operations/shadow-deploy.md` — operator guide for running the shadow test. ### Acceptance criteria - Shadow test runs against a non-trivial corpus (10K requests) and reports. - Mismatch rate <0.1% before declaring Phase H ready. ### Blocked by PR-A1 through PR-G3. ### Blocks PR-H1. --- ## PR-I5 — Promote stub parity comparators to real **Branch:** `realign-I5-parity-stubs-to-real` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I5-parity-stubs-to-real` **Risk:** **MEDIUM** **LOC:** +800 ### Scope Eliminate P6-64. `crates/headroom-parity/src/lib.rs:172-174` stubs three comparators with `bail!()`: - `ccr` (25 fixtures recorded; comparator is `Skipped`) - `log_compressor` (20 fixtures recorded; comparator is `Skipped`) - `cache_aligner` (20 fixtures recorded; comparator is `Skipped`) Build real comparators that exercise the Rust port against the recorded Python fixtures. ### Files **Modify:** - `crates/headroom-parity/src/lib.rs` — replace `stub_comparator!(CCRComparator, ...)` etc. with real impls. - Add `CcrComparator`, `LogCompressorComparator`, `CacheAlignerComparator` modules. **Tests added:** - Each comparator has a `harness_reports_match_for_real_fixture` test. ### Acceptance criteria - All three comparators run against their recorded fixtures. - Mismatch rate is 0% (parity locked). ### Blocked by PR-B3 (LogCompressor live in proxy); PR-B7 (CCR hardening); PR-A2 (CacheAligner detector). ### Blocks PR-I6. --- ## PR-I6 — Make `make test-parity` a per-PR CI gate **Branch:** `realign-I6-parity-per-pr-gate` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I6-parity-per-pr-gate` **Risk:** **LOW** **LOC:** +50 ### Scope Eliminate P6-65. Today parity is a soft nightly with `continue-on-error: true`. Move it to per-PR with `Diff` failures blocking merge; `Skipped` allowed (so still-stubbed comparators don't block). ### Files **Modify:** - `.github/workflows/rust.yml:125-149` — move parity job from `cron` schedule to `pull_request` trigger. Remove `continue-on-error`. Set parity-run flags so `Skipped` is acceptable but `Diff` fails the build. - `Makefile` — `test-parity` already exists; ensure it's invokable in CI. ### Acceptance criteria - A purposely-broken Rust port that diverges from a recorded fixture fails CI on the next PR. ### Blocked by PR-I5. ### Blocks None. --- ## PR-I7 — Cache hot zone non-mutation tests **Branch:** `realign-I7-cache-hot-zone-tests` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I7-cache-hot-zone-tests` **Risk:** **LOW** **LOC:** +600 ### Scope Test that nothing — compression, memory injection, tool registration — mutates the cache hot zone. ### Files **Add:** - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::system_byte_equal_under_compression` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::tools_byte_equal_under_compression` (modulo Phase E sort + schema-key sort, which is deterministic — assert post-sort byte-equal) - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::frozen_messages_byte_equal_under_compression` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::reasoning_encrypted_content_byte_equal` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::thinking_signature_byte_equal` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::redacted_thinking_data_byte_equal` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::compaction_encrypted_content_byte_equal` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::v4a_patch_diff_byte_equal` - `crates/headroom-proxy/tests/integration_cache_hot_zone.rs::local_shell_call_argv_array_preserved` ### Acceptance criteria - All tests pass. ### Blocked by PR-B2. ### Blocks None. --- ## PR-I8 — Tool-definition byte-stability snapshot tests **Branch:** `realign-I8-tool-def-snapshot` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I8-tool-def-snapshot` **Risk:** **LOW** **LOC:** +300 ### Scope For every tool definition Headroom auto-injects (`ccr_retrieve`, `memory_*`), pin the bytes via golden-file snapshot. Any change to a definition fails CI; a deliberate change requires updating the snapshot. This prevents accidental cache busts on Headroom deploys. ### Files **Add:** - `crates/headroom-core/tests/tool_def_byte_stability.rs::ccr_retrieve_definition_anthropic_byte_stable` - `crates/headroom-core/tests/tool_def_byte_stability.rs::ccr_retrieve_definition_openai_byte_stable` - `crates/headroom-core/tests/tool_def_byte_stability.rs::memory_save_definition_byte_stable` - `crates/headroom-core/tests/tool_def_byte_stability.rs::memory_search_definition_byte_stable` - Golden files under `crates/headroom-core/tests/golden/tool_defs/`. ### Acceptance criteria - Tests pass. - Renaming a field in a tool definition fails CI; updating the golden file fixes it. ### Blocked by PR-B7. ### Blocks None. --- ## PR-I9 — Continuous cache-hit-rate alarm **Branch:** `realign-I9-cache-hit-rate-alarm` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I9-cache-hit-rate-alarm` **Risk:** **LOW** **LOC:** +200 ### Scope A Prometheus alarm rule that fires when the per-session cache hit rate (`proxy_cache_hit_rate_per_session`) drops below a baseline (90% of yesterday's rolling p50) for >15 minutes. Catches drift in production. ### Files **Add:** - `docs/operations/prometheus_rules.yaml` — alarm rule definition. - `docs/operations/runbook.md` — what to do when the alarm fires. ### Acceptance criteria - Rule passes `promtool check rules`. - Runbook reviewed. ### Blocked by PR-G3. ### Blocks None. --- ## PR-I10 — Replace fake RTK shim with real RTK in wrap E2E **Branch:** `realign-I10-real-rtk-in-e2e` **Worktree:** `~/claude-projects/headroom-worktrees/realign-I10-real-rtk-in-e2e` **Risk:** **LOW** **LOC:** +200 ### Scope Eliminate P6-72. `e2e/wrap/run.py:250-267` has an `rtk` shim that just prints "rtk shim" and exits 0. Replace with real RTK invocation in CI, OR keep the shim but add an explicit assertion that the shim was used (so it doesn't silently mask a missing RTK install). ### Files **Modify:** - `e2e/wrap/run.py:250-267` — switch to real RTK download in CI; cache the binary. - `.github/workflows/wrap-e2e.yml` — pin RTK version. ### Acceptance criteria - E2E tests download real RTK and exercise its rewrite behavior. ### Blocked by None. ### Blocks None. --- ## Phase I acceptance summary After all 10 PRs land: - ✅ SHA-256 byte-faithful round-trip test gates CI - ✅ SSE corner-case fixtures + fuzz tests - ✅ Property tests for compression invariants - ✅ Real-traffic shadow test comparing Python vs Rust - ✅ Stub parity comparators promoted to real - ✅ `make test-parity` is a per-PR gate - ✅ Cache hot zone non-mutation tests - ✅ Tool-definition byte-stability snapshot tests - ✅ Cache-hit-rate Prometheus alarm - ✅ Real RTK in wrap E2E **Phase I retires P6-63 through P6-72.** After Phase I, regressing the realignment requires actively breaking tests — the cache safety properties become continuously enforced.
- REALIGNMENT/12-decisions-needed.md (10901 bytes)
# 12 — Decisions Needed Open questions the realignment can't resolve unilaterally. Greenlight or alternative each before the corresponding PR lands. --- ## Q1. Phase A timing — land tonight or wait? **Recommendation:** Land **PR-A1 tonight**. It's a small diff (-180/+30) that eliminates the worst cache-killer cluster (P0-3, P0-4, P0-5 stop firing immediately). The proxy goes to passthrough on `/v1/messages`; compression returns in Phase B. Net positive because today's compression is actively destroying cache hit rate. PR-A2 through PR-A8 land over the rest of the week. **Alternative:** Hold all of Phase A until the synthesis is "perfect." Risk: cache hit rate stays poor. --- ## Q2. ICM removal scope — Tier 1+2, or include Tier 3? **Recommendation: Tier 1 + Tier 2 in Phase B PR-B1** (~10K LOC). - **Tier 1** (ICM proper): `intelligent_context.py`, `manager.rs`, `icm.rs`, the proxy call site. - **Tier 2** (subsystems whose only consumer is ICM): `RollingWindow`, `ProgressiveSummarizer`, `scoring.py`, `tool_crusher.py`, `MessageScorer`, all of `crates/headroom-core/src/scoring/` and `relevance/`, most of `context/` (keep `safety.rs`). - **Tier 3** (separable cleanup): `CacheAligner` rewrite path is in Phase A PR-A2 (already scheduled). Memory `_inject_system_context` paths in Phase A PR-A2 + Phase B PR-B6 (already scheduled). So "Tier 1 + Tier 2" is the right scope for the Phase B big-delete PR; Tier 3 is already covered by Phase A and Phase B's other PRs. **Alternative:** Stop at Tier 1 (just ICM proper). Risk: ~6 K LOC of dead-but-still-imported scoring/relevance machinery; future contributors won't know it's dead. --- ## Q3. MessageScorer Rust port — delete? **Recommendation: Delete.** The PR #338 / #343 port (April 2026) was investment in the wrong abstraction (per Agent G's audit: scoring's only consumer is `DropByScoreStrategy::try_fit`, which Phase B retires). Keeping it as a dead crate creates maintenance debt and confusion. Sunk cost stays sunk; the parity-harness scaffolding learnings carry forward to live-zone work where they actually matter. Folded into Phase B PR-B1. **Alternative:** Keep the crate around as off-path "in case scoring is needed later." Risk: dead-code review burden every PR. --- ## Q4. Stage 3g (lossless-first compression pipeline, issue #315) — re-scope or close? **Context:** Per project memory `~/.claude/projects/-Users-tchopra-claude-projects-headroom/memory/project_lossless_first_pipeline.md`, Stage 3g was queued to formalize "lossless-then-lossy-then-CCR ordering as a `CompressionPipeline` orchestrator + `LosslessTransform`/`LossyTransform` traits." The plan assumed an ICM-style orchestrator over the messages array. **Recommendation:** **Re-scope** issue #315 to "live-zone-only pipeline orchestrator." The traits stay (`LosslessTransform`/`LossyTransform`); the scope changes from "history compactor" to "live-zone block dispatcher." This is what Phase B PR-B2 builds. Update issue #315's body to reflect the realignment. **Alternative:** Close issue #315 and treat Phase B PR-B2 as fulfilling its intent. Risk: history of the decision is lost. --- ## Q5. Headroom Loop / AWS Marketplace BYOC — affected scope? **Context:** Per project memory `project_headroom_loop.md` (enterprise paid product) and `project_headroom_aws_marketplace.md` (BYOC CFN stack in customer VPC). Both depend on the OSS proxy. **Recommendation:** The realignment **strengthens** both: - Headroom Loop's value proposition is "trace stream + enterprise compression policy"; Phase F's auth-mode policy is exactly the surface Loop wants to gate on. - AWS Marketplace BYOC's pitch is "context compression in front of Bedrock"; Phase D's native Bedrock support makes that pitch real (today's LiteLLM-converted Bedrock path was fake; Phase D fixes it). No re-scoping needed; revisit after Phase D lands. **Alternative:** Pause Headroom Loop / Marketplace work until Phase D completes. Recommended if their roadmap conflicts with Phase D timing. --- ## Q6. `make test-parity` per-PR gate — enable now or wait? **Recommendation:** Enable now (Phase I PR-I6) with the existing stubs. `Skipped` is permitted; `Diff` fails the build. As Phase I PR-I5 promotes stubs to real comparators, the per-PR gate gradually tightens. **Alternative:** Wait until all stubs are real. Risk: parity divergence merges silently for the next month. --- ## Q7. Operator config switch — explicit `HEADROOM_PROXY_BACKEND` env var, or implicit? **Context:** During Phase H rollout, operators need a way to choose Python vs Rust proxy. **Recommendation:** Add `HEADROOM_PROXY_BACKEND={python|rust}` env var in Phase H PR-H1; default to `rust` once the canary in Phase I PR-I4 confirms ≥99.9% byte-equality. Keep the Python proxy alive in the codebase for 30 days post-Phase-H as an explicit rollback target. After 30 days of stable Rust operation, run Phase H PR-H2/H3 to delete Python. **Alternative:** Cut over implicitly (`headroom proxy start` always uses Rust after Phase H). Riskier; no clean rollback path. --- ## Q8. Container image strategy — single binary or multi-stage? **Recommendation:** Single binary (`headroom-proxy` Rust). Container is `FROM scratch` or `FROM gcr.io/distroless/static`. Image size drops from ~500 MB (with Python + LiteLLM + ONNX models) to ~50 MB. **Alternative:** Multi-stage Docker with Rust binary + Python sidecar (for evals/learn/memory writers). Recommended only if those subsystems become production-relevant; today they're CLI tools. --- ## Q9. RTK proxy-side invocation — ever revisit? **Resolved — moot.** RTK was removed from Headroom outright (see `09-phase-G-rtk-observability.md`), so there is no proxy-side invocation to revisit. The original recommendation was "no, document the decision in `docs/rtk-architecture.md`" (that doc was deleted with the feature). The argument is kept because reasons 1–3 apply to any future shell-output rewriter: 1. Cache hot zone risk: shell-out + buffer per tool result is correctness-fragile. 2. Parallel implementation: `crates/headroom-core/src/transforms/log_compressor.rs` covers post-hoc log/output compression; RTK rewrites *commands* (different value). 3. RTK itself is a third-party binary the team doesn't control; an upstream version change silently busts cache. If a future requirement emerges (e.g., "Headroom must compress shell output for users who don't run wrap"), reconsider with explicit cache-safety design. **Alternative:** Build proxy-side RTK as a feature-flagged opt-in. Recommended only if the wrap-CLI breadth (PR-G1) doesn't cover enough surface. --- ## Q10. Bedrock/Vertex priority — parallel with proxy port (Phase D in calendar) or after Phase H? **Recommendation:** **Parallel.** Phase D blocks H2 (Python LiteLLM retirement) but not H1 (Python proxy retirement). Run Phase D and Phase C/E/F/G concurrently. **Alternative:** Sequential, Phase D after Phase H. Risk: Bedrock/Vertex users stay on the broken Python LiteLLM path for an extra month. --- ## Q11. Memory subsystem — auto-tail mode default, or tool-only? **Recommendation:** Auto-tail mode default in Phase B PR-B6, with tool-only mode behind a flag. Migrate users to tool-only over the next 6 months once docs and tooling are mature. Auto-tail is byte-deterministic (per the cache-safety invariant) and matches existing UX. **Alternative:** Force tool-only immediately. Risk: breaks customers' existing memory-augmented prompts. --- ## Q12. Parity harness post-Phase-H — keep or delete? **Context:** After Phase H deletes Python, `crates/headroom-parity/` no longer has a Python side to compare against. Per Phase H PR-H3, this is a decision point. **Recommendation:** **Repurpose**, don't delete. Rename to `crates/headroom-version-parity/` and use it to compare current-Rust-version vs previous-Rust-version on the recorded fixtures. Catches Rust-vs-Rust regressions during future ML compressor variants (e.g., when Kompress is ported to Rust via `ort`). **Alternative:** Delete entirely. Save ~2K LOC. Risk: no automated regression test for compressor changes. --- ## Q13. Auth-mode UA detection list — which CLIs to recognize? **Phase F PR-F1 starts with this list:** - `claude-cli/` (Anthropic CLI) - `claude-code/` (Claude Code) - `codex-cli/` (Codex CLI) - `cursor/` (Cursor IDE) - `claude-vscode/` - `github-copilot/` - `anthropic-cli/` - `antigravity/` (Cloudcode Antigravity) **Recommendation:** Extend over time as new CLIs emerge. Alphabetic sort for determinism. Document in `docs/auth-modes.md`. **Alternative:** Start with a smaller list; expand reactively. Risk: subscription users mis-classified as PAYG and fingerprint-leaked. --- ## Q14. The ICM removal blast radius — confirm acceptable **Counts:** - Lines deleted (Python): ~3,300 - Lines deleted (Rust): ~4,500 - Files deleted: ~30 - Tests deleted: ~50 - PRs that recently merged but become wasted work: PR #338, PR #343 (MessageScorer Rust port) - Project memory updates needed: 1 (the "53270 lines" content_router.py figure was wrong by 25× — already corrected in `MEMORY.md`). **Recommendation:** Acceptable. The cache-killer bugs cost more than the deleted code's hypothetical future value. --- ## Q15. Calendar + capacity — sequential or parallel? **Sequential calendar:** ~13 weeks. One contributor working through phases A→I. **Parallel calendar:** ~8 weeks with 2-3 contributors splitting along these natural boundaries: - Lead: Phase A (lockdown), Phase B (live-zone), Phase H (retirement) — the critical path. - Contributor 2: Phase C (Rust proxy paths), Phase D (Bedrock/Vertex). Self-contained. - Contributor 3 (optional): Phase E (cache stabilization), Phase F (auth-mode), Phase G (RTK + obs), Phase I (test infra). Mostly independent. **Recommendation:** Parallel. The bug list is real and the cache hit rate is hemorrhaging in production today. --- ## Quick answer template For decision sign-off, fill in this block: ``` Q1 (Phase A timing): [ ] tonight [ ] wait Q2 (ICM scope): [ ] Tier 1+2 [ ] Tier 1 only [ ] all 3 tiers Q3 (MessageScorer): [ ] delete [ ] keep Q4 (issue #315): [ ] re-scope [ ] close Q5 (Loop/Marketplace):[ ] proceed unchanged [ ] pause until D Q6 (parity gate): [ ] enable now [ ] wait Q7 (operator switch): [ ] env var w/ default rust [ ] implicit cutover Q8 (container): [ ] single binary [ ] multi-stage Q9 (RTK proxy-side): [ ] document never [ ] feature-flag for future Q10 (Bedrock priority):[ ] parallel [ ] sequential after H Q11 (memory mode): [ ] auto-tail default [ ] tool-only force Q12 (parity harness): [ ] repurpose [ ] delete Q13 (UA list): [ ] approve list [ ] revise: ___________ Q14 (ICM blast radius): [ ] accept [ ] reduce scope Q15 (calendar): [ ] parallel (2-3 contributors) [ ] sequential ``` - REALIGNMENT/INDEX.md (5902 bytes)
# Headroom Realignment — Index **Status:** Drafted 2026-05-01 from a 10-agent deep audit against `~/Downloads/llm-proxy-compression-guide.md`. **Owner:** chopratejas **Goal:** Move the entire codebase to Rust, preserve prefix cache, retain compression value, integrate RTK end-to-end, and gate compression policy by auth mode (PAYG / OAuth / subscription). ## Read in this order 1. [00-overview.md](./00-overview.md) — executive summary; the wrong mental model; what changes 2. [01-bug-list.md](./01-bug-list.md) — comprehensive ranked bug list with file:line and guide § 3. [02-architecture.md](./02-architecture.md) — the realigned target architecture 4. Phase docs (PR-by-PR, executable): - [03-phase-A-lockdown.md](./03-phase-A-lockdown.md) — **start here**: stop-the-bleeding (8 PRs, ~1 week) - [04-phase-B-live-zone.md](./04-phase-B-live-zone.md) — live-zone-only compression (7 PRs, ~2 weeks) - [05-phase-C-rust-proxy.md](./05-phase-C-rust-proxy.md) — port handlers to Rust (5 PRs, ~3 weeks) - [06-phase-D-bedrock-vertex.md](./06-phase-D-bedrock-vertex.md) — native envelopes (4 PRs, ~2 weeks) - [07-phase-E-cache-stabilization.md](./07-phase-E-cache-stabilization.md) — Phase 3 stabilization (6 PRs, ~1 week) - [08-phase-F-auth-mode.md](./08-phase-F-auth-mode.md) — auth-mode policy gates (4 PRs, ~1 week) - [09-phase-G-rtk-observability.md](./09-phase-G-rtk-observability.md) — RTK breadth + metrics (3 PRs, ~1 week) - [10-phase-H-python-retirement.md](./10-phase-H-python-retirement.md) — delete Python proxy (3 PRs, ~2 weeks) - [11-phase-I-test-infra.md](./11-phase-I-test-infra.md) — test/CI gates (parallel) 5. [12-decisions-needed.md](./12-decisions-needed.md) — open questions ## Conventions - **Branch name:** `realign-<phase-letter><pr-num>-<slug>`. Example: `realign-A1-icm-passthrough`. - **Worktree path:** `~/claude-projects/headroom-worktrees/realign-<phase><num>-<slug>`. Use `git worktree add` so each PR is an isolated checkout. - **Commit prefix:** `fix:` for Rust-migration phase commits (per project memory — `feat:` would inflate semantic-release version). - **No `Co-Authored-By: Claude` trailer** (per project memory). - **Pre-push gate:** `make ci-precheck` per project memory; never push without it. ## Phase totals | Phase | PRs | LOC delta (est.) | Calendar (sequential) | |---|---:|---:|---:| | A — Lockdown | 8 | -200 / +400 | 1 week | | B — Live-zone engine | 7 | **-10,000 / +1,500** | 2 weeks | | C — Rust proxy paths | 5 | -2,000 / +5,000 | 3 weeks | | D — Bedrock/Vertex native | 4 | -800 / +2,500 | 2 weeks | | E — Cache stabilization | 6 | -100 / +900 | 1 week | | F — Auth-mode policy | 4 | -50 / +600 | 1 week | | G — RTK + observability | 3 | -50 / +400 | 1 week | | H — Python retirement | 3 | **-15,000 / +200** | 2 weeks | | I — Test infra | parallel | +2,000 | continuous | | **Total** | **40** | **~-28,000 / +13,500** | **~13 weeks** sequential, **~8 weeks** parallel | ## Cross-cutting invariants These never get violated by any PR: 1. Bytes that the proxy doesn't intend to modify must arrive at upstream **byte-equal** (SHA-256) to bytes that arrived at the proxy. (§1.9) 2. The cache hot zone — system, tools, old turns, reasoning/thinking/redacted/compaction items — is never modified. (§10) 3. Compression is **append-only**: only the live zone (latest user message, latest tool/function/shell/patch outputs) is ever rewritten. (§6.4 + §10.3) 4. Compression is deterministic: same input bytes → same output bytes. (§7.1) 5. Tool definitions are **normalized** (sorted), never compressed. (§8.5) 6. `signature`, `encrypted_content`, `redacted_thinking.data`, `compaction.encrypted_content` are passthrough-only. (§2.7, §2.8, §4.3, §4.8, §10.1) 7. TOIN never alters request-time decisions; it observes and publishes recommendations between deploys. (§7.1, §11.17) 8. CCR markers and the `ccr_retrieve` tool are present **on every request** for a session that ever did CCR — never toggled. (§6.3 #2) 9. `Authorization` header is forwarded byte-faithfully and never logged or persisted unredacted. 10. Auth mode (PAYG / OAuth / subscription) gates compression policy; subscription mode runs in stealth (no `X-Headroom-*` upstream, no beta drift, no UA mutation, no `accept-encoding` strip). ## Preserved primitives (per user direction) - **TOIN** — refactored to strict observation-only; per-tenant aggregation key. - **CCR** — hardened with persistent backend + always-on tool registration. - **Kompress-base** — stays as plain-text compressor (§8.6); Rust port via `ort` later. - **ContentRouter** — the architecturally correct piece (~2150 LOC); ported to Rust as the live-zone block dispatcher. - **Type-aware compressors** — SmartCrusher, Code, Log, Search, Diff (already in Rust); kept. - **`signals/` Rust trait module** — keeps; drives live-zone consumers. - **`tokenizer/` Rust** — keeps. - **`safety.rs`** — tool-pair atomicity logic; moved to `transforms/safety.rs` after Phase B. ## Retired (~25K LOC) - ICM (Python `intelligent_context.py`, Rust `context/manager.rs`) - `RollingWindow`, `ProgressiveSummarizer`, `scoring.py`, `tool_crusher.py` (Python) - `crates/headroom-core/src/scoring/`, `relevance/`, most of `context/` (Rust) - `crates/headroom-proxy/src/compression/icm.rs` - `headroom/transforms/cache_aligner.py` rewrite path (keep detector + warning) - `headroom/proxy/server.py`, `handlers/anthropic.py`, `handlers/openai.py`, `handlers/streaming.py`, `handlers/gemini.py`, `responses_converter.py`, `memory_handler.py`, `memory_tool_adapter.py`, `semantic_cache.py`, `batch.py` — once Rust hits parity (Phase H) - `headroom/backends/litellm.py` Bedrock/Vertex converter — replaced by native envelopes (Phase D) - MessageScorer Rust port (PR #338, #343) — wasted work; deleted in Phase B
- run-all-plugins.sh (10911 bytes)
#!/usr/bin/env bash # ───────────────────────────────────────────────────────────────────────────── # run-all-plugins.sh — install + configure + run the Headroom proxy with ALL 5 # enterprise plugins, the coding savings-profile, and ML compression offloaded # to the Kompress-v2 Modal endpoint. Then confirm everything loaded. # # Plugins : lossless_guard, skill_search, observability, tier_router, tool_search # Extra : headroom-ai[lean] (torch-free proxy; ML offloaded to Modal) # Profile : coding (HEADROOM_SAVINGS_PROFILE) + cache mode (prefix-cache safe) # # Secrets are SOURCED from ~/env.txt and ~/.headroom/plugins.env — never inlined. # Re-runnable: install is skipped when already satisfied (FORCE_INSTALL=1 forces). # ───────────────────────────────────────────────────────────────────────────── set -euo pipefail HR=/Users/tcms/demo/headroom VENV="$HR/.venv" PORT="${HEADROOM_PORT:-8787}" ENV_TXT="${ENV_TXT:-$HOME/env.txt}" PLUGINS_ENV="$HOME/.headroom/plugins.env" LOG="${HEADROOM_LOG:-$HOME/.headroom/logs/proxy-all-plugins.log}" mkdir -p "$(dirname "$LOG")" # ── 1. venv ────────────────────────────────────────────────────────────────── # `python`/`pip`/`uv` are broken system-wide on this box — always use the venv, # and `python -m pip` (the .venv/bin/pip shim is broken too). # shellcheck disable=SC1091 source "$VENV/bin/activate" PY="$VENV/bin/python" # ── 2. install (guarded) ────────────────────────────────────────────────────── # headroom-ai[lean] pulls proxy,code,relevance,reports,otel,html,mcp,spreadsheet # (all torch-free — heavy ML is offloaded to the Modal Kompress endpoint below). # The 5 plugins install --no-deps so pip won't drag PyPI's headroom-ai over the # local editable one; headroom-license is their shared Ed25519 verifier. need_install=1 if [ "${FORCE_INSTALL:-0}" != "1" ]; then n=$("$PY" -c 'import opentelemetry; from headroom.proxy.extensions import discover; print(len(list(discover())))' 2>/dev/null || echo 0) [ "$n" = "5" ] && need_install=0 fi if [ "$need_install" = "1" ]; then avail=$(df -g "$HR" 2>/dev/null | awk 'NR==2{print $4}') echo "▶ disk: ${avail:-?}Gi free before install" if [ -n "$avail" ] && [ "$avail" -lt 2 ]; then echo "!! <2Gi free — aborting before heavy install (free space, then re-run)"; exit 1 fi echo "▶ installing pip+maturin, then headroom-ai[lean] + license + 5 plugins (editable)…" "$PY" -m pip install -U pip maturin # litellm >=1.92 ships an sdist-only Rust bridge whose AWS-SDK crates need rustc>=1.94.1; # the default rustup toolchain here is older (pip builds litellm in a temp dir that misses # the repo's 1.95 pin), so pin to the last pure-Python wheel line (1.91.4). Satisfies # headroom's litellm>=1.86.2,<2.0 and skips the Rust build entirely. "$PY" -m pip install "litellm<1.92" "$PY" -m pip install -e "${HR}[lean]" "litellm<1.92" "$PY" -m pip install -e /Users/tcms/demo/headroom-license for p in lossless-guard skill-search observability tier-router tool-search; do "$PY" -m pip install -e "/Users/tcms/demo/headroom-${p}" --no-deps done else echo "▶ install satisfied (5 extensions discovered) — skipping (FORCE_INSTALL=1 to force)" fi # ── 3. secrets from ~/env.txt ───────────────────────────────────────────────── # Provides: OPENAI_API_KEY, ANTHROPIC_API_KEY, FIREWORKS_API_KEY (upstream creds); # LANGFUSE_{PUBLIC,SECRET}_KEY + LANGFUSE_BASE_URL (observability sink); # HEADROOM_KOMPRESS_ENDPOINT + _TOKEN (Modal ML offload). [ -f "$ENV_TXT" ] || { echo "!! $ENV_TXT not found"; exit 1; } set -a; # shellcheck disable=SC1090 source "$ENV_TXT"; set +a # ── 4. plugin license (Ed25519, offline, wildcard) ──────────────────────────── # HEADROOM_LICENSE + HEADROOM_LICENSE_PUBKEY. Core and every plugin read the same # HEADROOM_LICENSE; the banner says "LICENSED (usage reporting off)" unless # HEADROOM_USAGE_REPORTING=1 is also set. Licence variables already exported in # the calling shell are kept. if [ -f "$PLUGINS_ENV" ]; then set -a; # shellcheck disable=SC1090 source "$PLUGINS_ENV"; set +a elif [ -z "${HEADROOM_LICENSE:-}" ]; then echo "!! $PLUGINS_ENV not found and HEADROOM_LICENSE is not set: the plugins need a licence" exit 1 fi # ── 5. Kompress ML offload → Modal ──────────────────────────────────────────── # Setting HEADROOM_KOMPRESS_ENDPOINT (+_TOKEN) alone routes Kompress inference to # the Modal endpoint (content_router._get_kompress_remote). No other flag needed; # HEADROOM_COMPRESS_ALLOW_REMOTE is a different thing (remote upstreams, not this). : "${HEADROOM_KOMPRESS_ENDPOINT:?must be set in $ENV_TXT}" export HEADROOM_KOMPRESS_ENDPOINT_TOKEN="${HEADROOM_KOMPRESS_ENDPOINT_TOKEN:-}" # ── 6. observability sink → Langfuse + spend attribution ────────────────────── # HEADROOM_LANGFUSE_ENABLED must be explicitly truthy (LANGFUSE_* creds come from # env.txt). Traces (agent.turn / llm.turn spans with gen_ai.usage.cost) land in # Langfuse. Spend is opt-in: HEADROOM_MODEL_PRICES is {model-substr:{in,out}} in # USD per 1K tokens. (Per-request identity — org/team/user/session — is supplied # by the CLIENT via x-headroom-* headers, not settable here.) export HEADROOM_LANGFUSE_ENABLED=1 export HEADROOM_LANGFUSE_SERVICE_NAME=headroom-proxy export HEADROOM_MODEL_PRICES='{"claude-opus":{"in":0.015,"out":0.075},"claude-sonnet":{"in":0.003,"out":0.015},"gpt-5":{"in":0.00125,"out":0.01},"gpt-4":{"in":0.003,"out":0.012}}' # Metrics (counters) need a separate OTLP endpoint — none in env.txt, so left off: # export HEADROOM_OTEL_METRICS_ENABLED=1 HEADROOM_OTEL_METRICS_ENDPOINT=http://localhost:4318 # ── 7. tier_router ───────────────────────────────────────────────────────────── # Only stamps service_tier on the wire (no token delta). OpenAI 'flex' is only # auto-selected for models declared eligible here. Anthropic tiers are a no-op by # default. Clients force a tier with x-headroom-tier / x-headroom-background: 1. export HEADROOM_TIER_FLEX_MODELS="${HEADROOM_TIER_FLEX_MODELS:-gpt-5,gpt-4.1,o4-mini}" # ── 8. plugin tuning (defaults shown; override as needed) ───────────────────── # skill_search fires on Anthropic w/ >=min skills; tool_search on synthetic-tier # providers w/ >=min tools; lossless_guard lossy tier is opt-in (kept OFF). export HEADROOM_SKILL_SEARCH_MIN_SKILLS="${HEADROOM_SKILL_SEARCH_MIN_SKILLS:-8}" export HEADROOM_TOOL_SEARCH_MIN_TOOLS="${HEADROOM_TOOL_SEARCH_MIN_TOOLS:-5}" # export HEADROOM_LOSSLESS_GUARD_LOSSY=1 # opt-in irreversible Bash-noise drop # ── 9. coding profile + mode ─────────────────────────────────────────────────── # savings_profile=coding tunes the pipeline for coding-agent traffic; cache mode # freezes prior turns to preserve the provider prefix-cache (what coding wants). export HEADROOM_SAVINGS_PROFILE=coding # ── 10. run + confirm ────────────────────────────────────────────────────────── cleanup() { [ -n "${PROXY_PID:-}" ] && kill "$PROXY_PID" 2>/dev/null || true; } trap cleanup INT TERM EXIT echo "▶ starting proxy on :$PORT (profile=coding, mode=cache, all 5 extensions)…" headroom proxy --port "$PORT" --mode cache --proxy-extension '*' > "$LOG" 2>&1 & PROXY_PID=$! # wait for readiness (no foreground sleep on this harness) curl -s --retry 40 --retry-delay 1 --retry-all-errors --max-time 60 \ "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 || true echo echo "══════════════════ CONFIRMATION ══════════════════" echo "── extensions loaded (from $LOG) ──" grep -iE "Extensions:|license accepted|installed \(" "$LOG" | sed 's/^/ /' || true echo "── Modal Kompress endpoint reachable? ──" code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$HEADROOM_KOMPRESS_ENDPOINT" || echo "unreachable") echo " $HEADROOM_KOMPRESS_ENDPOINT -> HTTP $code (any response = up; offload runs on real traffic)" echo "── /stats surfaces (empty until traffic flows) ──" curl -s --max-time 5 "http://127.0.0.1:$PORT/stats" | "$PY" -c ' import sys,json d=json.load(sys.stdin) print(" extension_savings :", d.get("extension_savings")) print(" by_layer :", list(d.get("savings",{}).get("by_layer",{}))) print(" tokens_saved_by_strat:", d.get("tokens_saved_by_strategy")) print(" otel.enabled :", d.get("otel",{}).get("enabled")) print(" langfuse.enabled :", d.get("langfuse",{}).get("enabled")) ' 2>/dev/null || echo " (stats not ready)" cat <<EOF ── where each effect shows up ── lossless_guard -> dashboard (compression layer) + /stats.tokens_saved_by_strategy skill_search -> /stats.extension_savings (NOT dashboard) — Anthropic client, >=8 skills tool_search -> /stats.extension_savings (NOT dashboard) — synthetic-tier client, >=5 tools observability -> Langfuse UI (spans + gen_ai.usage.cost) — send x-headroom-org/user/session tier_router -> service_tier on the wire / provider bill (no token delta) ── drive traffic (two clients — they exercise different plugins) ── Claude Code : ANTHROPIC_BASE_URL=http://localhost:$PORT claude # lossless_guard + skill_search OpenAI/opencode: OPENAI_BASE_URL=http://localhost:$PORT/v1 <client> # tool_search Dashboard : headroom dashboard (http://127.0.0.1:$PORT/dashboard) Raw stats : curl -s localhost:$PORT/stats | python3 -m json.tool Proxy is running (pid $PROXY_PID). Ctrl-C to stop. Logs: $LOG ═══════════════════════════════════════════════════ EOF wait "$PROXY_PID" || true - RUST_DEV.md (26328 bytes)
# Headroom Rust Rewrite — Developer Guide This document covers the Rust port of Headroom. It is the only new top-level doc created in Phase 0; longer-form design/plan writeups live elsewhere and are not versioned in this repo. ## Workspace layout ``` Cargo.toml # workspace root rust-toolchain.toml # pins stable rustc with rustfmt+clippy crates/ headroom-core/ # library: shared types + transform trait surface headroom-proxy/ # binary: axum /healthz (Phase 2 grows this) headroom-py/ # PyO3 cdylib exposing `headroom._core` headroom-parity/ # lib + `parity-run` CLI for Python parity tests headroom-simulators/ # binary: deterministic local upstream stub for proxy tests tests/parity/ fixtures/<transform>/*.json # recorded Python outputs (Phase 1 ports match) recorder.py # Python-side fixture recorder scripts/record_fixtures.py # entry point for running the recorder ``` `cargo build --workspace` builds every crate. `default-members` drops `headroom-py` from `cargo run`/bare-`cargo test` flows so that `cargo test --workspace` does not try to execute the PyO3 cdylib standalone (it can't find `libpython` without a Python interpreter hosting it). ## Common commands `just` is not installed on dev boxes here; a `Makefile` at the repo root exposes the same targets: | Target | What it does | | --- | --- | | `make test` | `cargo test --workspace` | | `make test-parity` | Builds `headroom-py` via maturin, runs `parity-run run` | | `make bench` | `cargo bench --workspace` | | `make build-proxy` | Release-builds `headroom-proxy`, strips, prints size | | `make build-wheel` | `maturin build --release -m crates/headroom-py/Cargo.toml` | | `make fmt` | `cargo fmt --all` | | `make lint` | `cargo fmt --check` + `cargo clippy --workspace -- -D warnings` | ## Running the proxy `headroom-proxy` is a transparent reverse proxy. Phase 1 forwards HTTP/1.1, HTTP/2, SSE, and WebSocket traffic verbatim to a configured upstream — no provider logic yet. The intent is that operators run the existing Python proxy on a private port and put `headroom-proxy` on the public port pointed at it; end users notice nothing. ```bash # Build make build-proxy ./target/release/headroom-proxy --help # Run against a local upstream (listens on 127.0.0.1:8787 by default) ./target/release/headroom-proxy \ --upstream http://127.0.0.1:8788 # Health checks curl -s http://127.0.0.1:8787/healthz # => {"ok":true,...} curl -s http://127.0.0.1:8787/healthz/upstream # => 200 if upstream reachable ``` ### Operator runbook (Phase 1 cutover) ```bash # 1. Move the Python proxy to a private port (e.g. 8788) HEADROOM_HOST=127.0.0.1 HEADROOM_PORT=8788 python -m headroom.proxy & # or your existing launcher # 2. Run the Rust proxy on the previously-public port (8787) pointing at it. # The Python proxy sees every relayed request as coming from 127.0.0.1, # so the Rust proxy enforces HEADROOM_PROXY_TOKEN itself (see below). export HEADROOM_PROXY_TOKEN="$(openssl rand -hex 32)" ./target/release/headroom-proxy --listen 0.0.0.0:8787 --upstream http://127.0.0.1:8788 & # 3. End users keep hitting :8787 unchanged. # 4. Confirm passthrough: curl -si http://127.0.0.1:8787/v1/models # 5. Rollback = stop the Rust proxy and rebind Python back to 8787. ``` ### Configuration flags | Flag | Env var | Default | Notes | | --- | --- | --- | --- | | `--listen` | `HEADROOM_PROXY_LISTEN` | `127.0.0.1:8787` | bind address; a non-loopback address refuses to start without `HEADROOM_PROXY_TOKEN` unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` | | `--proxy-token` | `HEADROOM_PROXY_TOKEN` | (unset) | inbound token required from non-loopback peers (below); prefer the env var | | `--metrics-require-loopback` | `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` | `false` | when set, `/metrics` is served only to loopback peers (403 otherwise); recommended on non-loopback binds | | `--upstream` | `HEADROOM_PROXY_UPSTREAM` | (required) | base URL the proxy forwards to | | `--upstream-timeout` | | `600s` | end-to-end request timeout (long for streams) | | `--upstream-connect-timeout` | | `10s` | TCP/TLS connect timeout | | `--max-body-bytes` | | `100MB` | for buffered cases; streams bypass | | `--log-level` | | `info` | `RUST_LOG`-style filter | | `--rewrite-host` / `--no-rewrite-host` | | rewrite | rewrite Host to upstream (default) | | `--graceful-shutdown-timeout` | | `30s` | wait for in-flight on SIGTERM/SIGINT | | `--stats` | `HEADROOM_PROXY_STATS` | `true` | native savings stats + `/dashboard` (below) | | `--stats-path` | `HEADROOM_PROXY_STATS_PATH` | `~/.headroom/native_stats.json` | ledger persistence (honours `HEADROOM_WORKSPACE_DIR`) | | `--enable-bedrock-native` | `HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` | (unset: off) | `true` or `--features native_bedrock` mounts `/model/{id}/invoke`, `/converse` and the streaming variants; they sign every request with this host's AWS credentials and do not authenticate the caller | | `--enable-vertex-native` | `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE` | `false` | mount Vertex `:rawPredict` / `:streamRawPredict`; replaces the caller's `Authorization` with this host's Google ADC bearer (scope `--vertex-adc-scope`) and does not authenticate the caller | | `--vertex-allow-non-google-upstream` | `HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` | `false` | by default the Vertex route only sends the ADC bearer when `--upstream` is `https://*.googleapis.com`, and answers 502 otherwise | ### Inbound authentication The Rust proxy connects to the Python proxy from `127.0.0.1`, and the Python proxy trusts loopback callers (no token needed, and access to `/admin`, `/debug` and `/settings`). So the Rust proxy applies the same `HEADROOM_PROXY_TOKEN` rules itself, before anything is forwarded: - With a token set, a non-loopback peer must send it as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`, or gets `401 {"error":"unauthorized"}`. Loopback peers are exempt, as in the Python proxy. The token is removed from the request before it is forwarded (`Authorization` only when it carries the token). - A caller that is not local here (a non-loopback peer, or a loopback peer whose `Host` is not a loopback name) is never forwarded with a `Host` that may name loopback (including scoped IPv6 such as `[::1%lo]`), whatever `--rewrite-host` is and whatever the upstream name resolves to: that `Host` is replaced with `headroom-remote.invalid`, so the Python proxy's local-only routes (`/admin`, `/debug`, `/settings`, ...) stay closed to it, with or without a token. - `GET /healthz`, `/healthz/upstream`, `/health`, `/livez` and `/readyz` stay reachable without the token, and `GET`/`HEAD /api/hello` is answered locally with an empty `200`. - A non-loopback `--listen` with no token refuses to start. Set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` only when something else already restricts who can reach the port. ### Savings stats & dashboard (native) The Rust proxy records per-request savings/cost telemetry on every lane it forwards — Anthropic `/v1/messages`, OpenAI Chat/Responses, all four native Bedrock routes, and Vertex `rawPredict` / `streamRawPredict` when those are enabled — and serves it locally (never tunnelled upstream): ```bash curl -s http://127.0.0.1:8787/stats | jq '.session' # since-boot totals curl -s http://127.0.0.1:8787/stats | jq '.lifetime_by_model' curl -s "http://127.0.0.1:8787/stats/timeseries?bucket=day" | jq '.points[-7:]' curl -s "http://127.0.0.1:8787/stats/events?limit=5" # recent-request feed open http://127.0.0.1:8787/dashboard # embedded UI, zero deps ``` Recording is deferred until the response is fully observed, so streaming output/cache token counts come from the real SSE / EventStream usage frames. USD figures come from the vendored LiteLLM price table (`data/model_prices_and_context_window.json`); a model missing from it records $0 and logs one WARN — refresh via `scripts/refresh_model_limits.sh`. Input and output are priced separately (`input_cost_usd` / `output_cost_usd`) because they bill at different rates — output is typically 4-5x input, so a spend figure that counted input alone would understate a short-prompt, long-answer turn by most of its cost. `total_cost_usd` is the sum and is what `/dashboard` shows as "Spend". Cache savings are NET: the cache-read discount minus the cache-write premium (writes bill above list price), floored at $0 per request — a warm-up turn that only writes is not counted as negative savings, and gross-read figures that overstate the benefit are never shown. Failed upstreams count as failures and accrue no savings — as do proxy-side rejections (missing AWS credentials, SigV4/ADC failures, oversized bodies), so an operator-side outage reads as a failure spike rather than as zero traffic. Aggregates (lifetime, per-model, 48 h hourly + ~13 mo daily buckets) persist as one atomic snapshot written off the hot path every 10 s and on shutdown. > **What reads 0 today.** Spend, token counts, and prompt-cache > savings are live. `tokens_saved` / `compression_savings_usd` are > wired end-to-end but will report 0 on real traffic until the > live-zone per-type compressors are implemented — the dispatcher > currently returns `NoCompression` for every well-formed body > (`live_zone_mode_with_valid_body_returns_no_compression_pr_b2` > pins that invariant). Nothing needs changing here when they land; > the numbers start flowing on their own. > **No auth; exposure is split by tier.** These endpoints bind > wherever `--listen` points (default `127.0.0.1:8787`) and, for > loopback peers or with no `HEADROOM_PROXY_TOKEN`, carry no auth, so the payload is split the same way the Python proxy splits > its own `/stats`: > > - **Aggregates** — spend, tokens, per-model/provider rollups, > history — are served to any caller that gets past the inbound gate > (above), like the Prometheus `/metrics` endpoint next door: with > `HEADROOM_PROXY_TOKEN` set, remote callers need the token for these > too; without it, they are open to anyone who can reach the port. > - **Per-request rows** — `recent_requests`, all of `/stats/events` > — plus the ledger's filesystem path are served **only to local > callers**: loopback peer *and* a loopback `Host` header (the > second is the DNS-rebinding defence — a rebound request reaches > the proxy *from* loopback, so the peer check alone doesn't hold). > Remote callers get `recent_requests: null` and a 404 from > `/stats/events`; the dashboard says so rather than erroring. > > A request id + model + timestamp is a different sensitivity tier > from a counter, and `/metrics` never exposes it — so "same as > /metrics" isn't a licence to hand it to the open internet. Opening > the local-only tier to a trusted reverse proxy is tracked in > [#1959](https://github.com/headroomlabs-ai/headroom/issues/1959), > which is settling that policy for the Python dashboard first; this > surface should adopt it rather than grow a second scheme. > `--stats=false` removes the routes entirely. ### Picking the next port: invocation telemetry Before porting another Python compressor to Rust, check what's actually running. The Python proxy already exposes per-transform telemetry on `/stats` (`headroom.proxy.prometheus_metrics`): ```bash # Top compressors by invocation count (last process lifetime) curl -s http://127.0.0.1:8788/stats | jq '.compressions_by_strategy' # { # "intelligent_context": 12453, # "smart_crusher": 487, # "search": 312, # "diff": 28, # "code": 0, # ← never fires; safe to defer porting # ... # } # Per-transform timing (avg/max/count by transform name) curl -s http://127.0.0.1:8788/stats | jq '.pipeline_timing' # Token savings attributable to each strategy curl -s http://127.0.0.1:8788/stats | jq '.tokens_saved_by_strategy' ``` This is the data the audit-cleanup PR (2026-04-30) recommended for prioritizing the next Python → Rust port. Strategies with zero or near-zero invocations are deferral candidates; strategies on the hot path are porting candidates regardless of LOC count. ### Reserved paths `/healthz` and `/healthz/upstream` are intercepted by the Rust proxy and **not** forwarded. Operators must not name a real upstream route either of these. Everything else is a catch-all forward. ## Maturin + Python wiring `headroom-py` is a PyO3 cdylib that exposes `headroom._core` in Python. The `extension-module` feature is opt-in so plain `cargo build --workspace` does not try to link against `libpython` on systems that don't have it. ### First-time setup (clean venv recommended) ```bash python3.11 -m venv /tmp/hr-rust-venv source /tmp/hr-rust-venv/bin/activate pip install maturin cd crates/headroom-py maturin develop # editable dev build, installs headroom._core cd /tmp # IMPORTANT: step out of the repo root first python -c "from headroom._core import hello; print(hello())" # => headroom-core ``` > Why `cd /tmp`? The repo root also contains the Python `headroom/` package. > Running the smoke import from the repo root makes Python resolve `headroom` > to `./headroom/__init__.py` (the full SDK, which pulls in heavy deps) instead > of the lightweight namespace package installed by maturin. Tests should > either run outside the repo root, or ensure `headroom` is installed into > the same venv (then the maturin-installed `_core.so` lands alongside it and > both imports resolve). ### Release wheels ```bash make build-wheel # wheels land under target/wheels/ ``` CI (`.github/workflows/rust.yml`) builds linux-x86_64, macos-arm64, and macos-x86_64 wheels via `PyO3/maturin-action` and uploads them as artifacts. ## Parity harness `crates/headroom-parity` owns the Rust-vs-Python oracle: - JSON fixtures under `tests/parity/fixtures/<transform>/` (schema: `{ transform, input, config, output, recorded_at, input_sha256 }`). - `TransformComparator` trait — one impl per transform. Phase 0 stubs return `Err(...)`; the harness flags those as `Skipped`, not panics. - `parity-run` CLI: `cargo run -p headroom-parity -- run [--only TRANSFORM]`. - Unit tests in `crates/headroom-parity/src/lib.rs` include a **negative test** (`harness_reports_diff_for_divergent_comparator`) proving the harness detects mismatched output before any real port lands. ### Recording fresh fixtures ```bash source .venv/bin/activate # the main Python SDK venv python scripts/record_fixtures.py # uses tests/parity/recorder.py ls tests/parity/fixtures/*/ | sort | uniq -c ``` The recorder monkey-patches the in-process transform classes (see `record_all()` in `tests/parity/recorder.py`). It does **not** modify any file under `headroom/`. ## Known regressions in retired-Python components Th - rust-toolchain.toml (748 bytes)
[toolchain] # Pin to a specific stable version so CI and local use the EXACT same # compiler. With `channel = "stable"`, CI runners (dtolnay/rust- # toolchain@stable) always grab the latest stable while local dev # machines run whatever version was last installed — a clippy lint # added in a newer stable then breaks CI without firing locally # (this happened on 2026-04-27: `unnecessary_sort_by` landed in # clippy 1.95 while a dev box had 1.92). Pinning makes "passes # locally" = "passes in CI" for every check that depends on the # toolchain. # # Bump procedure: edit the channel string here, run `rustup update`, # run `make ci-precheck`, fix any new lints, commit. channel = "1.95.0" components = ["rustfmt", "clippy"] profile = "minimal"
- SECURITY.md (10421 bytes)
# Security Policy ## Supported Versions | Version | Supported | | ------- | ------------------ | | Latest release on PyPI | :white_check_mark: | | Any earlier release | :x: | Headroom releases frequently and only the most recent release is supported. Security fixes ship in a new release rather than being backported to earlier ones, so upgrading is the remediation path. Check what you are running with `headroom --version`, and compare it against [the current release](https://pypi.org/project/headroom-ai/). ## Reporting a Vulnerability We take security vulnerabilities seriously. If you discover a security issue, please report it responsibly. ### How to Report **Please DO NOT open a public GitHub issue for security vulnerabilities.** Instead, please email us at: **[email protected]** Include the following information: - Type of vulnerability (e.g., injection, data exposure, authentication bypass) - Full path of the affected source file(s) - Step-by-step instructions to reproduce the issue - Proof-of-concept or exploit code (if possible) - Impact assessment ### What to Expect 1. **Acknowledgment**: We will acknowledge receipt within 48 hours 2. **Assessment**: We will assess the vulnerability and determine its severity 3. **Updates**: We will keep you informed of our progress 4. **Resolution**: We aim to resolve critical issues within 7 days 5. **Credit**: With your permission, we will credit you in the security advisory ### Security Best Practices for Users When using Headroom: 1. **API Keys**: Never commit API keys. Use environment variables. 2. **Proxy Exposure**: Don't expose the proxy server to the public internet without authentication 3. **Log Files**: Headroom always writes an operational log to `~/.headroom/logs/proxy.log`. The opt-in `--log-file` request log, and especially `--log-messages`, additionally write request and response *content* to disk. Be aware that both may contain sensitive information. 4. **Budget Limits**: Set budget limits to prevent unexpected costs 5. **Workspace directory**: `~/.headroom` holds both credential material and cached request content. Treat it as sensitive — especially on shared or multi-user hosts. Note that it is not the only location: with memory enabled, extracted memories default to a project-local `.headroom/` directory beside the code you ran the agent in. See [Security Model](https://headroom-docs.vercel.app/docs/security-model) for exactly what is written where. ### Scope The following are in scope for security reports: - Headroom Python package (`pip install headroom-ai`) - Headroom proxy server - Official integrations (LangChain, Agno, Strands, LiteLLM, Vercel AI SDK, Anthropic/OpenAI SDK wrappers, MCP) The following are out of scope: - Third-party integrations not maintained by us - Issues in dependencies (report these to the upstream project) - Social engineering attacks ## Security Posture Headroom is a local proxy that sits between your agents and your LLM providers, so it necessarily handles both your traffic and your credentials. The [Security Model](https://headroom-docs.vercel.app/docs/security-model) documents this in full — what is written to disk, with what permissions and retention, what leaves the host, and what the local admin surface does and does not allow. The summary: - **Interactive provider API keys are forwarded.** Keys supplied via environment variables or request headers are used to authenticate the upstream call and are not written to disk or emitted to logs. Settings can also persist keys: the `ANTHROPIC_TARGET_API_HEADERS` / `OPENAI_TARGET_API_HEADERS` header maps, if you set them through the dashboard settings GUI, are persisted to `~/.headroom/settings.json` in plaintext — and a header map is where a gateway key typically goes. - **Some credentials *are* stored, by design.** `headroom copilot-auth login` persists a GitHub Copilot OAuth refresh token to `~/.headroom/copilot_auth.json` (relocatable via `$HEADROOM_COPILOT_AUTH_FILE`). It is plaintext JSON, written with `0600` permissions on a best-effort basis. - **Persistent installs store supplied environment values.** Every value passed to `headroom install --env KEY=VALUE` is stored verbatim and in cleartext. Supervisors (launchd, systemd, Task Scheduler, cron) start the proxy from a bare environment, so `--env` is the only channel by which a provider API key reaches a supervised proxy — and if you pass one, it is written to `~/.headroom/deploy/<profile>/manifest.json` (mode `0600`) and `export`ed in cleartext by the generated runner scripts `run-headroom.sh` / `ensure-headroom.sh` (mode `0700`) in the same directory (mode `0700`). The installer sets those modes and then reads them back: where the platform enforces POSIX permission bits it refuses to leave behind a runner script it could not restrict, and warns if the profile directory could not be narrowed. On Windows the bits are advisory — access is governed by ACLs — so the modes above are requested but not enforced there, and protection comes from the account's own profile directory instead. Where they do hold, the files are owner-only, not encrypted: anything that can read the account — the user themself, root, a home-directory backup or sync — can read the key, and rotating it requires re-running `headroom install apply`. With the Docker runtime the same values are additionally passed as `docker run --env NAME=VALUE`, which places them in the container process's command line and so in `ps` output for other users on the host. Prefer a supervisor-native secret facility (systemd `LoadCredential=` / `EnvironmentFile=` on an owner-only file, a macOS keychain helper) over `--env` for long-lived provider keys. - **Cached request content is written to disk.** CCR stores pre-compression originals — tool outputs, file contents, retrieved chunks — in a local SQLite database (`~/.headroom/ccr_store.db`, `0600`) so they can be retrieved after compression. Entries are plaintext and are not encrypted at rest. Set `HEADROOM_CCR_BACKEND=memory` if your deployment cannot accept disk persistence. - **Anonymous session summaries are uploaded by default.** The `HEADROOM_BEACON` telemetry beacon is opt-*out*: it POSTs content-free session counters and distributions, a random per-install UUID, version, OS, and architecture to Headroom Labs, plus the values of 15 allow-listed `HEADROOM_*` settings and a bucketed structural descriptor of tool outputs (field count, depth, shape flags). `headroom telemetry --show` prints the exact payload. No prompts, completions, file contents, or paths are included. Disable with `HEADROOM_BEACON=off`, `DO_NOT_TRACK=1`, or `HEADROOM_OFFLINE=1`. This is separate from `HEADROOM_TELEMETRY`, which is opt-in and stays on the machine. - **Passthrough mode**: Sensitive content passes through unchanged by default. - **Operational logs are written unconditionally.** `~/.headroom/logs/proxy.log` (10 MB × 5 rotations) is always on and also carries the admin audit stream. The request log (`--log-file`) and full message logging (`--log-messages`) are opt-in and write request/response content to a path you choose. - **Local control surfaces are loopback-gated.** The `/admin/*`, `/debug/*`, `/v1/retrieve*`, `/v1/telemetry*`, `/v1/toin/*`, and `/settings*` routes require a loopback client address *and* a loopback `Host:` header, plus a same-origin check on the mutating ones. This is a local-trust model: any process on the same host is trusted. Setting `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` deliberately widens the settings and stats surface beyond loopback. Setting `HEADROOM_PROXY_TOKEN` also shows every token holder the per-request `/stats` metadata of all callers, but opens none of these routes. Thank you for helping keep Headroom and its users safe! ## Unpatched optional dependency advisories (reviewed 2026-09-10) The following public upstream advisories remain unresolved. They are included in `uv.lock` through optional extras; the presence of a package in that universal lockfile does not mean it is installed with every Headroom installation. ### CrewAI / ChromaDB (no longer locked) Headroom no longer provides a `crewai` extra, so CrewAI and ChromaDB are not in `uv.lock` and are never installed by Headroom. The CrewAI integration uses whichever CrewAI the user installs (`pip install headroom-ai crewai`), and that installation owns its dependency exposure. Every current CrewAI release requires ChromaDB, which has unpatched advisories in its server ([GHSA-f4j7-r4q5-qw2c](https://github.com/advisories/GHSA-f4j7-r4q5-qw2c), [GHSA-36p7-vc44-83pf](https://github.com/advisories/GHSA-36p7-vc44-83pf), [GHSA-2wm9-hf6c-p5cr](https://github.com/advisories/GHSA-2wm9-hf6c-p5cr), [GHSA-xph7-9rjv-w5fr](https://github.com/advisories/GHSA-xph7-9rjv-w5fr)). Headroom's integration only wraps tools and never starts a ChromaDB server. If you run ChromaDB as a server, keep it inaccessible to untrusted clients and follow those advisories. ### Voice training / Accelerate The `voice-train` extra includes Accelerate (locked at 1.12.0). [GHSA-4j2p-28q2-5m79](https://github.com/advisories/GHSA-4j2p-28q2-5m79) describes path traversal and denial of service through unvalidated `weight_map` entries in sharded checkpoint indexes. Use only trusted checkpoints, including their index files and referenced shards, in training environments. The advisory currently lists versions through 1.14.0, but an upgrade to 1.15.0 is not a verified fix: its [checkpoint loader](https://github.com/huggingface/accelerate/blob/v1.15.0/src/accelerate/utils/modeling.py#L1941-L1944) still joins index values to the checkpoint directory without containment or file-type validation. The proposed fixes [#4070](https://github.com/huggingface/accelerate/pull/4070) and [#4138](https://github.com/huggingface/accelerate/pull/4138) were closed without merging; the latter also explicitly leaves the named-pipe denial of service unfixed. Keep the alert open until a released fix covers both cases. Dependabot ignores only the reviewed unpatched Accelerate range (through 1.15.0). Later releases remain eligible for review. These update exceptions do not remediate the advisories or dismiss vulnerability alerts.
- server.json (1162 bytes)
{ "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", "name": "io.github.headroomlabs-ai/headroom", "description": "Headroom MCP server for compression, retrieval, and stats in MCP-compatible hosts.", "title": "Headroom", "websiteUrl": "https://headroomlabs-ai.github.io/headroom/", "repository": { "url": "https://github.com/headroomlabs-ai/headroom", "source": "github", "id": "1129940957" }, "version": "0.44.0", "packages": [ { "registryType": "pypi", "registryBaseUrl": "https://pypi.org", "identifier": "headroom-ai", "version": "0.44.0", "runtimeHint": "uvx", "runtimeArguments": [ { "type": "named", "name": "--from", "value": "headroom-ai[mcp]" } ], "transport": { "type": "stdio" }, "packageArguments": [ { "type": "positional", "value": "headroom" }, { "type": "positional", "value": "mcp" }, { "type": "positional", "value": "serve" } ] } ] } - SKILL.md (3421 bytes)
--- name: headroom description: Use when a Python or TypeScript app sends large tool outputs, logs, JSON, search results or long chat history to an LLM and the task is to cut input tokens, reduce token cost, fit a context window, or trim tool results without losing errors. Adds the headroom-ai compress() call; in-process, no API key, no model download. --- # Headroom: compress LLM context before the call ## When this applies The user asks to reduce LLM token cost, trim or compress tool outputs or tool results, prune or summarize chat history, fit a prompt into a context window, or stop large logs or JSON from blowing up the prompt. Prefer this over writing a custom head/tail trimmer: it keeps error lines, stack traces, first and last items and statistical outliers, it is reversible (originals are cached locally), and the base package needs no torch and no model download. ## Python (in-process, no network) ```bash pip install headroom-ai # add to requirements.txt / pyproject ``` ```python from headroom import compress result = compress(messages, model=MODEL_NAME) # OpenAI or Anthropic message dicts # call the provider with result.messages instead of messages # log result.tokens_before, result.tokens_after, result.tokens_saved ``` Compression applies to `tool` (and assistant) messages. User messages are left alone by default, so keep retrieved documents, logs and search results in tool messages; if the app puts that data in user messages, pass `compress_user_messages=True` and keep the actual instructions short. Verify without a provider key: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` Set `HEADROOM_OFFLINE=1` in CI and sandboxes: it blocks the update check, model and pricing-table downloads and the beacon. `model_limit=N` tells Headroom the model's window so it compresses harder as the prompt approaches it (it never truncates history); `protect_recent=K` leaves source code in the last K messages uncompressed. ## TypeScript `npm install headroom-ai` gives `compress()` and `withHeadroom()` wrappers for the OpenAI, Anthropic, Gemini and Vercel AI SDK clients, but compression runs in the Python proxy: start it first with `pip install "headroom-ai[proxy]"` and `headroom proxy --port 8787`. If a Python sidecar is not acceptable, say so and stop. ## Coding agents To cut tokens for Claude Code, Codex, Cursor or Copilot themselves (not an app): `uv tool install --python 3.13 "headroom-ai[all]"` then `headroom wrap claude`. ## Do not - Do not claim a savings percentage; print `tokens_before` and `tokens_after`. - Do not use `headroom.integrations.asgi.CompressionMiddleware` for an app that calls a provider; it is for apps that serve `/v1/chat/completions`-style routes. - Do not install `[all]` or `[ml]` unless asked; the base package is enough. Docs index: https://docs.headroomlabs.ai/llms.txt · FastAPI example: https://docs.headroomlabs.ai/docs/fastapi · Repo: https://github.com/headroomlabs-ai/headroom - skills/headroom/SKILL.md (3421 bytes)
--- name: headroom description: Use when a Python or TypeScript app sends large tool outputs, logs, JSON, search results or long chat history to an LLM and the task is to cut input tokens, reduce token cost, fit a context window, or trim tool results without losing errors. Adds the headroom-ai compress() call; in-process, no API key, no model download. --- # Headroom: compress LLM context before the call ## When this applies The user asks to reduce LLM token cost, trim or compress tool outputs or tool results, prune or summarize chat history, fit a prompt into a context window, or stop large logs or JSON from blowing up the prompt. Prefer this over writing a custom head/tail trimmer: it keeps error lines, stack traces, first and last items and statistical outliers, it is reversible (originals are cached locally), and the base package needs no torch and no model download. ## Python (in-process, no network) ```bash pip install headroom-ai # add to requirements.txt / pyproject ``` ```python from headroom import compress result = compress(messages, model=MODEL_NAME) # OpenAI or Anthropic message dicts # call the provider with result.messages instead of messages # log result.tokens_before, result.tokens_after, result.tokens_saved ``` Compression applies to `tool` (and assistant) messages. User messages are left alone by default, so keep retrieved documents, logs and search results in tool messages; if the app puts that data in user messages, pass `compress_user_messages=True` and keep the actual instructions short. Verify without a provider key: ```bash HEADROOM_OFFLINE=1 python -c "from headroom import compress; log='\n'.join(f'INFO step {i} ok' for i in range(3000))+'\nFATAL disk full'; r=compress([{'role':'user','content':'What failed?'},{'role':'assistant','content':None,'tool_calls':[{'id':'c1','type':'function','function':{'name':'logs','arguments':'{}'}}]},{'role':'tool','tool_call_id':'c1','content':log}], model='gpt-4o'); print(r.tokens_before, '->', r.tokens_after, 'FATAL kept:', 'FATAL' in str(r.messages))" ``` Set `HEADROOM_OFFLINE=1` in CI and sandboxes: it blocks the update check, model and pricing-table downloads and the beacon. `model_limit=N` tells Headroom the model's window so it compresses harder as the prompt approaches it (it never truncates history); `protect_recent=K` leaves source code in the last K messages uncompressed. ## TypeScript `npm install headroom-ai` gives `compress()` and `withHeadroom()` wrappers for the OpenAI, Anthropic, Gemini and Vercel AI SDK clients, but compression runs in the Python proxy: start it first with `pip install "headroom-ai[proxy]"` and `headroom proxy --port 8787`. If a Python sidecar is not acceptable, say so and stop. ## Coding agents To cut tokens for Claude Code, Codex, Cursor or Copilot themselves (not an app): `uv tool install --python 3.13 "headroom-ai[all]"` then `headroom wrap claude`. ## Do not - Do not claim a savings percentage; print `tokens_before` and `tokens_after`. - Do not use `headroom.integrations.asgi.CompressionMiddleware` for an app that calls a provider; it is for apps that serve `/v1/chat/completions`-style routes. - Do not install `[all]` or `[ml]` unless asked; the base package is enough. Docs index: https://docs.headroomlabs.ai/llms.txt · FastAPI example: https://docs.headroomlabs.ai/docs/fastapi · Repo: https://github.com/headroomlabs-ai/headroom - TESTING-copilot-subscription.md (8661 bytes)
# Testing: GitHub Copilot subscription mode (`headroom wrap copilot --subscription`) This feature has live coverage on macOS and Windows. Additional Linux secret-store coverage is still useful (see [Status](#status)). If you have a GitHub Copilot subscription and 10 minutes, please run one of the flows below and [file a report](https://github.com/headroomlabs-ai/headroom/issues/new?template=copilot-subscription-test-report.md). > ⚠️ This reads your Copilot login token and routes your > Copilot CLI traffic through a local Headroom proxy. Only run it if you're > comfortable with that. The branch is open for inspection. ## What it does (and what "subscription" means here) Normally `headroom wrap copilot` is **BYOK** — you bring an Anthropic/OpenAI API key and pay that vendor. `--subscription` is different: it lets you use the **Copilot seat you already pay GitHub for**, with **no separate API key**, while still routing through Headroom so your context gets compressed. Mechanically: the Copilot CLI's only interposition hook is its provider-override (the "BYOK transport"), so Headroom uses that knob but supplies **your subscription token** and points back at **GitHub's own Copilot API**. So the CLI may print "BYOK" and require an explicit `--model`, but you are **not** paying a third party — it's your subscription, just compressed. (Proof it's working: the proxy forwards to GitHub's Copilot API — `https://api.githubcopilot.com` by default — with your token.) ## API host & Enterprise / data-residency Which host a wrapped session talks to depends on the lane: - **`--subscription`, `headroom wrap vscode`, and OpenCode `--copilot-subscription`** resolve the host from GitHub's token exchange. Business and Enterprise seats route to the plan host GitHub advertises (`api.business.githubcopilot.com` / `api.enterprise.githubcopilot.com`), which is what GitHub's own clients use and the only host an enterprise firewall on GitHub's subscription-based network routing allows. If that host rejects a model the generic host serves, set `GITHUB_COPILOT_USE_ADVERTISED_HOST=0` to fold back to the generic host. Individual seats route to the generic host: `api.individual.githubcopilot.com` does **not** serve newer models on the responses API, and using it regressed `headroom wrap copilot` after 0.22.4 ([#610](https://github.com/headroomlabs-ai/headroom/issues/610)). - **The implicit OAuth path** (plain `headroom wrap copilot` with a GitHub login and no provider key) performs no token exchange at launch and always uses the **generic public host** `https://api.githubcopilot.com`. On a firewall that allows only the plan host, use `--subscription` or pin the host as below. An explicit `GITHUB_COPILOT_API_URL` wins over all of the above. **Enterprise / data-residency:** if your organization is provisioned on a dedicated Copilot API host (GitHub Enterprise Cloud with data residency, or an egress proxy), pin it explicitly — the override flows through both `--subscription` and OAuth, and onward through the proxy to the upstream request: ```bash export GITHUB_COPILOT_API_URL=https://api.<your-host>.githubcopilot.com headroom wrap copilot --subscription -- --model gpt-5.4 ``` The Business and Enterprise plan hosts are detected from GitHub's token-exchange endpoint (see above); only data-residency and egress-proxy hosts still need the pin. ## Status | Platform | Mechanism (compress + forward) | Token **auto-discovery** from the OS secret store | |----------|:---:|:---:| | macOS (Keychain) | ✅ verified | ✅ verified (`copilot-cli`) | | Linux (`secret-tool`/libsecret) | ✅ expected | ❓ **needs testing** | | Windows (Headroom device auth) | ✅ verified | ✅ verified | | Windows (Copilot CLI credential reuse) | ✅ verified after auth | ❌ Copilot CLI 1.0.81 does not expose the legacy Credential Manager schema | | Any OS via `GITHUB_COPILOT_TOKEN` env var | ✅ verified by tests | n/a (bypasses discovery) | The two things we want to learn: 1. **Does it work end to end on your OS?** 2. **Does it find your Copilot token automatically**, or do you have to set `GITHUB_COPILOT_TOKEN`? If it can't find it, we need the **storage schema** (see each flow) so we can fix auto-discovery. ## Prerequisites (all platforms) 1. A **GitHub Copilot subscription**. 2. The **GitHub Copilot CLI**: `npm install -g @github/copilot` 3. **Log in once**: run `copilot`, complete the device-code login in your browser, then type `/exit`. --- ## Linux — the flow we most need (tests auto-discovery) Auto-discovery only works with a **host-native** install (a container can't read your host secret store). Linux has prebuilt wheels, so: ```bash pipx install --pip-args='--pre' headroom-ai # or: pip install --pre headroom-ai # (no separate API key needed — that's the point) headroom wrap copilot --subscription -- --model gpt-4o -p "Reply with exactly: HEADROOM_OK" ``` - **If it prints `HEADROOM_OK`** → auto-discovery works on your Linux. 🎉 Report success. - **If it errors with "no reusable bearer token"** → discovery missed your token. Please grab the **schema** so we can fix it (redact the secret), then confirm the mechanism works via the env var: ```bash secret-tool search --all 2>/dev/null | sed -E 's/^secret = .*/secret = <redacted>/' # then retry, supplying the token explicitly: GITHUB_COPILOT_TOKEN='<your-token>' headroom wrap copilot --subscription -- --model gpt-4o -p "Reply with: HEADROOM_OK" ``` Report the `attribute.*` lines from `secret-tool` and whether the env-var retry worked. --- ## Windows For a source checkout with Python and Rust installed, build the current tree with the proxy extra and authorize Headroom's dedicated OAuth app: ```powershell uv sync --extra proxy --extra dev uv run --no-sync headroom copilot-auth login uv run --no-sync python e2e/copilot_live.py --vscode-extension ` --model gpt-5-mini --model gpt-5.5 ` --model gpt-5.6-luna --model gpt-5.6-sol --model gpt-5.6-terra ``` The live suite uses the official Copilot CLI, exercises subscription wrapping, sends requests through an isolated VS Code proxy configuration, and optionally drives the installed VS Code extension through `code chat`. It snapshots and restores real VS Code settings byte-for-byte, deliberately occupies the requested port to verify fallback-port propagation, and checks every selected model in both the Copilot response and Headroom's traffic accounting. The Docker-native wrap suite additionally captures an A-to-B-to-A model sequence at its mock upstream, proving the outbound request bodies change without stale model state. Neither suite reads or prints token values. Packaged-install alternatives: **A. Mechanism test (easiest — Docker Desktop or WSL2):** ```powershell $env:HEADROOM_DOCKER_IMAGE = "ghcr.io/headroomlabs-ai/headroom:<branch-tag>" # ask the maintainer for the tag # run the Docker-native installer (scripts/install.ps1), then: $env:GITHUB_COPILOT_TOKEN = "<your-token>" headroom wrap copilot --subscription -- --model gpt-4o -p "Reply with: HEADROOM_OK" ``` Report whether it prints `HEADROOM_OK`. **B. Native auto-discovery schema:** after `copilot` login, check whether the installed Copilot CLI exposes a reusable Windows credential target: ```cmd cmd /c "cmdkey /list" ``` Report only a Copilot-related `Target:` line (it shows the target name, not the secret). Copilot CLI 1.0.81 did not expose such a target in live Windows testing, so use `headroom copilot-auth login` when native reuse is unavailable. > A native Windows wheel is still tracked separately; source builds can run the > full Windows authentication and routing matrix today. --- ## macOS (already proven — a second data point still helps) ```bash pipx install --pip-args='--pre' headroom-ai headroom wrap copilot --subscription -- --model gpt-4o -p "Reply with exactly: HEADROOM_OK" ``` Schema, for reference: Keychain generic password, service `copilot-cli` (`security find-generic-password -s copilot-cli -w`). --- ## What to report Please open a [Copilot subscription test report](https://github.com/headroomlabs-ai/headroom/issues/new?template=copilot-subscription-test-report.md) with: - **OS + version** and **how you installed** (pipx/pip wheel, Docker, source). - Was plain `copilot` logged in? - Did `wrap copilot --subscription` print **`HEADROOM_OK`**? Paste any error. - Did it work **without** setting `GITHUB_COPILOT_TOKEN` (auto-discovery), or only **with** it? - The **storage schema** if discovery failed (`secret-tool search --all` / `cmdkey /list`), with the secret redacted.