How it works
A gateway that stays out of the way — and writes down everything it does.
TokenTriage sits between your app and your model providers. It costs and logs every request off the hot path, and — only when you opt in — routes each one to the cheapest tier that can still do the job.
The request flow
One hop in. Four things happen. One hop out.
Your app
OpenAI / Anthropic SDK
TokenTriage gateway
- 1 Classify the request
- 2 Cost it (token types priced)
- 3 Decide the route (ledger / shadow / route)
- 4 Write one decision-log row
On any error → verbatim passthrough
Providers
OpenAI · Anthropic · Bedrock · Gemini · …
The classify / cost / decide / log work happens beside the request, not in front of it — so the ledger can never become a new way for your agents to break.
The decision ledger
One row per request. One truth.
Every proxied request appends one JSONL row to the decision log. Prompt text is never stored by default — only a hash and counted features. The dollar is hand-checkable against the embedded price table: multiply the token counts by the stamped snapshot and you get the same number the dashboard shows.
- Everything derived is rebuildable from the log — and disposable.
- Two surfaces may never disagree, because both read the same rows.
would_usdrecords the counterfactual, so shadow savings are auditable.
An illustrative decision-log row
{
"trace": { "id": "tr_9f2c…", "agent": "support-bot" },
"request": { "model": "gpt-5", "prompt_sha256": "b1e4…", "step_class": "reply" },
"decision":{ "mode": "route", "tier": "cheap", "upstream": "groq" },
"usage": { "input": 88, "output": 40, "cache": 32, "source": "provider" },
"cost": { "usd": 0.000514, "pricing_key": "openai/gpt-5",
"pricing_snapshot": "litellm@2026-07-14#179affb",
"estimated": false, "would_usd": 0.001470 }
}The honesty invariants
The rules that make every number auditable
These aren’t taglines — they’re constraints the system holds itself to. They’re why the dashboard can be trusted in a diligence review.
Off the request hot path
No dashboard or analytics work sits on, blocks, or backpressures the proxy pipeline. A dead dashboard is invisible to your traffic.
Fail-open, byte-faithful
Any internal error fails open to a verbatim passthrough. The proxied bytes equal going direct — streaming included. Nothing here adds a new way to break your agents.
Provenance on every dollar
Any number that is produced or transformed keeps its pricing snapshot, estimated flag, and source lineage. Estimates are labeled; UNPRICED is excluded, never shown as $0.00.
No heuristic identity stitching
Attribution comes only from explicit identity on the wire. A wrong ledger is worse than a narrow one, so nothing is guessed.
No stored secrets
Secret values are never persisted or displayed — only references (env-var names, secret-manager refs) and presence checks.
The honest-verdict rule
Served-model labels can only place routing on the frontier; only counterfactual shadow duplication can show a beat. Insufficient labels produce a refusal, never a number.
Earned enforcement
Promotion is gated on evidence
You don’t flip routing on and hope. A route is promoted only when the shadow evidence clears the bar — and the dashboard shows the gate, the labels, and the verdict behind the decision.
See it on your own traffic
Start in ledger mode — pure observation, zero request mutation. Climb to shadow, then route, only when the evidence says it’s safe.