My LLM cost was 3x wrong for two months. Audit your own dashboard.

The bug, the cost, the lesson — and a checklist for anyone running LLM-powered SaaS.

Jun 3, 2026

The trigger

I run a SaaS where every user interaction triggers LLM calls. The product is aicofounders.co — 6 AI cofounders that produce strategic output for solo founders. Token spend is the largest variable cost in the business.

Two months ago I built a per-user cost dashboard for the admin panel. Looked clean. Showed me $4.50/week in LLM cost. Felt expensive for 2 active users.

Made me nervous. Started slowing beta invites because the math felt fragile. Considered raising the Starter tier price from $29 to $39 to cover real cost. Deprioritized some token-heavy features.

Last week I looked at the dashboard again and noticed something. OpenRouter — my LLM provider router — returns response.usage.cost on every API call. The actual cost in USD, calculated by them, returned in the response.

I’d been ignoring it. My dashboard was using a static pricing table I’d built months ago, calculating cost from token counts. I wired up the real cost reading.

The corrected number for that same week: $1.50.

Three times less than I thought.

What went wrong

The static pricing table had this structure:

const PRICING = [
  { prefix: "anthropic/claude-haiku-4-5", input: 1, output: 5 },
  { prefix: "anthropic/", input: 3, output: 15 }, // catch-all
];

For two months, my product had been silently using anthropic/claude-haiku-4-5 (cheap, $1/$5 per million tokens). My pricing table HAD that specific entry — but a typo in the model name. The real model returned by OpenRouter is sometimes served as anthropic/claude-4.5-haiku (version before tier, different convention).

My exact-match check failed. Costs fell through to the catch-all anthropic/* which pointed to Sonnet pricing ($3/$15 per million). Three times the real cost.

The bug was invisible because:

The dashboard “worked” — it showed numbers
The numbers were plausible — $4.50/week feels like a reasonable LLM cost
There was no error log because the catch-all matched correctly (just with wrong rates)

This is the most dangerous category of bug. Loud bugs you catch immediately. Silent bugs that produce plausible-but-wrong numbers can run for months.

Strategic decisions I made on bad data

In the two months when my dashboard was lying:

I slowed beta invites. Was worried about cost-per-user. Should have invited more aggressively.
I considered raising Starter pricing $10/month. Would have hurt conversion for no real reason.
I deprioritized features that consumed tokens. Specifically, the deeper research tools (multi-step reasoning, longer context). Those should have stayed.
I planned a tool-model swap to reduce cost on what I thought was the second-biggest token consumer. Turned out it wasn’t.

Four bad decisions, partially mitigated by being a careful person, but the bias was real. Bad dashboard data shapes product strategy in ways that look rational from the inside.

The fix

Two parts.

Part 1: read the real cost.

export function extractUsage(responseData: any) {
  const usage = responseData?.usage;
  if (!usage) return null;
  const rawCost =
    typeof usage.cost === "number"
      ? usage.cost
      : typeof usage.total_cost === "number"
        ? usage.total_cost
        : null;
  const cost =
    typeof rawCost === "number" && Number.isFinite(rawCost) && rawCost >= 0
      ? rawCost
      : null;
  return {
    promptTokens: usage.prompt_tokens || 0,
    completionTokens: usage.completion_tokens || 0,
    totalTokens: usage.total_tokens || 0,
    cost,
    servedModel: typeof responseData?.model === "string" ? responseData.model : null,
  };
}

When the provider returns real cost, use it. Static table is fallback only.

Part 2: backfill historical data.

Rather than just fixing forward, I wrote a script that walked every UsageLog row in the database, looked up the correct rate for the model recorded, and updated estimatedCost to match reality. About 5,000 rows. Total time: under a minute.

The dashboards now show truth all the way back to the product launch.

A checklist for anyone running LLM-powered SaaS

If you have a per-user cost dashboard and you’ve never re-verified the methodology, do it tonight. Bullet checklist:

Read your provider’s response.usage.cost field if available. OpenRouter, Anthropic, OpenAI all return it. Use it as the source of truth.
Match on both requested AND served model names. The model you REQUEST and the model your provider SERVES can differ in naming conventions. Catch both shapes in your pricing table.
Add an explicit pricing entry for every model variant you use. Don’t trust catch-alls. When you add a model, add the entry. When you remove a model, remove the entry.
Re-audit your cost methodology quarterly. Models update. Env vars change. Provider pricing changes. Your code didn’t notice.
Track requested model vs served model in your log. Saved me a debug round when I needed to figure out which model was actually doing the work in production.
Backfill historical rows when you fix the methodology. Otherwise your dashboards show two different truths — old wrong, new right — and your eyes will lie to you about trends.
Cross-check by hand monthly. Pick a single user, look at their token spend in the dashboard, manually multiply by current rates, compare. Catches drift before it becomes habit.

What this cost me

Two months of slightly suboptimal decisions. No actual lost money — the real cost was always lower than I thought. But the opportunity cost of slowing beta invites and dragging on pricing decisions: probably one or two beta users who would have been ready to be paid converters by now.

The lesson generalizes beyond LLM cost. Dashboards lie. Especially the ones you built yourself. Especially the ones you stopped checking the math on.

Bigger principle

A founder I respect once told me: half your job is being skeptical of your own data.

If you can’t reproduce the dashboard math by hand on a single row, you’re flying blind.

If you haven’t re-verified the methodology in 90 days, the methodology is probably wrong.

If your strategy depends on a number in a dashboard, audit the number before you commit the strategy.

Where this work happened

I’m building aicofounders.co — 6 AI cofounders for solo founders. The cost dashboard described here is part of the admin panel that lets me run unit economics in real time.

Closed beta. Free idea teardown (no signup): aicofounders.co/teardown.

If you’ve caught your own silent dashboard bug, I’d love to hear the story. Reply on X or email kyle@aicofounders.co.