Why Price Per 1M Tokens Is a Misleading Metric for LLM Costs

Every AI pricing page leads with the same number: dollars per million tokens. OpenAI, Anthropic, Google, DeepSeek - they all compete on this metric. But comparing LLMs by their per-token pricing alone is fundamentally flawed.

A new analysis making the rounds on Hacker News breaks down exactly why - and proposes a better metric that changes which models look like good value.

The Core Problem

Token pricing fails for two reasons that compound on each other:

Tokenizers are not standardized. Different labs use proprietary tokenizers that split identical text differently. The same content might require 160 tokens for GPT-4o but 200 tokens for GPT-4. Anthropic recently modified its tokenizer, causing a 30% increase in tokens for the same input.

When you compare $X per million tokens across providers, you are comparing apples to oranges. A "token" from OpenAI is not the same unit as a "token" from Anthropic.

Token efficiency varies dramatically. Hidden chain-of-thought processing - where models reason before producing output - consumes tokens billed at standard rates but varies wildly between models and use cases. A model that thinks more might produce fewer output tokens but consume many more thinking tokens you do not see in the final response.

Cost Per Task: A Better Metric

The proposed alternative: measure "cost per benchmark task" using real benchmark data. This reveals actual economic value delivered rather than nominal pricing.

The comparison table from the original analysis demonstrates the problem starkly:

• GPT-5.5 costs more per token than Claude Opus 4.8, yet completes tasks at nearly half the price
• DeepSeek V4 Pro charges dramatically less per token ($0.435/$0.87 input/output) but costs only $0.04 to $0.05 per task - revealing extreme efficiency
• Claude Sonnet 5 underperforms Opus while costing more per task

That last point is notable. Anthropic's own initial benchmarks showed Sonnet 5 with lower performance at higher costs than expected. The per-token price looked competitive; the per-task economics did not.

What HN Is Saying

The Hacker News discussion added several important nuances:

Caching matters enormously. One commenter noted: "Caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens)." A model with better caching support can be dramatically cheaper in multi-turn conversations even with higher nominal token prices.

Thinking levels change the equation. "Setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash)." More thinking can mean fewer failed attempts, fewer tokens wasted on wrong paths, and faster completion.

Benchmark difficulty matters. Cost per benchmark task is only useful if the benchmark matches your workload. "Cost per benchmark task is meaningless if your task is difficult enough that the cheaper model has no chance of cracking it." For trivial tasks, the smaller model wastes tokens backtracking while the larger model does it right the first time.

Local LLM users see this too. "tok/s isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals in the minimum amount of time." Some models have better tok/s but are so verbose they generate many more tokens - making clock time longer despite the higher throughput.

The real problem is black box uncertainty. "You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume 'the cheaper and older model is probably going to cost less overall.'" This assumption is often wrong.

The Subscription Wrinkle

Several commenters pointed out that token pricing is even more misleading for subscription users. Monthly plans price tokens extremely differently than their per-token billing rates. Most developers using Claude Code or ChatGPT Plus are not paying API rates at all.

Cost-per-task analysis should ideally account for subscription token allocations, but that data is rarely available.

Practical Implications

If you are selecting models based on per-token pricing alone, you are likely choosing suboptimal solutions. Here is what to do instead:

Run your own benchmarks. The only cost metric that matters is cost for your actual workload. Generic benchmarks help, but your task distribution is unique.

Track total cost per task. Instrument your agent workflows to log total tokens consumed (input, output, thinking) and correlate with task success rates. A model that fails 20% of the time costs more than one that succeeds consistently even at higher per-token rates.

Account for caching. Multi-turn conversations with good cache hit rates can reduce costs 10x. Check each provider's caching behavior with your prompt patterns.

Test thinking levels. Higher thinking settings sometimes complete tasks faster and cheaper by avoiding failed attempts. Do not assume "low" is always cheapest.

Consider latency. A model that costs more per token but finishes in 2 seconds might be cheaper than one that takes 30 seconds if your time has value. One commenter wanted a model for commit messages that finishes quickly - high benchmark scores were irrelevant if it took a minute.

The Open Model Question

Several commenters advocated for local models to avoid per-token uncertainty entirely. Fixed hardware costs are predictable; token costs are not.

The counterargument: open models are not yet competitive for end-to-end agentic workflows. They excel at bounded tasks but struggle with the kind of multi-step reasoning that frontier models handle.

One detailed response described success with Mimo v2.5 at $0.017 per million tokens - building an orchestrator that handled planning, execution, and review with quality "that makes me laugh at things like Opus." The open model space is catching up fast.

The Bottom Line

Price per million tokens is a unit measure, not a value measure. It tells you what you pay for a unit of computation but says nothing about what that computation accomplishes.

Just as price per gallon does not tell you trip cost without knowing fuel efficiency and distance, price per token does not tell you task cost without knowing model efficiency and task complexity.

The right question is not "which model is cheapest per token" but "which model completes my tasks most cost-effectively." Those are often different answers.

Sources

• Price per 1M tokens is meaningless (janilowski.pl)
• Hacker News Discussion
• Artificial Analysis Benchmark Data