OpenAI published a post yesterday titled ‘A scorecard for the AI age’. The core question: how do you actually measure whether your AI spend is paying off?
The answer OpenAI proposes is ‘Useful Intelligence per Dollar’.
The argument
For years software was measured by adoption: seats purchased, users active, licenses renewed. That doesn’t work for AI, OpenAI argues. What counts is work accomplished.
And for that, price per token is the wrong lens:
“A lower-cost model may have cheaper tokens, but getting great results may require more attempts, more time, or more human review. A more capable model may have more expensive tokens, but complete the same task in one pass.”
The proposed math is simple. Add up the full cost of the work — including employee time, review, retries, rework. Count the tasks that met the quality bar. Divide. There’s your cost per successful task.
Four questions are meant to fill out the scorecard: is AI completing work that matters? What does each successful task cost? Can people depend on the result? And does each dollar buy more value as usage grows?
The timing is anything but accidental
The post lands two days after Kimi K3 showed up — 2.8 trillion parameters, open weights promised, and API pricing at $3 per million input tokens and $15 per million output. That’s roughly half of Anthropic’s Opus 4.8. In blind testing, developers preferred Kimi over every leading US model for front-end coding.
When your price advantage disappears, you change the metric. That’s how this works.
And of course the same post sells the fix: GPT-5.6 ships in three tiers — Sol as the flagship, Terra for balance, Luna as the fastest and cheapest. Pick the right tier per task and you optimize exactly that equation. Convenient.
My take
I like the argument even though I can see the motive behind it.
Because it’s true. Anyone who has watched a cheap model botch the same task three times in a row and then cleaned up by hand knows the token price is just the tip of the bill. I notice this with Claude Code daily — the more expensive model that lands on the first try is almost always the cheaper one.
But the metric has a catch. Cost per successful task can be defined beautifully so that your own model wins. Who decides what “successful” means? Who counts the rework? As long as every vendor writes their own scorecard, this isn’t a benchmark — it’s marketing with a formula attached.
Still: OpenAI is asking the right question. Someone else should probably answer it.
Sources: