The Hidden AI Cost of JavaScript Frameworks

The agent spent eleven minutes reading a migration guide before it wrote a single line of code. Not your agent — the one a colleague pointed at a SvelteKit project last week, using the same prompt you'd given your React app an hour earlier. Yours finished in about eight minutes and four million tokens. Theirs took seventeen minutes and nearly ten million, and a big chunk of that difference was the model patiently re-teaching itself a framework version it had never seen during training. That gap has a name now: the AI cost of JavaScript frameworks. And thanks to a genuinely unusual act in our industry — somebody actually measuring things instead of asserting them — we finally have real numbers for it.

Nolan Lawson, who has spent years replacing frontend folklore with actual benchmarks, recently did the obvious thing nobody had bothered to do. He specced a small but non-trivial Pokémon app — search, filtering, detail pages, data from the PokéAPI — and had a coding agent build it twelve different ways: Astro, Next.js, Nuxt, SvelteKit, React Router, SolidStart, Qwik, Angular, TanStack Start, Hono with htmx, Gracile with Lit, and plain Vite with no UI framework at all. Each framework got five independent runs with the same prompt, each build was checked against a conformance suite so the agent couldn't cheat, and the results were scored on cost, time, tokens, and Lighthouse metrics. It's not a perfect experiment, and Lawson says so himself. But it's the first honest attempt I've seen to put a price tag on a decision most teams make on pure vibes.

What the Benchmark Actually Measured

The setup was deliberately neutral. The agent got the spec, was told to use the latest version of a given framework, and was left to choose its own architecture — SSR or CSR, whatever it liked. Where a framework ships an official MCP server, agent skill, or plugin, it was made available, so no framework could complain it was underrepresented in the training data. The conformance tests ran against the accessibility tree rather than the raw DOM, which mattered because one contender renders through shadow DOM. Each framework got five runs and the results are reported as medians — except Angular, which twice failed to produce a testable app at all, so some of its numbers come from three runs.

• Build cost ranged from a median of about $2.67 to $4.97 per app — roughly a 2x spread between the cheapest and most expensive frameworks.
• Token consumption told the same story louder: around 4 million tokens for the cheapest stacks, nearly 10 million for the most expensive.
• Wall-clock time tracked cost closely, from about 8 minutes at the fast end to 17 minutes at the slow end.
• SvelteKit, Gracile+Lit, and Angular were consistently the most expensive builds. Astro and the Hono+htmx pairing were consistently cheap.
• Surprisingly, the no-framework Vite build underperformed several actual frameworks — some structure beats no structure, even for a machine.

Two bucks per build sounds like nothing. It is nothing, for one build. But agentic development is iterative by nature — nobody accepts the first output. Multiply the spread by dozens of agent sessions per developer per week, across a team, across a year, and a 2x token multiplier on your stack becomes a real line item. More importantly, tokens are a proxy for something I care about more than money: how much trial-and-error the model needs to produce correct code in your framework. High token counts usually mean the agent is confused, re-reading docs, and retrying. That's a quality signal wearing a cost costume.

Why Some Frameworks Make Agents Expensive

Having spent years training and evaluating models, I find the results less mysterious than they look. Three mechanisms explain nearly all of the spread, and none of them have anything to do with which framework is "better" in the traditional sense.

Training-data density is destiny

React's single biggest advantage in the agentic era isn't its programming model — it's that a decade of tutorials, GitHub repos, and Stack Overflow answers have compressed its idioms into the weights of every frontier model. An agent writing React is mostly recalling. An agent writing Gracile is mostly discovering: in the benchmark runs, agents that didn't know the framework resorted to cloning the framework's repository or reading its compiled source out of node_modules. That's tokens burned on remedial education before any user code exists. This is the same dynamic reshaping how developers find knowledge generally — I wrote about what happens to the corpus of developer knowledge as Stack Overflow fades, and the framework angle is a sharp special case: unpopular tools don't just lack community, they lack presence in the model itself.

API churn is a recurring token tax

SvelteKit's poor showing has a specific, almost funny cause: the agent read the SvelteKit v3 migration guide on every single run, because that version was too new to be in the model's training data. Every breaking change a framework ships invalidates patterns the model has memorized and forces it to re-derive the correct API from documentation at inference time — paid for in tokens, every session, by every user. Frameworks treat major version churn as a marketing event. For agent users, it's a recurring surcharge. A boring framework that hasn't broken its API in three years is, by this metric, a premium feature.

Verbosity compounds through the loop

Angular's cost seemed to track its sheer code volume — more generated lines means more tokens out, more tokens to re-read on every subsequent edit, and more surface area for the agent to trip over. This generalizes: any stack with verbose tool output, noisy build logs, or slow test commands forces the agent to wade through more text per iteration, and every failed lint or type check triggers another expensive round trip. Practitioners report the same pattern everywhere: the cost of iteration, not the cost of generation, is what dominates agent spending. A framework with a fast, quiet, strict toolchain is cheaper to operate than a permissive one, because strictness catches agent mistakes in one cycle instead of five.

Every breaking change your framework ships is now a tax that someone pays in tokens — and unlike a human reading a changelog, the model pays it on every single session.

The Dice Roll Problem With Agent Benchmarks

Here's where my old ML evaluation habits kick in, because the most important detail in the whole study is the error bars. LLM output is a distribution, not a point. React Router's median build cost was about a dollar worse than Hono+htmx's — yet React Router's single cheapest run came in below Hono's most expensive one. The ranges overlap substantially. SvelteKit's runs swung from about $3.20 to $6.80. Sometimes the agent just rolls low, wanders off on a tangent, and torches a few million tokens for reasons that have nothing to do with the framework.

The tooling usage data makes the same point. Frameworks shipped MCP servers and skill files, and the agents used them... sometimes. Nuxt's MCP server was never touched in five runs. TanStack Start's skill files were discovered in two runs out of five. React Router's skill was used every time. Same prompt, same harness, wildly different behavior — that's the stochastic nature of the tool. Which means every Twitter thread claiming "framework X is so much better for agents, I rewrote my whole app" based on one afternoon of vibe-coding is measuring noise. One run is an anecdote. If you're going to let AI cost influence an architectural decision — and I think you should — demand medians over repeated runs before you believe anything, including this benchmark.

On Output Quality, Everything Converges to Fine

The most surprising result is what didn't vary. Lighthouse Performance, Accessibility, and Cumulative Layout Shift scores were nearly saturated across all twelve stacks — 99s and 100s almost everywhere. Given a clear spec and a conformance suite, modern agents produce competent, accessible small apps in basically any framework. The quality floor has risen to the point where the framework barely differentiates output on a content-oriented site.

The one quality metric with real spread was JavaScript bytes shipped: zero KiB for the islands-style Astro and Hono+htmx builds, up to around 142 KiB for the heaviest SPA output. Total Blocking Time stayed trivial across the board, and LCP was too noisy to read much into. Two honest caveats apply here. The test app is closer to Wikipedia than to Figma, so it flatters multi-page architectures, and Lighthouse itself is structurally biased against SPAs. Still, the vanilla-JavaScript result deserves a pause: the agent told to use plain Vite with no framework produced worse scores than several actual frameworks, including the highest layout shift of any build. Frameworks encode conventions, and agents lean on conventions hard. Give the model a well-worn groove and it stays in it; give it a blank field and it improvises, not always well.

What This Benchmark Can't Tell You

Before anyone rewrites their production app based on a Pokémon demo, let's be honest about scope. This is one small, mostly content-driven application with little cross-cutting state, no localization, no complex forms, and no team of humans who'll maintain it for five years. That's a real limitation, and it's the same trap I keep circling on this blog: AI benchmarks systematically miss most of what real work looks like. The properties that make a framework good for agents on a greenfield toy — familiarity, simplicity, static structure — aren't the properties that matter at month eighteen of a messy product.

There's also a deeper point the framework skeptics in the discussion got right. Much of a framework's historical value was never about what it lets you do — it's about what it forbids. Java conquered enterprise development partly because it forced ten thousand programmers to write code that looked alike. Frameworks are constraint systems for teams of humans who join, leave, and misunderstand each other. An agent doesn't need guardrails against boredom or resume-driven development; it needs a clear spec and a test suite. One practitioner reported exactly this inversion with htmx: the agent breezed through simple pages but floundered on a complex stateful form, dropping state and breaking toggles — problems React would have one-shotted — until enough tests existed to keep it honest. The constraint that matters for agents isn't the framework. It's the conformance suite.

And yes, the cheapest build isn't the cheapest codebase. Paying an extra two dollars per agent session for a stack your team understands deeply and can maintain for years is obviously correct. But notice what that argument requires: numbers. The entire point of measuring AI cost isn't to crown a winner — it's to replace "this framework is better for agents" with a quantity you can weigh against hiring pools, ecosystem maturity, and bundle size. Vibes lost the DX debate; they shouldn't win the agentic one by default.

How to Measure Your Own Stack's AI Cost

Don't extrapolate from anyone else's benchmark — including this one — to your codebase. Your repo, your spec, and your model of choice will produce their own distribution. The good news is that measuring AI cost is genuinely easy, because coding agents already report everything you need. If you want to understand how the loop, context, and permissions interact before you instrument them, our piece on how an agentic coding CLI actually works is a good foundation. Then do this:

1. Pick a vertical slice that represents your real work — a feature touching your data layer, your state management, and your UI, not a landing page.
2. Write the conformance tests first. Tests are both your quality gate and your anti-cheating device; an agent without tests will declare victory early.
3. Run the same prompt at least five times per candidate stack or configuration. Report medians and ranges, never single runs.
4. Log total tokens (split by cache reads, input, and output), wall-clock time, cost, and the number of tool calls and retry loops.
5. Track how many tokens go to documentation reads — migration guides, changelogs, node_modules spelunking. That's your training-data-deficit tax, measured directly.
6. Re-run the benchmark after major framework upgrades. AI cost drifts as APIs churn and training cutoffs move.

The harness itself can be an afternoon of scripting. Here's the skeleton I've used — the specifics of the CLI flags will depend on your agent, but the shape is universal:

import json
import statistics
import subprocess
FRAMEWORKS = ["astro", "react-router", "sveltekit"]
RUNS = 5
SPEC = open("spec.md").read()
def build_once(framework: str) -> dict:
"""One full agent session: build the app, return usage stats."""
result = subprocess.run(
[
"agent-cli", "run",
"--prompt", SPEC,
"--context", f"Build this with the latest {framework}.",
"--output-format", "json",
],
capture_output=True,
text=True,
check=True,
)
return json.loads(result.stdout)
for framework in FRAMEWORKS:
runs = [build_once(framework) for _ in range(RUNS)]
costs = [r["cost_usd"] for r in runs]
tokens = [r["tokens_total"] for r in runs]
print(
f"{framework}: "
f"median ${statistics.median(costs):.2f} "
f"(range ${min(costs):.2f}-${max(costs):.2f}), "
f"median {statistics.median(tokens) / 1e6:.1f}M tokens"
)

Watch the retry count as closely as the token total. In my experience, tokens-per-accepted-test is the single most honest metric: it collapses cost, confusion, and correctness into one number, and it punishes the failure mode that actually hurts — the agent thrashing through five compile-error loops before landing something that passes. A framework where the agent passes cleanly on the first or second attempt will beat a nominally "cheaper" one that needs constant correction, every time.

The Pragmatic Takeaway

So should you rewrite your Angular app in Astro because agents find it cheaper? No. The dollar deltas are small, the variance is large, the test app is a toy, and your migration would cost more agent tokens than you'd save in a decade. Anyone selling you a framework migration "for the agents" is doing the same vibes-based reasoning the industry has always done, just with a new justification bolted on.

But AI cost has earned a seat at the decision table, next to hiring pool, ecosystem depth, and bundle size. For greenfield projects, the evidence points in a consistent direction: choose stacks with deep training-data representation, slow API churn, strict and quiet toolchains, and conventions the model has seen ten thousand times. Boring technology was always the pragmatic choice; agents just made boring measurable. Pin your framework versions, because every upgrade is now also an inference-time tax until the models catch up. And invest in conformance tests regardless of which framework you pick — they're the one asset that makes every agent cheaper, keeps it honest when it flounders, and pays off long after whatever model you're using today is obsolete. The frameworks will keep churning. The tests, and the habit of measuring, are what survive.