AI Code Comprehension Checks vs Tests: When Each Wins

Here's a scenario that's becoming common: your coding agent writes 600 lines across four files, the tests pass, the linter is happy, and you merge it. Two weeks later it breaks in production and you stare at the diff like a stranger wrote it — because, functionally, one did. The failure wasn't in the code. It was in the gap between the code and your mental model of it. That gap is what AI code comprehension checks — gates that quiz you on a diff before it ships — are trying to close. A new open-source Git hook called Sphinx made the rounds this week doing exactly that: before your push goes through, it reads the diff and asks you a question about what the code does when things go wrong.

My instinct as a former ML engineer was to be skeptical. Another gate? Another ritual? But the more I thought about the failure modes I actually see with agent-generated code, the more I think this idea targets something tests structurally cannot. So let's do the honest comparison: comprehension checks versus the automated safety net you already have — tests, linters, type checkers, CI — and when each one actually wins.

What Tests and Linters Actually Verify

Automated checks answer one question: does the code behave the way the checks expect? That's a loop closed entirely inside the machine. A test encodes someone's belief about correct behavior, the code either satisfies it or it doesn't, and no human understanding is required at merge time. This is a feature. It's what lets a hundred engineers ship into the same repo without reading each other's work.

The problem is the assumption baked into that model: that the person merging has an accurate mental model of what the code should do, and that the test suite reflects it. With hand-written code, that assumption mostly holds — you wrote it, you fought with it, you know where the bodies are buried. With agent-generated code it quietly breaks. You described a goal, the agent produced an implementation, and the tests pass because the agent wrote those too. The green checkmark is now certifying that the agent's code matches the agent's tests. Your understanding was never in the loop.

This is the same trap I wrote about in why AI benchmarks miss most of real work: a metric can only certify what it measures. A test suite measures behavioral conformance on the cases someone thought to encode. It says nothing about whether the retry logic double-charges a customer on a Stripe timeout, whether the agent picked the wrong endpoint entirely, or whether the 'fix' papers over a race condition by swallowing an exception. Those are comprehension failures, and they sail through CI.

What Comprehension Checks Catch That Tests Miss

A comprehension check flips the direction of verification. Instead of asking 'does the code work?', it asks 'does the human shipping this know what it does when things go wrong?' Sphinx's illustrative example is a good one: your webhook handler now saves the Stripe event ID before calling charge(). What happens on a retry after a timeout? If you say 'the customer gets charged twice,' you're wrong — the saved ID short-circuits the retry. But then the follow-up: if charge() itself fails, what does the next retry do? It skips the charge. The customer is never billed, and no test fails, because this is a correct-looking idempotency pattern applied in the wrong order.

Notice what kind of bug that is. It's not a syntax error, a type error, or a failing assertion. It's a design flaw that only surfaces under a specific failure sequence, and the only cheap way to catch it before production is for a human to reason about the failure path. Tests could catch it — if someone wrote a test for 'charge fails after event ID is persisted.' But the person who would write that test is the person who understood the ordering risk, and if that person existed, we wouldn't be here. Comprehension checks probe for exactly the understanding that would have written the missing test.

There's a second benefit that's less about catching bugs and more about reversing a slow skill leak. Plenty of developers describe the same experience: bashing an agent toward a working result consumes real energy but leaves nothing learned behind. You ship more and understand less, and the gap compounds. A quiz at push time forces at least a shallow reconstruction of the diff's logic — which is often enough to turn 'code I approved' back into 'code I own.' I've argued before that AI tools are quietly changing how developers think; gates like this are one of the first tools that push in the opposite direction.

When Automated Tests Win

Let's not romanticize the human gate. Tests win decisively in several arenas, and pretending otherwise is how you end up with a quiz as your only safety net.

• Regression detection at scale. Nobody's mental model survives a 30,000-line refactor. Tests re-verify thousands of behaviors per commit for pennies. A comprehension check can cover maybe three questions' worth of one diff.
• Mechanical correctness. Off-by-one errors, wrong format strings, mishandled time zones — machines are better at exhaustively checking these than a human answering questions under time pressure.
• Repeatability. A test runs identically forever. A quiz question's quality depends on whatever model generated it and however awake you are at 5pm on a Friday.
• Refactoring confidence. When behavior shouldn't change at all, tests are the only check that means anything. Understanding the code doesn't help you verify you didn't change it.

There's also a genre of code where comprehension checks add friction without value: generated boilerplate, mechanical migrations, dependency bumps, config changes. Quizzing me on what a Prettier reformat does is a tax, not a gate. Any tool in this space needs a good answer for low-risk diffs or people will route around it — which, tellingly, Sphinx anticipates with a sphinx yolo escape hatch that logs every skip. That logging matters more than it looks; a gate everyone bypasses silently is worse than no gate.

When Comprehension Checks Win

Comprehension checks win precisely where the human is the weakest link, which in agent-assisted development is more often than we'd like to admit.

• Failure-path reasoning. Retries, timeouts, partial writes, distributed transactions — the code paths that only execute when something else is already broken. Agents are decent at the happy path and cavalier about the sad one.
• High-blast-radius diffs. Payments, auth, data migrations, anything touching systems that should resist catastrophe. The cost of a wrong answer here dwarfs two minutes of quiz friction.
• Code you didn't write and barely reviewed. The 1,500-line PR from a non-dev colleague, the agent's autonomous overnight branch. One practitioner reported using a 'quiz me on this PR' prompt just to stay focused during structured review of exactly these.
• Onboarding and skill maintenance. If your team is shipping agent code faster than anyone is learning the domain, a comprehension gate is the cheapest forcing function available.

The common thread: comprehension checks earn their keep when the diff is consequential and the human's mental model is suspect. That's a specific niche, not a universal one.

The Failure Modes of the Gate Itself

I'd be doing you a disservice without the honest list of ways this goes wrong, because the skeptics have real points.

First, confirmation fatigue. We've been here before — cookie banners, 'are you sure?' dialogs, security questions everyone clicks through. A gate that fires on every push trains people to answer fast rather than think, and within a month the quiz is ritual theater. The mitigation is scoping: run it on protected branches and high-risk paths, not on every commit to a feature branch.

Second, question quality. The quiz is generated by an LLM reading the diff, which means it inherits LLM failure modes — shallow questions, wrong 'correct' answers, fixation on the obvious change while missing the subtle one. One reviewer noted their version helped structure review but didn't catch poor decisions like calling the wrong endpoint. A bad question is worse than no question, because it certifies understanding it didn't test. This is a real limitation and I don't think it's fully solvable; it's the same reason agent harness design matters more than model choice in so many of these tools.

Third — and this one made me laugh because it's true — someone will eventually point an agent at the gate to answer the questions for them. The tool's design shows awareness of this: your answers never reach the coding agent, and secrets stay closed. But nothing stops a determined developer from pasting the question into a chat window. At that point the tool is measuring willingness to cheat, which is a culture problem no hook can fix.

A comprehension gate doesn't verify the code. It verifies that a human with context and accountability actually engaged with it — which is the thing agent workflows quietly removed.

Building the Gate Into a Real Workflow

If you want to try this, you have two practical options. The low-friction path is a prompt-based skill in your existing agent: 'quiz me on the current branch before I open the PR.' Several people built exactly this and report it works well for structuring review — but notice it relies on you choosing to run it, which fails exactly when you're most rushed.

The enforced path is a Git hook. Sphinx installs in a minute — npm install -g sphinx-understand, then sphinx install, which wires up Git and your coding agents and asks which model should review. It works with Claude Code, Codex, Cursor, and others, or plain Git, and it can run off any OpenAI-compatible endpoint. If you'd rather roll your own, the skeleton is a pre-push hook:

#!/usr/bin/env bash
# .git/hooks/pre-push — quiz gate for protected branches
protected_branch="main"
current_branch=$(git rev-parse --abbrev-ref HEAD)
if [ "$current_branch" != "$protected_branch" ]; then
exit 0  # only gate the branches that matter
fi
diff=$(git diff origin/main...HEAD)
if [ -z "$diff" ]; then
exit 0
fi
# Send the diff to your model of choice, get a scenario question,
# check the answer. Pseudocode for the core loop:
question=$(quiz_generate "$diff")
echo "$question"
read -r answer
if ! quiz_verify "$diff" "$question" "$answer"; then
echo "Not quite — here's why. Push blocked; try again."
exit 1
fi

Two design decisions matter more than the code. Gate branches, not pushes — fire on pushes to main and release branches, and ideally also on 'request review,' because catching slop before it consumes a teammate's attention is arguably the highest-value placement. And log every bypass. The escape hatch is necessary; the audit trail is what keeps it honest.

Also consider where this sits relative to human review. I've written about how every layer of review makes a team slower, and I stand by it — so don't add this on top of everything. Use it to replace the first, shallowest pass of review: the 'does the author even know what this does' pass. That's the pass comprehension checks genuinely automate.

The Recommendation

Tests and comprehension checks aren't competitors — they verify different subjects. Tests verify the code; comprehension checks verify the person accountable for it. Agent-assisted development broke the implicit assumption that the two were the same thing, and these gates are a pragmatic patch, not a gimmick.

My recommendation: keep your test suite as the primary safety net — nothing here replaces it — and add a comprehension gate narrowly, on protected branches and high-risk paths, with a logged escape hatch. If you write most of your code by hand, you don't need it; the gate's value scales with how much of your diff you didn't write. And if the quiz starts feeling like a chore you route around, that's signal too: it means you're shipping code you don't understand fast enough that a two-minute pause feels expensive. Sit with that. The uncomfortable version of this tool's thesis is the correct one — you can outsource the thinking, but the understanding has to stay yours, because at 3am when the retry loop bills nobody, the agent isn't on call. You are.