The worst AI code isn't the code that crashes. Code that crashes, you notice.
The worst one looks right, passes a quick glance, and the AI says "Done! ✅". Then it breaks in production two weeks later.
You're not alone in this. In Stack Overflow's 2025 developer survey, the most common complaint about AI tools was "AI solutions that are almost right, but not quite" (66% of developers). In the 2026 survey, nearly half said they trust AI output only when they can check it easily.
So here's my list of how "almost right" sneaks in, with a check for each that takes about 30 seconds. I use AI coding tools every day, and I keep running into every one of these.
1. "Fixed!" … but nothing was run
The AI edits the code, explains the fix nicely and stops. No test ran. Sometimes a test command did run, but it found 0 tests and still exited green.
30-second check: ask "Which command did you run, and what was the exit code?" Then look at the output for the number of tests that actually ran. 0 passed is not a pass.
2. The tests pass because the test changed
The fastest way to make a red test green is to change the test, not the code. Look out for:
-
it.skip(/xit(/@pytest.mark.skip -
expect(total).toBe(90)quietly becomingexpect(total).toBeDefined() - a snapshot "updated" to match the new (wrong) output
30-second check: git diff -- '*test*' '*spec*' before you accept anything. If a test file changed and you didn't ask for that, ask why.
3. A function or package that doesn't exist
It reads perfectly: import { parseSafe } from "zod/utils". That function is not there. Or it suggests npm install of a package with a very believable name. Attackers now register those made-up names on purpose, so that slip can turn into a real risk.
30-second check: open the real docs or the npm page. A brand-new package with a handful of downloads is a red flag.
4. Only the happy path works
The demo input works. An empty list, null, a timezone, a 0, a very long string or a double click doesn't.
30-second check: ask the AI, "Which 3 inputs would break this?" Then run those 3. It is surprisingly honest when you ask that directly.
5. Errors get swallowed
The code stops crashing because the crash is hidden:
try {
return await getOrders(userId);
} catch {
return []; // the user just sees "no orders"
}
Nothing fails, nothing logs, and the bug becomes a support ticket.
30-second check: search the diff for catch, || [], ?? null and except: pass. Each one should log, rethrow or have a reason.
6. The right answer for the wrong version
The code is correct for a version you don't use: the Next.js pages router in an app-router project, an old React Router API, a removed config option. It compiles in the AI's head, not in your project.
30-second check: compare the imports with the versions in your package.json (or lock file), and paste your version into the prompt next time.
7. Security that "works"
The login works, so it looks done. But the JWT is decoded instead of verified, the secret is hard-coded, or an API route never checks who is calling it.
30-second check: for every new route, ask "Who is allowed to call this, and where is that checked?" Search the diff for strings that look like keys.
The habit that catches most of these
Before accepting any AI change, run one command:
git diff --stat
Did it touch files you never mentioned? Did test files change? Is the diff 10× bigger than the task? That one look catches half of this list.
Your turn 👇
Which one has bitten you most recently? And what's the 8th way that's missing from this list? I'll add the best ones (with credit) to a follow-up post.
Small note: I got tired of doing checks 1 and 2 by hand, so I built a free Claude Code plugin, proof-of-done, that does them automatically, and a free security-audit skill for number 7.
Happy Diwali in advance 🪔: may your builds be green and your tests honest.
Everything I've built (all free and open source)
which-ai-to-use: picks the best AI and model for each task, and hands your work to another AI when you hit a usage limit
- Try it:
npx which-ai-to-use - npm · VS Code Marketplace · Open VSX (Cursor, Windsurf) · GitHub · Website
proof-of-done: a Claude Code plugin that won't let the AI say "done" until the tests pass
- Install:
/plugin marketplace add WST2002/claude-code-skillsthen/plugin install proof-of-done@claude-code-skills - GitHub · Featured on the HOL blog
security-audit: a Claude Code skill that finds security holes before attackers do
All my Claude Code skills: github.com/WST2002/claude-code-skills
Watch the demos on YouTube: @wst2002rk · proof-of-done Short · which-ai-to-use Short · security-audit Short
Read more: Switch AIs mid-task without losing context · Your AI said "fixed" but never ran the tests · My AI finds security holes before attackers do
Top comments (30)
2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking
git diff --statfirst is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
Thank you, that's a fair point. Each edit passes review on its own, so the only place the extra changes show up together is the final diff. That makes --stat useful even with step-by-step review, not only with auto-accept.
Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
Thank you, that's a good addition to #3. The package is real, so nothing looks wrong, and the cost only shows up in the lockfile. Checking the lockfile's line count is a quick way to catch it. When this happens to you, do you usually swap the package for a few lines of your own code, or look for a smaller package that does the same job?
Depends on what it does. If it's formatting or a small helper, I write the fifteen lines myself and add a test. For dates, parsing or anything crypto I keep a well-known library even if it's heavier, because there my fifteen lines would be the bug. The smaller lookalike package is the option I trust least, a lot of them have one maintainer and no release in years.
Thank you, that's a sensible split. Your point about lookalike packages is a good one. Before I accept a package an agent suggests, I'll start checking its last release date and number of maintainers on npm. That takes about ten seconds.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is
toBe/toEqualdowngrading totoBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding
catch { return [] }.Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags
toBe/toEqualturning intotoBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for
.toBeDefined()or wrapping a network failure in a silentcatch { return []; }.My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The
git diff --stathabit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
the one that bites me isn't on the list: right shape, wrong value. no exception, nothing to catch, the field is filled in, it's just not what the page said. a parser can read annual as monthly across a whole layout and never once fail.
check: 5 rows at random, diffed against the source by hand. checking only the rows that already looked odd catches nothing.
Thank you, this one should have been on the list. It's the hardest kind to catch because every check passes. Random rows is a good habit. One more that helps with parsers: compare a total from the source page (sum of a column, row count) with the same total from the parsed data. If annual was read as monthly, the totals stop matching right away. Do you run your 5-row check on every parser change, or only for new sources?
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Thank you for reading! Fully agree, the AI should never be the only one approving its own work. Which review tool do you use?
the check on number 2 is the one i would make automatic, because a green run where the test file also changed tells you nothing. we added a rule where any diff touching a test file needs a one line reason in the PR, and a test that got skipped or loosened to toBeDefined fails the review bot. the failure that still slips through for us is number 5, a catch that returns an empty array, since it passes every test written by the same agent. do you have a way to catch swallowed errors besides grepping the diff for catch?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.