My Rule for Deciding When to Throw Out AI-Written Code

Ever explained something calmly, in writing, to someone, and they still went and did the exact opposite of what you'd agreed on? That "wait, we already talked about this" moment leaves you unsure whether to laugh or get annoyed. It happens to me a lot directing AI to write code.

I work on Work Connect, a product that helps hospitals quickly fill the nursing shifts left open when someone calls in sick. A mistake there isn't just an inconvenience. If a shift goes uncovered, someone has to fix it fast, and the usual options (calling everyone you know, paying an outside agency a premium) are slow and expensive. So there's no room for "we'll clean it up later."

That's exactly the setting where, while building a new feature, the AI I was directing ended up building the exact opposite of what we'd decided to do.

A spec that outranks the code

At Work Connect, before the AI writes a single line, the documents already exist: a product spec, a requirements doc per feature with its own acceptance criteria, a technical blueprint, and the work orders that split it into pieces. Only then does the AI turn all of that into code, and I check 3 things: the documents that define what needs to get built, the feature that actually got built, and the tests backing it up.

When what got built and the plan disagree, what got built is what gets corrected. Never the plan.

The moment I said "this doesn't work"

It happened with an ordinary feature, one that already had its plan fully defined before anyone touched a keyboard. The AI read that plan, wrote the code, and ended up building the exact opposite of what that plan called for.

What came out was a duplicate approach, one we'd already evaluated before and ruled out in writing. The plan didn't just say what to do. In effect, it also said, "we already thought about this, and it's not the way."

When I saw the AI had done it anyway, I didn't hesitate long. I closed that branch of work and went back to the original plan.

Why not just fix what the AI had already built?

Because what the AI built wasn't a small thing to tweak: it directly contradicted a decision we'd already made in writing. Patching it in place would have left that wrong decision half-alive in the codebase, like a bad example the next person (or the same AI) would copy without ever knowing we'd already ruled it out.

It wasn't a one-off bug either. What got built went against a decision that was already made, which means the instruction the AI followed was wrong at the root, not just in the details. An instruction that gets misread once can get misread again somewhere else.

I'd rather lose that work and keep a clean plan than keep a half-fix built on a foundation I already knew was wrong.

The same principle, now inside a team of 6 agents

That episode was about one feature, and I reviewed it myself before it ever reached production.

But Work Connect also has a part that doesn't go through that kind of human, case-by-case review: a team of 6 AI agents that automates a big chunk of the shift-covering process every single day. There, that same rule (never take what the AI did on faith) couldn't depend on me checking every action by hand anymore. It had to become something the system enforced on its own.

Each of those 6 agents handles a piece of work a person used to do manually: they notice when a nurse reports she can't make it in, look for a replacement, screen the candidates who apply, and even make the phone call to confirm someone isn't missing a credential before their shift starts.

But none of them act on their own just because they can. They all ship switched off, and only turn on for a rule a hospital coordinator wrote out explicitly, in plain language.

Before that rule actually takes effect, the panel shows that person exactly what the agent understood, so they can confirm it. And if anything is unclear, the agent doesn't improvise: the shift goes to a manual queue for a person to handle.

This exact tension, how much to trust AI without someone checking all of it by hand, is being argued about in public this year, and I'm not the only one asking it. The reason underneath is simple: an AI can produce work far faster than a person can carefully review it.

Open source projects have shut down their bug bounty programs because more AI-generated reports came in than their human team could verify, and GitHub is weighing an emergency brake for pull requests that, in practice, nobody's really reviewing.

It's not that AI is bad. It's that if nobody sets a limit, speed wins and verification loses.

If you direct AI somewhere a mistake actually matters, you already know what I mean. And if it hasn't happened to you yet, it will. The question isn't whether the AI will ever go off plan. It's what you do when it does. Thanks for reading this far. 🙂