Why My Tests Passed While My Blog Was Broken in Production

Ever checked something carefully, saw green lights everywhere, and the result still turned out broken? That happened to me with my own website. I wrote the code, ran every automated check I have set up, and all of them passed. The problem is that none of those checks were looking at the same thing a real visitor was about to see.

This site is my portfolio, and I build it by directing AI against a process with written rules and automated checks before anything counts as done. One of those checks reviews that every page looks right, another that it's accessible, another that the code compiles without errors. The blog section passed every one of them.

And yet every post I published broke completely the moment a real person opened it. The page showed a generic error message. Not a single word of the article showed up.

What I took away

• An automated check only tests what it actually runs. If it runs in an environment more permissive than the real one, it can say everything is fine when in production it isn't.
• The easiest fix isn't always the right one, especially when it weakens something as important as security.
• Fixing a visible bug can uncover an invisible one that was living right next to it.

How my blog ended up showing a generic error

The blog's content is written in a format that mixes regular text with small pieces of code. It's called MDX. Before you can read it, something has to turn that format into what a browser understands.

My site has a security policy that blocks a specific technique for generating code on the fly inside the browser. Browsers call it eval, and it's a common door for attacks, so I block it on purpose. It turned out the piece that converted the blog's format into readable text used exactly that technique, inside every visitor's own browser.

Why my tests never caught it

Every one of my tests ran against a development version of the site, and that version does allow the blocked technique, because it needs it for something else: refreshing the page while I'm coding, without a full reload. Nobody decided that on purpose for the blog. It was an old assumption that never got checked again: that only that other thing needed the permission.

So my accessibility tests passed. My visual tests passed. The check that the page loads without errors passed. All of them ran against the version that has the permission, never against the real version a visitor sees.

Why not just allow that technique in production?

Because that security policy exists to block real attacks, and that specific technique is one of the most common doors they use. Removing that restriction site-wide, to fix one problem in the blog, would have left the door open everywhere else someone could try to inject something. The fix had to avoid the technique, not allow it.

A fix that weakens the whole site's security to solve one section's problem isn't a fix. It's debt you'll pay somewhere else.

Fixing this uncovered a second, hidden bug

The real solution was to move that conversion to the server, where the technique is allowed because it never runs in anyone's browser. The blog's text reaches the visitor already converted, ready to read.

That change fixed the main problem, but it uncovered a second one. One of the blog's code blocks had a button to copy it with a single click. That button lived in the part of the site that runs in the browser. Once I moved the conversion to the server, that piece stopped connecting correctly, and the button disappeared with no error at all, silently.

I found it by comparing the new version against the old one, line by line, not because anything warned me. I now have 2 tests dedicated exactly to this: one that checks that no part of the browser-side code tries to do this conversion, and another that opens a real post and confirms the copy button is still there.

If you've ever seen everything green and still had something break in production, you already know that feeling. The good part is that every check that failed silently, once actually fixed, becomes a check that will warn you next time. Thanks for reading this far. 🙂