Suffering from BUGS: How I Almost Deleted My Entire Project

I almost wiped a year of work because I trusted a flaky test suite and skipped the obvious step. The story has a happy ending, but it is not the kind of post you read for inspiration. It is the kind you read so you do not do what I did on a Tuesday night with two coffees in me.

What happened

I was refactoring a backend service that handled webhook ingestion (the server-side code that receives event notifications from external systems like payment providers or git hosts). The tests passed. I tagged a release. I went to bed. Six hours later a customer support thread woke me up. The new release was eating payloads in production. Not dropping them. Eating them. The data went in, the database said it was there, and then it was not.

The bug was a race condition (a flaw where two operations run at the same time and step on each other, producing unpredictable results) between two async workers that updated the same row. The unit tests missed it because they ran against an in-memory SQLite instance with a single thread. Production had Postgres with twenty worker processes.

I spent the next eight hours reverting, debugging, and staring at logs. At one point, somewhere around 3 AM, I opened a terminal, typed rm -rf build/, and sat there with my finger over Enter. I did not run it. But I will not pretend I was thinking clearly.

The setup that created the risk

The codebase was not large, maybe twelve thousand lines. The test suite was reasonable. We had integration tests, end-to-end tests, and a CI pipeline that ran all of them on every PR. What we did not have was a test environment that resembled production.

Here is the thing nobody wants to admit. Most of us ship with test setups that are dramatically simpler than the real system. The reasons are practical. Spinning up real Postgres, real Redis (an in-memory key-value store commonly used for caching and queues), and real workers for every CI run is expensive and slow. So we make do. We use SQLite. We use a single thread. We mock the queues. And we tell ourselves the integration tests cover the rest.

Sometimes they do. Often they do not.

What the actual problem was

The race condition lived in code I had written eighteen months earlier. It was a status update handler that did this:

1. Read the row from the database
2. Check the status
3. If eligible, do the work
4. Write the new status back

That is the textbook shape of a race condition. Two workers can both read at step 1, both see an eligible status at step 2, both do the work at step 3, and both write at step 4. Whichever writes last wins, and the other update is gone.

The fix was not subtle. Wrap the read-check-write in a database transaction with row-level locking. Update the status with a conditional WHERE clause. Add an integration test that actually runs the race scenario against real Postgres. Each of those steps is well known. I had just not done any of them, because the test suite said everything was fine.

What the test suite did not catch

The unit test for this handler used a mock database that returned whatever the test told it to. The integration test ran against a real database but in a single thread, with a single worker, and never exercised the concurrency path. The end-to-end test used the staging environment, which was a small Postgres instance with no parallel traffic.

Three layers of testing. Zero coverage of the actual failure mode. The bug was hiding in the gap between layers, where everyone assumed the other layer was checking.

This is a common shape. The unit test covers the logic. The integration test covers the components. The end-to-end test covers the user journey. Nobody covers the boundary between components when multiple instances are running. That boundary is where most of the production bugs I have shipped have lived.

How I fixed it

The fix had three parts. I will be specific because the abstract advice on race conditions is useless.

• Replaced the read-check-write pattern with a single SQL UPDATE that includes the status check in the WHERE clause. The database does the locking, not the application code.
• Added a row-level lock around the same path as a belt-and-suspenders measure. Postgres supports SELECT ... FOR UPDATE (a query that grabs an exclusive lock on the matching rows, blocking other writers until the current transaction commits). I used it in addition to the conditional update, because the cost is small and the safety margin is real.
• Wrote a new integration test that spawns fifty goroutines (lightweight threads used in Go, the language the service was written in) all hitting the same row at the same time and asserts that exactly one wins. The test runs against real Postgres, not SQLite, and it runs in CI on every PR.

The new test fails against the old code. That is the test I should have written eighteen months earlier.

What I would tell past me

If I could send a message back to the version of me that wrote the original handler, I would say three things.

• Do not trust a green CI to mean the system works. It means the tests passed. Those are not the same thing.
• When the data store is part of the bug surface, test against the data store you actually use, not the closest approximation.
• Race conditions are not exotic. They show up any time two workers touch the same row. Assume the worst and write the locking code first.

The last one is the hardest to internalize. I had read about race conditions. I knew they were common. I still did not write the locking code, because the simple version was working in the simple test environment. The simple version was not the production version. I learned that the expensive way.

Trade-offs

The new code is not free. Row-level locking serializes some operations that used to run in parallel, which means a small throughput hit on that specific handler. In practice the contention is rare and the hit is invisible in our metrics. The test suite is slower now, because the new concurrency test runs real Postgres for a few seconds. CI time went up by about twelve seconds per run. Both costs are worth it.

There is also a meta cost. Every developer who joins the team will have to learn this pattern, and there is a real risk that someone, on a Friday afternoon, will write a new handler that looks like the old one because they copy-pasted from a similar feature. Code review catches it most of the time. Not all of the time. That is a permanent tax on the team for as long as the codebase lives.

When this kind of bug shows up

Race conditions are most likely when you have any of these conditions.

• A web framework that handles requests concurrently and uses a database with row-level writes
• A queue or message bus with multiple consumer processes
• A cache layer that does not use atomic operations
• A scheduled job that runs on multiple instances of the same service

If you are building any of those patterns, the question is not whether you have race conditions. The question is whether your tests would catch them. Most test suites, including the one I had, would not.

What I am doing differently now

Three things have changed in how I work since that night.

First, every PR that touches a database handler gets a comment in the review asking “what happens if two of these run at the same time?” If the answer is anything other than a clear explanation of the locking strategy, the PR is not ready.

Second, we run a weekly chaos test (a deliberately aggressive test that pushes the system into failure states to verify it handles them) where a script fires concurrent requests at a few hot paths and checks the database for inconsistent state. It is ugly, it is not deterministic, and it has caught two more bugs in the last six months. The chaos test costs us about forty minutes of CI time per week. That is the cheapest insurance I have ever bought.

Third, the team has a shared doc called “near misses” that lists every bug we almost shipped and how we caught it. The race condition is in there. The doc is read by every new hire. It is the most useful piece of internal documentation we have.

The lesson I keep relearning

Software engineering is full of advice that sounds obvious in the abstract and is hard to apply in the moment. “Test against the real system” sounds obvious. Writing the test, getting the budget for the slower CI run, arguing for the test environment that costs more, those are the parts that take effort.

The bug that almost made me delete the project did not teach me anything new. I already knew the rules. What it taught me is that knowing the rules is not the same as following them under deadline pressure, and that the cost of not following them is paid by your customers, not by you.

That is the part I keep relearning. The part where the cost of skipping the boring step lands on someone else.