That's the question I now ask of every green check on agent-written code: which wrong version would it reject? For shared database state, you answer it by controlling when two workers read and write the same records. Pick the ordering that breaks the rule your data must keep, such as "both increments to a counter survive." That rule gives you a final value to assert, plus any error the caller has to handle.

Below, I'll reproduce two different failures on PostgreSQL 17 and MongoDB 8. A deadlock aborts a transaction loudly. A stale write loses an update silently. Then I'll compare four guards: atomic updates, consistent lock ordering, cooperative advisory locks and version checks. The examples are constructed, small enough to paste into two terminals, and every one ran on scratch servers. Each ends as a regression test you can keep.

Two Workers Are Enough to Expose the Gap

You don't need production traffic to make two operations overlap. Two workers touching one record are enough, and a small schedule lets you follow each step.

Say worker A processes records 41 and 42, while worker B processes 42 and 43. The jobs have different IDs, but both touch record 42. If a test runs A to completion before starting B, B sees A's finished changes. That test never asks what happens when B changes 42 between A's read and its dependent write.

Time runs downward. The diagram shows operation order, not measured durations.

At A's final arrow, its calculation depends on a value B has already changed. Whether the database accepts that write depends on the operation and its guards. Keep that read/write boundary in mind for the MongoDB section.

So check both the record IDs and the operation order. A test that gives every worker separate fixtures avoids the overlap entirely. A test with shared fixtures can still miss it if A finishes before B reaches the record. Starting two jobs at once leaves the interleaving to the scheduler, so neither setup guarantees the sequence in the diagram.

The timeline narrows the investigation to one question: can another worker change the state this operation still relies on? PostgreSQL adds a second failure to tell apart, where two transactions each hold a lock the other needs.

PostgreSQL: Follow the Lock Cycle

A deadlock starts with each transaction holding something the other needs. Here, two workers each increment two counters, but visit the rows in opposite order. Both operations must finish for each counter to reach 2. The SQL uses database-side increments, so there's no stale application calculation to blame.

Open two independent psql connections to the same disposable database. Run this setup once, outside a transaction, before either worker starts. I ran everything here on PostgreSQL 17.11 at READ COMMITTED.

CREATE TABLE race_counters (id integer PRIMARY KEY, n integer NOT NULL);
INSERT INTO race_counters VALUES (1, 0), (2, 0);

Run the numbered groups in order, switching terminals where indicated. Submit A's second update, then switch to B while A waits. Each transaction holds its first row lock until it ends.

-- 1. Session A
BEGIN ISOLATION LEVEL READ COMMITTED;
UPDATE race_counters SET n = n + 1 WHERE id = 1;

-- 2. Session B
BEGIN ISOLATION LEVEL READ COMMITTED;
UPDATE race_counters SET n = n + 1 WHERE id = 2;

-- 3. Session A: waits for B
UPDATE race_counters SET n = n + 1 WHERE id = 2;

-- 4. Session B: closes the cycle
UPDATE race_counters SET n = n + 1 WHERE id = 1;

A now needs row 2, held by B, and B needs row 1, held by A. Neither can finish without the other releasing a lock. After deadlock_timeout (one second by default), PostgreSQL detects the cycle and aborts one transaction. Don't depend on which one it picks. PostgreSQL's deadlock documentation describes this and recommends a consistent acquisition order.

In my run, A was the victim:

ERROR:  deadlock detected
DETAIL:  Process 872 waits for ShareLock on transaction 741; blocked by process 873.
Process 873 waits for ShareLock on transaction 740; blocked by process 872.

Run ROLLBACK; in the terminal reporting deadlock detected, and COMMIT; in the survivor. SELECT * FROM race_counters ORDER BY id; then shows 1 in both counters. The victim's first increment disappeared with its transaction. That's an explicit failure the caller must recover from, unlike two successful writes silently replacing each other's result.

Now change the acquisition order. With both transactions closed, reset with UPDATE race_counters SET n = 0;. Both workers use this same sequence:

BEGIN ISOLATION LEVEL READ COMMITTED;
UPDATE race_counters SET n = n + 1 WHERE id = 1;
UPDATE race_counters SET n = n + 1 WHERE id = 2;
COMMIT;

Pause A after its first update and start B. B waits on row 1 before it holds row 2, which leaves A free to finish. Complete A, then B, and both counters reach 2. That removes the cycle at the cost of waiting where the workers overlap. Apply the order across every write path that takes part, including any extra rows those paths touch.

Sorting IDs in application memory only helps if the database acquires the locks in that order. An UPDATE ... WHERE id IN (1, 2) doesn't promise to visit rows in the listed order.

Keep a bounded retry path for SQLSTATE 40P01 (deadlock_detected). Roll back and start a fresh transaction that reruns its reads, decisions and writes. The aborted transaction can't continue, and rerunning only its last statement in a new transaction drops the earlier work. PostgreSQL's retry guidance makes that whole-transaction boundary explicit and separates deadlocks from serialization failures (40001). Cap the attempts, add backoff with jitter and surface exhaustion to the caller. When I replayed the victim's whole transaction after the survivor committed, both counters reached 2. Repeated failures still mean you should look at the lock cycle.

Choose the Resource Your Lock Protects

A batch ID is a tempting advisory-lock key. But batches 101 and 102 can both modify tenant 42's records, and different keys let both workers proceed. Choose the resource that owns the invariant. If tenant 42 may hold at most two reservations, both workers must coordinate on tenant 42.

PostgreSQL's advisory locks are cooperative. Every writer that affects the invariant must take the agreed key before reading or changing the protected state, maintenance scripts included. An advisory lock doesn't lock a table, and it doesn't stop a writer that ignores the convention.

I use the key (7, 42): application namespace 7, tenant 42. The two-integer lock function lets you reserve a namespace for this operation. Run the setup once in a scratch database, then the transaction on one connection:

CREATE TABLE reservations (
    tenant_id integer NOT NULL,
    booking_id integer PRIMARY KEY
);
INSERT INTO reservations VALUES (42, 1);

BEGIN ISOLATION LEVEL READ COMMITTED;
SELECT pg_advisory_xact_lock(7, 42);
INSERT INTO reservations (tenant_id, booking_id)
SELECT 42, 2
WHERE (SELECT count(*) FROM reservations WHERE tenant_id = 42) < 2;
COMMIT;

Keep BEGIN, the lock, the capacity check and the insert in the same transaction on the same connection. pg_advisory_xact_lock releases at commit or rollback. In autocommit, a standalone lock call ends its protection before the next statement. A session-level pg_advisory_lock lives longer and survives rollback, which gets messy with pooled connections. The explicit READ COMMITTED gives the insert a fresh snapshot after any lock wait.

Pause worker A before its commit. On a second connection, worker B runs the same transaction with booking ID 3. B waits on (7, 42). After A commits, B sees two reservations and inserts nothing. Commit B to finish this case.

With both transactions closed, reset with TRUNCATE reservations; INSERT INTO reservations VALUES (42, 1);. Repeat A's transaction and pause before commit. Give B the key (7, 43) and booking ID 3, while its insert still targets tenant 42. B no longer waits. Let both inserts finish, then commit both transactions, and the count reaches three.

A tenant-wide key serializes participating work for that tenant. Keep the protected transaction short. A global key would make unrelated tenants wait too.

Use SKIP LOCKED when claiming any other available queue item satisfies the request. PostgreSQL explicitly allows an inconsistent view there. If your batch must update every required record, silently skipping busy rows changes the result you promised. It doesn't complete that batch safely.

MongoDB: An Atomic Write Can Still Replace Stale Data

MongoDB makes each single-document write atomic. A separate read, calculation and write still leaves room for another worker to change the document between those steps. MongoDB's atomicity documentation describes exactly this overwrite when both updates filter only on _id.

Two workers each want to add one to a counter that starts at 10. Both read 10, both calculate 11, and both successfully store 11. The missing increment never produces a database error.

You need a disposable MongoDB server and the mongosh shell. I ran these on MongoDB 8.0.16 with mongosh 2.5.9. Open mongosh --nodb and paste the blocks in order into the same shell. Set uri to your scratch server's address (27118 is just my test port, not MongoDB's default). A and B use independent connections, and the script completes both reads before either write. The reset replaces only this demo's document in race_demo.counters.

const uri = "mongodb://127.0.0.1:27118"
const A = new Mongo(uri).getDB("race_demo").counters
const B = new Mongo(uri).getDB("race_demo").counters
const reset = () =>
  A.replaceOne(
    { _id: "demo" },
    { _id: "demo", count: 10, version: 0 },
    { upsert: true },
  )
reset()
let a = A.findOne({ _id: "demo" })
let b = B.findOne({ _id: "demo" })
A.updateOne({ _id: a._id }, { $set: { count: a.count + 1 } })
B.updateOne({ _id: b._id }, { $set: { count: b.count + 1 } })
A.findOne({ _id: "demo" }) // count: 11

Both updates report matchedCount: 1, so nothing looks wrong. Each $set replaces a field with a value the application computed. Replacing the whole document from a stale snapshot can overwrite other fields too. For this counter, express the increment directly with $inc, so MongoDB applies it to the stored value:

reset()
A.updateOne({ _id: "demo" }, { $inc: { count: 1, version: 1 } })
B.updateOne({ _id: "demo" }, { $inc: { count: 1, version: 1 } })
A.findOne({ _id: "demo" }) // count: 12, version: 2

When the calculation has to happen in your application from fields it read earlier, use a version check to reject stale input. Put the snapshot's version in the filter, then increment that version in the same write. I kept the counter arithmetic so the difference stays visible without extra business logic:

reset()
a = A.findOne({ _id: "demo" })
b = B.findOne({ _id: "demo" })
const save = (collection, snapshot) =>
  collection.updateOne(
    { _id: snapshot._id, version: snapshot.version },
    { $set: { count: snapshot.count + 1 }, $inc: { version: 1 } },
  )
save(A, a).matchedCount // 1
save(B, b).matchedCount // 0: stale version rejected

The first write moves the version to 1. The second write matches nothing and changes nothing. Check matchedCount yourself, because a zero match doesn't throw a concurrency exception. With this filter, zero can also mean the document was deleted.

On a version conflict, reread the document and recompute from the fresh state. When I reread B's snapshot and saved again, it matched and the count reached 12 at version 2. For an automatic retry, cap the attempts (three is a reasonable start), then return a conflict if contention continues. An edit that needs a user's judgment should surface the conflict instead. Retrying the same stale value with a newer version would overwrite the change you just detected.

Every writer that changes fields used by the calculation must advance version, maintenance scripts included. Keep upsert disabled for the guarded write. A missing match is a conflict to handle.

The invariant here covers one document. A version check on this counter can't protect a calculation whose inputs also live in other documents.

Make the Interleaving a Regression Test

Automate the terminal sequences with two independent connections and a shared committed fixture. Record the server version and transaction settings. Signal completed reads and lock acquisitions before releasing the next operation. Confirm blocked PostgreSQL requests from an observer connection with pg_stat_activity or pg_blocking_pids. A sleep can't prove that wait.

These schedules make good test specifications:

1. PostgreSQL deadlock. Reset both counters to 0. A locks row 1, B locks row 2, then dispatch both second-lock requests without awaiting either. Expect one transaction to report 40P01, the survivor to commit, and both counters at 1.
2. Ordered locks. Reset to 0. A locks row 1, B requests row 1. Confirm B waits, then let A finish and commit. Expect B to complete afterward with both counters at 2.
3. Advisory key. Reset to one reservation. A inserts under (7, 42), and B requests that key before checking capacity. Expect B to wait and the count to stay at 2. Repeat with B on (7, 43): both insert, and the count reaches 3.
4. MongoDB stale write. Reset to count 10, version 0. Both read, A writes 11, then B writes its cached 11 after A's acknowledgement. Expect both writes to match and a final count of 11.
5. Atomic increment. Reset to count 10, version 0. Each connection runs the $inc update. Expect count 12 and version 2.
6. Version guard. Reset to count 10, version 0. Both read, A saves, B tries its stale snapshot. Expect match counts of 1 then 0, count 11 and version 1.

Add recovery assertions to the failure cases. For the deadlock, run the whole-transaction retry after the survivor commits. Both counters must reach 2 no matter which worker failed. For MongoDB, exercise reread and recompute if that's your contract. The result must be count 12 at version 2, and a swallowed conflict must fail the test.

Give every wait a deadline and clean up both connections on failure. With ordered locks, avoid a barrier that needs both workers to hold row 1, since the guard prevents exactly that. Keep the reproduced case next to your legacy-code regression tests and run it in CI against a disposable database. Passing it covers this schedule, not every possible interleaving.

Test One Write at a Time

Choose one write in your application: a counter increment, a balance change, or a document replaced from an earlier read. Give it a specific correctness rule before choosing a guard. For a counter, the rule might be "two accepted increments both appear in the final value."

Then turn that write into a regression test:

• Inventory the writers. Name the request handlers, background jobs and maintenance scripts that can change the same records.
• State the invariant. Write down the expected final state and which conflicts or aborts the caller must handle.
• Force the overlap. Use two independent sessions and a controlled schedule that exposes the unguarded failure.
• Verify the guard. Repeat the test after the change. Assert the final state and the expected errors, including the outcome after any required retry.

This covers PostgreSQL and MongoDB operations only. Redis, Elasticsearch, Celery delivery and external side effects bring their own interleavings, and these examples don't prove anything about them.

If an agent wrote the code, this test is the one its green suite most likely lacks. Keep the failing schedule next to the guard, so a future edit has a concrete behavior to preserve.

More on this on potapov.dev:

• Using Claude Code Without Technical Debt
• Detecting AI Slop: A Code Review Checklist
• When to Trust AI Agent Tests and Reviews
• Testing Untested Legacy Code: A Practitioner's Guide
• Setting Up CI/CD When You Have None

This post is the hands-on part. I've told readers before to check that a test fails when you break the code, most recently in the post on agents declaring victory. I never showed how. So here's the procedure on a bug small enough to hold in your head: five lines of Python, four checks, real output from every command, and a table of which check catches which false pass.

Your new test is an oracle for one bug, and the only question is whether it can tell the buggy code from the fixed code. Each of the four checks catches a false pass the others miss:

1. Fail on the base, for the right reason. Catches the vacuous test, and filters out tests that go red for an unrelated reason.

2. Break the repair. Catches a test that still passes when the fix is subtly wrong.

3. A preservation probe. Catches the collateral regression the fix's own test never looks at.

4. An oracle the test's author didn't write. Catches a test and code that share one wrong assumption.

The Fixture: Five Lines and an Off-by-One

The demo is chunk(items, size), which splits a list into lists of size. The base version drops the trailing partial chunk: chunk([1, 2, 3, 4, 5], 2) returns [[1, 2], [3, 4]].

def chunk(items, size):
    """Split items into lists of `size`; the last list may be shorter."""
    if size <= 0:
        raise ValueError("size must be positive")
    return [items[i:i + size] for i in range(0, len(items) - size + 1, size)]

chunk([1, 2, 3, 4], 2) returns [[1, 2], [3, 4]] on both versions, so a test that only uses exact multiples passes the bug and the fix alike. That's what bothers me about this bug: nothing in such a test is wrong except the numbers it picked. The repair changes one range() bound:

-    return [items[i:i + size] for i in range(0, len(items) - size + 1, size)]
+    return [items[i:i + size] for i in range(0, len(items), size)]

The whole fixture is a tiny git repo you can rebuild in a few minutes. The code above is src/chunks.py. Commit it and tag it base, apply the diff on main and tag that repair. pyproject.toml needs two sections:

[tool.pytest.ini_options]
pythonpath = ["src"]

[tool.mutmut]
source_paths = ["src/"]
pytest_add_cli_args_test_selection = ["tests/test_bug.py"]

Two test files go in with the repair. tests/test_bug.py is the test under suspicion, with one test, test_keeps_the_partial_last_chunk, asserting chunk([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4], [5]]. tests/test_vacuous.py holds two tests an agent might write: test_chunk_runs calls chunk() and asserts nothing, and test_chunk_returns_something asserts the result is truthy.

A wrong-reason tag, one commit on top of repair, adds a chunk_count(items, size) helper that returns len(chunk(items, size)), plus two tests that both assert chunk_count([1, 2, 3, 4, 5], 2) == 3. tests/test_count.py::test_counts_the_partial_chunk imports chunk_count at the top of the file. tests/test_count_runtime.py::test_counts_at_runtime imports it inside the test function. That difference matters, as you'll see. The probe files for Check 3 and the two scripts get added on main as you reach them.

You need Python 3.12 or later (Check 4 uses itertools.batched), pytest and git. mutmut is optional and POSIX-only, so on Windows run it in WSL. The sed -i commands below are GNU sed. On macOS write sed -i ''. Every output below came from Python 3.12.15, pytest 9.1.1 and mutmut 3.8.0, run in this order on main. The output blocks keep the pytest summary lines and drop tracebacks.

python3.12 -m venv .venv && . .venv/bin/activate
pip install pytest==9.1.1 mutmut==3.8.0
export PYTHONDONTWRITEBYTECODE=1    # before the first run, in this shell

Set that variable before anything runs. It stops Python writing new .pyc files, but it doesn't stop Python loading old ones, and a same-length edit made within the same second as the last run can execute the previous code. In an existing checkout, delete the caches first: find src tests -name __pycache__ -prune -exec rm -rf {} +. A .gitignore keeps caches out of commits. It doesn't disable them.

Check 1: Fail on the Base, for the Right Reason

The key move is tests from the repair, source from the base. Check out the whole base commit and you run the base's old tests, which say nothing about the new one. (Benchmarks call the code before the fix the base and the reference fix the gold patch. SWE-bench sorts tests into FAIL_TO_PASS and PASS_TO_PASS.)

check_test.sh does the swap in a throwaway git worktree, for one test node. It matches the node id literally, so quote a parametrized one: ./check_test.sh base repair 'tests/test_x.py::test_partial[5]'.

#!/usr/bin/env bash
# Usage: ./check_test.sh BASE_REF REPAIR_REF TEST_NODE_ID
# Runs one test from REPAIR_REF against BASE_REF's source, then REPAIR_REF's.
set -euo pipefail
base=$1 repair=$2 node=$3
export PYTHONDONTWRITEBYTECODE=1
wt=$(mktemp -d); trap 'git worktree remove --force "$wt"' EXIT
git worktree add -q --detach "$wt" "$repair"
run() { (cd "$wt" && pytest -q -p no:cacheprovider -rA --tb=line "$node" 2>&1) | sed "s|$wt|<wt>|g"; }
git -C "$wt" checkout -q --no-overlay "$base" -- src/     # base source, exact tree
b=$(run) && bcode=0 || bcode=$?
git -C "$wt" checkout -q --no-overlay "$repair" -- src/
r=$(run) && rcode=0 || rcode=$?
echo "--- base (exit $bcode)";   grep -E '^(FAILED|ERROR|PASSED|SKIPPED|E )' <<<"$b" || true
echo "--- repair (exit $rcode)"; grep -E '^(FAILED|ERROR|PASSED|SKIPPED)' <<<"$r" || true
has() { awk -v l="$1" 'index($0, l " - ") == 1 || $0 == l { f = 1 } END { exit !f }'; }  # literal id
if [ $bcode -eq 1 ] && has "FAILED $node" <<<"$b" && ! grep -q '^ERROR' <<<"$b" \
   && [ $rcode -eq 0 ] && has "PASSED $node" <<<"$r"; then
  echo "CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?"
else
  echo "REJECT: this run does not show the test catching the bug"; exit 1
fi

--no-overlay is there because a plain git checkout <ref> -- src/ only adds and overwrites files. On a test repo where the repair added one source file and deleted another, the "base" run without the flag still had the repair's new file, and the swap back left the base's deleted one behind. Any git failure, like a mistyped ref, aborts the script before a verdict.

From the repo root, on main:

$ ./check_test.sh base repair tests/test_bug.py::test_keeps_the_partial_last_chunk
--- base (exit 1)
E   assert [[1, 2], [3, 4]] == [[1, 2], [3, 4], [5]]
FAILED tests/test_bug.py::test_keeps_the_partial_last_chunk - assert [[1, 2],...
--- repair (exit 0)
PASSED tests/test_bug.py::test_keeps_the_partial_last_chunk
CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?
$ ./check_test.sh base repair tests/test_vacuous.py::test_chunk_runs
--- base (exit 0)
PASSED tests/test_vacuous.py::test_chunk_runs
--- repair (exit 0)
PASSED tests/test_vacuous.py::test_chunk_runs
REJECT: this run does not show the test catching the bug
$ ./check_test.sh base wrong-reason tests/test_count.py::test_counts_the_partial_chunk
--- base (exit 4)
ERROR: found no collectors for <wt>/tests/test_count.py::test_counts_the_partial_chunk
E   ImportError: cannot import name 'chunk_count' from 'chunks' (<wt>/src/chunks.py)
ERROR tests/test_count.py
--- repair (exit 0)
PASSED tests/test_count.py::test_counts_the_partial_chunk
REJECT: this run does not show the test catching the bug
$ ./check_test.sh base wrong-reason tests/test_count_runtime.py::test_counts_at_runtime
--- base (exit 1)
E   ImportError: cannot import name 'chunk_count' from 'chunks' (<wt>/src/chunks.py)
FAILED tests/test_count_runtime.py::test_counts_at_runtime - ImportError: can...
--- repair (exit 0)
PASSED tests/test_count_runtime.py::test_counts_at_runtime
CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?

The last two are the trap. Both tests need chunk_count, which the base doesn't have. The first imports it at the top of the file, so the base run dies during collection and the script rejects it. The second imports it inside the test function, so it fails at run time with exit 1 and a FAILED line, and the script calls it a candidate. It looks red. It proves nothing.

So the script narrows, it doesn't prove. It filters out what an exit code can show (a pass on the base, a collection error, a skip, a repair that doesn't pass), then hands you the base failure to read. The first run's E line is the missing [5], which is the bug. The fourth run's is an ImportError, which isn't.

Drop pythonpath from pyproject.toml and pytest -q tests/ stops on 6 collection errors with exit 2, on the base too. A check that only asks "did it fail on base?" would wave every test through.

One more hole: a test that returns its comparison instead of asserting it. With return chunk([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4]] in tests/test_return.py, pytest -q tests/test_return.py prints 1 passed, 1 warning, though the comparison is False. Add -W error::pytest.PytestReturnNotNoneWarning and it prints 1 failed.

The other half of the check matters as much. In my experiments, the first diagnostic probe for a Click regression bundled two properties, and it failed the upstream fix because of a separate, known bug. That made it invalid, and it was thrown out. A test that can't pass the real fix is measuring something else.

A team I work with hit the base-side failure in July. A regression test's setup rebuilt a database view with CREATE ... IF NOT EXISTS, silently reused a stale view from earlier work, and passed whatever the new SQL said. Running it once against the unfixed code would have shown it green. In September that team adopted OpenClaw's public test-audit skill, and its authoring gate puts it best: "A regression test that never demonstrably failed proves the mock, not the fix."

Check 2: Break the Repair and Watch the Test Go Red

A mutant is a copy of the code with one small deliberate change. If a test fails, the mutant is killed. If every test passes, it survived, and you've found something your tests can't see. Check 1 shows the test sees the original bug. Check 2 asks whether it would see a slightly wrong fix.

Start by hand. Read the repair diff and write the mistake an agent would most plausibly make in that exact line. Here the fix is a loop bound, so the obvious wrong fix is a different off-by-one in the same bound. That's mutant A, applied to the repaired source on main and then undone:

$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py
$ pytest -q tests/test_bug.py tests/test_vacuous.py
FAILED tests/test_bug.py::test_keeps_the_partial_last_chunk - assert [[1, 2],...
1 failed, 2 passed in 0.01s
$ git checkout main -- src/

The bug test kills it. Both vacuous tests pass it. The zero-tool version is a source rollback: revert one changed source file, and the test must fail. In my experiments, three of eight hand-made mutants that reverted urllib3 and Click code to obsolete API contracts passed the narrow feature test, and only the broader regression module caught them.

For a PR, that hand-written mutant of the changed line is the default. A tool sweep is broader, optional work. It costs one test run per mutant plus a manual read of every survivor, so skip it when that's more than the PR is worth.

I hadn't used a mutation tool before this experiment. Here's what mutmut found on the fixture. Its config is the two keys in pyproject.toml above, and older blog posts still show the deprecated paths_to_mutate and tests_dir. A small sweep.sh writes the test list into the second key, deletes mutants/, runs mutmut run and lists survivors:

#!/usr/bin/env bash
# Usage: ./sweep.sh TEST_PATH...
# Sets mutmut's test selection in pyproject.toml, runs a fresh sweep, lists survivors.
set -euo pipefail
sel=$(printf '"%s", ' "$@"); sel="[${sel%, }]"
sed -i.bak "s|^pytest_add_cli_args_test_selection = .*|pytest_add_cli_args_test_selection = $sel|" pyproject.toml
rm -f pyproject.toml.bak && rm -rf mutants
mutmut run > mutmut.log 2>&1 || { cat mutmut.log; exit 1; }
tr '\r' '\n' < mutmut.log | grep '🎉' | tail -1
mutmut results

Still on main, with the repair restored (🎉 killed, 🙁 survived):

$ ./sweep.sh tests/test_vacuous.py
13/13  🎉 4 🫥 0  ⏰ 0  🤔 0  🙁 9  🔇 0  🧙 0
$ ./sweep.sh tests/test_bug.py
13/13  🎉 8 🫥 0  ⏰ 0  🤔 0  🙁 5  🔇 0  🧙 0

The bug test alone left five survivors: three mangled error messages and two changes to the guard. One of them, from mutmut show chunks.x_chunk__mutmut_2:

-    if size <= 0:
+    if size <= 1:

Nothing calls chunk() with size=1, so nothing notices. The bug test was never meant to guard that line, and that's Check 3's job.

What the sweep missed matters as much. None of its 13 mutants shifted the bound the way mutant A does, and none deleted the guard. Its operators swap + for -, 0 for 1, values for None. The two mutants I'd most expect from an agent came from reading the diff, not from the tool.

After you add a test for a survivor, re-run the sweep. I've watched an agent publish a mutation table it never re-ran.

Know the limits. mutmut run exited 0 with 9 survivors, and cosmic-ray exec exited 0 with 16 of 21. Stryker fails a build only when thresholds.break is set, and its default is null. mutmut 3 doesn't mutate module-level code, so a wrong constant is invisible to it. Some survivors are equivalent mutants that no test can kill, so I'd read survivors one by one and skip the score. Google's 2018 report describes the same idea at scale: mutants only on the lines a change touches, shown during code review.

Check 3: A Preservation Probe Has to Pass Twice

A preservation probe tests behavior the change must not touch. It passes on the base, passes on the repair, and fails a mutant that breaks the preserved behavior. That's the mirror image of the bug test. A probe that fails the upstream fix is invalid, like the bundled probe in Check 1. A probe that fails the base isn't a preservation probe at all: it's a second bug test.

Why bother, when the project already has tests? Because a fix's own test only looks at the line that changed, and an agent's patch rarely stops there. It renames, simplifies a guard, reorders arguments. The two probes behind those 8 rejections in my experiments were exactly this kind: one checked that a Click multiple-value callback still received its collected arguments, the other that a zero-byte urllib3 read didn't stop a later full read from being cached.

On the fixture, the probe guards the size check, and it can be vacuous too. Mutant B deletes the guard. The weakest probe, tests/test_preserve_typeonly.py, only checks pytest.raises(ValueError) at size=0:

$ sed -i '/if size <= 0:/,+1d' src/chunks.py
$ pytest -q tests/test_preserve_typeonly.py
1 passed in 0.00s
$ git checkout main -- src/

It passes because range() raises ValueError for a zero step on its own. At size=-1 there's no error at all: range(0, 3, -1) is empty, and the guard-less chunk() returns []. OpenClaw's skill lists this exact junk pattern: "negative controls that pass for an unrelated reason".

tests/test_preserve_first.py adds match="size must be positive" and an exact-multiple case, chunk([1, 2, 3, 4], 2) == [[1, 2], [3, 4]]. That kills mutant B at size=0, and the sweep finds the next gap:

$ ./sweep.sh tests/test_bug.py tests/test_preserve_first.py
13/13  🎉 11 🫥 0  ⏰ 0  🤔 0  🙁 2  🔇 0  🧙 0
    chunks.x_chunk__mutmut_2: survived
    chunks.x_chunk__mutmut_4: survived

The first survivor is the size <= 1 guard from Check 2. The second raises "XXsize must be positiveXX". The pytest docs say match "is matched with the re.search() function", so the mangled message still matched. Neither trap is exotic. Both are a probe leaning on something other than the behavior it names.

The final probe, tests/test_preserve.py, probes -1, anchors the pattern and covers size=1:

import pytest
from chunks import chunk

@pytest.mark.parametrize("size", [0, -1])
def test_rejects_non_positive_size(size):
    with pytest.raises(ValueError, match=r"^size must be positive$"):
        chunk([1, 2, 3], size)

def test_size_one_is_allowed():
    assert chunk([1, 2, 3], 1) == [[1], [2], [3]]

def test_exact_multiple_unchanged():
    assert chunk([1, 2, 3, 4], 2) == [[1, 2], [3, 4]]

Pass on the base, pass on the repair, fail on mutant B, then mutant A and the sweep (summary lines only):

$ git checkout --no-overlay base -- src/ && pytest -q tests/test_preserve.py; git checkout main -- src/
4 passed in 0.00s
$ pytest -q tests/test_preserve.py
4 passed in 0.00s
$ sed -i '/if size <= 0:/,+1d' src/chunks.py && pytest -q tests/test_preserve.py; git checkout main -- src/
FAILED tests/test_preserve.py::test_rejects_non_positive_size[0] - AssertionE...
FAILED tests/test_preserve.py::test_rejects_non_positive_size[-1] - Failed: D...
2 failed, 2 passed in 0.01s
$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py && pytest -q tests/test_preserve.py; git checkout main -- src/
FAILED tests/test_preserve.py::test_size_one_is_allowed - assert [[1], [2]] =...
1 failed, 3 passed in 0.01s
$ ./sweep.sh tests/test_bug.py tests/test_preserve.py
13/13  🎉 13 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0  🧙 0

The probe also catches mutant A, through its size=1 case: chunk([1, 2, 3], 1) loses the [3]. The probe wasn't aimed at mutant A. That's the bonus of pinning plain boundary behavior: it catches more than the regression you had in mind.

On an agent PR, I'd pick probes from what the changed function promised before the change: its error cases, its boundary inputs, the flags or state it leaves behind for the next caller. One assertion per promise, run on the base first. If it fails there, you've misread the old behavior, and that's worth knowing before you judge the patch.

Check 4: An Oracle the Test's Author Didn't Write

A test and its implementation can agree because both misread the issue. Then Checks 1 through 3 all pass, and both are wrong. That's the risk when one agent writes the code and the expected values from the same reading: as Kent Beck puts it, copying computed values into expectations "defeats double checking".

On the fixture, the standard library supplies an independent oracle: itertools.batched already splits an iterable into batches where "the last batch may be shorter than n". oracle.py sits in the repo root:

"""Independent oracle: compare chunk() to itertools.batched on random inputs.
Exit 0 = agrees everywhere, 1 = first disagreement."""
import itertools, random, sys
sys.path.insert(0, "src")
from chunks import chunk

rng = random.Random(0)
for _ in range(2000):
    items = list(range(rng.randrange(0, 12)))
    size = rng.randrange(1, 6)
    want = [list(b) for b in itertools.batched(items, size)]
    if (got := chunk(items, size)) != want:
        sys.exit(f"DISAGREE chunk({items}, {size}) = {got}, expected {want}")
print("oracle: 2000 random cases agree")

On the repair, the base and mutant A:

$ python oracle.py
oracle: 2000 random cases agree
$ git checkout --no-overlay base -- src/ && python oracle.py; git checkout main -- src/
DISAGREE chunk([0, 1, 2, 3, 4, 5], 4) = [[0, 1, 2, 3]], expected [[0, 1, 2, 3], [4, 5]]
$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py && python oracle.py; git checkout main -- src/
DISAGREE chunk([0, 1, 2, 3, 4, 5, 6], 3) = [[0, 1, 2], [3, 4, 5]], expected [[0, 1, 2], [3, 4, 5], [6]]

The oracle passes Check 1 itself: it disagrees on the base and agrees on the repair. It also found its own counterexamples, [0, 1, 2, 3, 4, 5] with size 4, not the five-item list the bug test uses. The seed is fixed, so a disagreement reproduces every time.

Most PRs have no batched to lean on. Then I'd start smaller: before opening the patch, write two input and output pairs from the issue text, so the agent's code can't shape what you expect. When the issue has no examples, a reference implementation (a slow brute-force version, or the old code on inputs the change shouldn't affect) or one recorded real input from the system that produces it does the same job.

In June a webhook relay my agents built passed its tests, its review rounds and CI while silently dropping real comments: the code and its tests were written against the same invented webhook shape, and the real producer sent a flat one. Logging one real payload exposed it.

What I Ask of a Test Before I Trust It

Here's the whole map:

Two rows lean on Check 1, which is why it comes first. The last row is the only one that running the agent's own test more carefully can never fix, because the mistake sits in what the test expects.

The bar I set for my own graders is the bar I'd set for any test offered as evidence: it fails on the base for an assertion, passes on the fix, and goes red when a changed source file is reverted. My grading harness has since adopted one more rule: each task's hidden tests must also reject at least two targeted wrong patches. The fixture already has two good kinds. Mutant A is a partial repair and mutant B a regression, and Checks 2 and 3 are that rule in its cheapest form.

If you only run one check on an agent's test, run Check 1. It's one script and two pytest runs, and it filters the two failures in the table that look most like success. Build the five-line fixture, run ./check_test.sh base repair tests/test_bug.py::test_keeps_the_partial_last_chunk, and then point it at the next agent PR that says the tests pass.

When an agent's PR says "tests pass", which of these checks do you actually run before merging, and on which PRs do you skip them?

More on this on potapov.dev:

• When to Trust AI Agent Tests and Reviews

• Why AI Agents Declare Victory Too Early

The rewrite debate in software engineering is over 25 years old. Joel Spolsky wrote his famous "Things You Should Never Do" essay in 2000. Fred Brooks described the second-system trap in 1975. The evidence has been accumulating for decades - and companies keep making the same mistake.

I recently wrote about my own experience with this decision across ad-tech monoliths and govtech platforms. That was the personal view - war stories and a practitioner's framework. This is the industry view. What happens when you zoom out and look at every major rewrite and migration attempt over the past 25 years? The pattern is remarkably consistent. The failures share the same traits. The successes share different ones. And the line between them is clearer than most teams realize when they're standing at the crossroads.

The Graveyard

The failures share a pattern so consistent it's almost formulaic. Big-bang replacement, feature freeze on the old system, a timeline that doubles and then doubles again, and competitors gaining ground while your team rebuilds what already existed.

Netscape: The Canonical Disaster

In 1998, Netscape decided to rewrite their browser from scratch. Navigator 4.0 was messy, sure - years of accumulated code, rushed releases, the usual. But it worked. It had market share. And then they threw it away.

Version 5.0 never shipped. Three years of development produced a buggy, feature-incomplete version 6.0 while Internet Explorer went from afterthought to market dominance. Lou Montulli, one of Navigator's five original engineers, wrote to Joel Spolsky confirming: "I agree completely, it's one of the major reasons I resigned from Netscape." The rewrite didn't just fail technically. It killed the company.

Spolsky turned this into his 2000 essay calling a from-scratch rewrite "the single worst strategic mistake that any software company can make." Twenty-five years later, people keep proving him right.

Borland: The Same Mistake, Twice

Borland managed to repeat the pattern on two products simultaneously. Their dBase for Windows rewrite took so long that Microsoft Access launched first and ate the market. Their Quattro Pro rewrite shipped with significantly fewer features than the DOS version it replaced. Two products, same company, same mistake, same result - competitors shipped while Borland rebuilt.

Chandler: Six Years and Nothing to Show

Mitch Kapor - the guy who created Lotus 1-2-3 - poured millions of his own money into Chandler, a personal information manager that was going to be the next big thing. Development started in 2002. By 2008, after six years and an entire book documenting the project's dysfunction ("Dreaming in Code" by Scott Rosenberg), the team had produced a barely usable preview release. Meanwhile, Google Calendar launched for free and solved the same problem.

Kapor had the resources, the reputation, and the talent. What he didn't have was scope discipline. The project kept expanding - email, calendaring, task management, sharing, synchronization - and the clean-slate architecture meant every feature had to be built from zero.

The Pattern

Every one of these failures shares the same traits. A big-bang approach with no incremental value delivery along the way. Development frozen on the old system while the new one crawls toward feature parity. Timelines that blow past estimates by 2-3x. Institutional knowledge embedded in the old code - years of bug fixes, edge-case handling, adaptations to real-world conditions - quietly discarded and then painfully rediscovered. And competitors, unburdened by a rewrite, shipping features and gaining ground the entire time.

The pattern is so reliable that the interesting question isn't why rewrites fail. It's why companies keep attempting them despite the evidence.

The Survivors

The companies that successfully modernized their systems did something the failures didn't. They kept shipping. Every one of them found a way to replace pieces of the old system while the whole thing stayed running and users kept getting value.

Twitter: Five Years, One Component at a Time

Twitter's Ruby on Rails backend was buckling under scale by 2008. The Fail Whale - that cartoon error page - became a meme because it appeared so often. But Twitter didn't shut down and rebuild. They replaced components one at a time over five years.

Message queues first. Then storage. Then search - replacing the Ruby search frontend with a Java server dropped search latencies by 3x. Then the frontend. Each phase delivered measurable improvements while the service continued operating. The Fail Whale didn't disappear in a dramatic cutover. It faded out gradually as each bottleneck was addressed.

By 2013, the migration from Ruby to Scala/JVM was essentially complete. No feature freeze. No lost market share. No three-year gap where competitors could catch up.

Facebook: They Didn't Rewrite PHP - They Evolved the Language

Facebook's approach was even more radical. Instead of migrating away from PHP, they changed what PHP was.

In 2009, HipHop compiled PHP to C++ and doubled performance. In 2010-2013, HHVM replaced it with a virtual machine that compiled PHP on the fly for even better performance. In 2014, Hack introduced gradual typing that allowed file-by-file migration from PHP to a typed language. The entire codebase was converted to Hack "thanks to both organic adoption and a number of homegrown refactoring tools" - all while Facebook continued shipping features to billions of users.

Nobody on the team ever had to stop delivering value to rewrite infrastructure. The language evolved underneath them.

Shopify: The Modular Monolith

When Shopify's Ruby on Rails codebase hit 2.8 million lines with over 1,000 developers working on it, the conventional wisdom said: decompose into microservices. They went a different direction.

Instead of breaking the monolith into dozens of independently deployed services (with all the operational complexity that brings), they enforced boundaries within the monolith using their open-source tool Packwerk. Business domains got clear interfaces. Code stayed in one deployable unit. The result handles 32+ million requests per minute and hundreds of deployments daily - without the distributed systems headaches that microservices introduce.

Shopify's bet was that the monolith wasn't the problem. The lack of boundaries was the problem. Fix the boundaries, keep the monolith.

Stripe: Every API Version Still Works

Stripe has maintained backward compatibility with every API version since 2011. Think about that - over a decade of API versions, all simultaneously supported.

When they needed to fundamentally redesign their payments model, a small team built the PaymentIntents API from first principles between 2017 and 2019. Then migration bridges connected old and new abstractions. Per-request version pinning with compatibility modules meant no customer was forced to migrate on Stripe's schedule.

The compatibility layers are ugly. They're also what lets Stripe ship fundamental changes without breaking millions of integrations.

The Pattern

The contrast with the failures is stark. Where Netscape stopped everything for three years, Twitter replaced pieces over five. Where Chandler tried to build everything from scratch, Facebook evolved what they had. Where conventional wisdom said "decompose," Shopify said "add boundaries."

Every successful migration shares these traits: incremental replacement with continuous value delivery, preserved institutional knowledge (nobody threw away working code and started guessing), no competitive window for rivals, reversibility at each step, and compatibility layers enabling old and new to coexist.

The strategy has a name. Martin Fowler called it the Strangler Fig pattern in 2004, after watching rainforest vines in Queensland that germinate in a tree, grow around it, and eventually replace the host entirely. Grow the new alongside the old. Redirect behavior gradually. Decommission the legacy only when the new system has fully taken over.

The Second-System Trap

Fred Brooks explained in 1975 why rewrites tend toward bloat, and nobody has improved on his explanation since.

During the lifetime of a first system, designers accumulate a wish list. Features they couldn't ship because of deadlines. Architectural choices they regret. Shortcuts they had to take. When the opportunity for a rewrite comes, they dump every deferred wish into the second system. "This second is the most dangerous system a man ever designs."

This is different from normal scope creep. A greenfield product can cut scope ruthlessly - you're building something new, and "we'll add that in v2" is a legitimate answer. A rewrite can't cut scope the same way because it has a floor: everything the old system already does. Users expect at least feature parity. Then the team adds "and while we're at it, let's also improve..." on top of that floor. The scope balloons in a way that's unique to rewrites.

The Chandler project from the graveyard section is the purest example of this mechanism. Each new capability was added because the team was already rebuilding everything - why not do it right this time? The scope expanded from a simple organizer to a full communication suite, and "doing it right" consumed the entire budget.

Brooks recommended making resource costs visible per feature, resisting what he called "functional ornamentation," and ensuring architectural leadership from people who had designed at least three comparable systems. Fifty years later, this advice remains underused.

The Playbook

Four people shaped how the industry thinks about this problem. They all arrived at the same conclusion from different directions.

Joel Spolsky made the business case in 2000. His core insight: "It's harder to read code than to write it." Developers consistently overestimate how messy old code is and underestimate the cost of reproducing its behavior. Those ugly lines - the weird edge cases, the inexplicable conditionals, the comments that say "don't touch this" - are usually hard-won solutions to real problems. Throwing them away means rediscovering every one of those problems the hard way.

Martin Fowler provided the architectural strategy. His Strangler Fig pattern - the approach every successful migration in this article used - gives teams a concrete method: build new functionality alongside the legacy system, incrementally redirect behavior from old to new, and eventually decommission the original. Fowler emphasizes the necessity of transitional architecture - temporary code that allows old and new to coexist. It looks wasteful. It dramatically reduces risk.

Michael Feathers wrote the practitioner's toolkit. His definition of legacy code - "code without tests" - reframes the entire problem. What makes code dangerous to change isn't age or ugliness. It's the absence of automated verification. His characterization tests capture what code actually does (not what it should do), creating a safety net for refactoring. His Legacy Code Change Algorithm gives teams a step-by-step process: identify change points, find test points, break dependencies, write tests, make changes. The whole approach is fundamentally anti-rewrite.

Kent Beck crystallized it in 2012: "For each desired change, make the change easy (warning: this may be hard), then make the easy change." Rather than fighting existing code structure, reshape it incrementally to accommodate what you need. His "two hats" rule - you're either adding functionality or restructuring code, never both simultaneously - prevents the scope confusion that turns refactoring into accidental rewrites.

The common thread across all four: don't throw away working code. Make it incrementally better. The unglamorous path is almost always the safer one.

When Context Changes Everything

The blanket "never rewrite" advice ignores that risk varies dramatically depending on what you're rewriting. A frontend is not a database. A microservice is not a monolith. The same decision framework needs different weights in different contexts.

Frontend Rewrites Carry Lower Risk

UI frameworks evolve on 3-5 year cycles. Frontend code is typically decoupled from data stores. And the blast radius of a frontend failure is contained - you can roll back to the old UI without losing data or breaking integrations. Netlify rewrote their entire frontend from Angular to React because their JAMstack architecture (standalone UI talking to APIs) meant the rewrite risk was genuinely low. Smartly.io migrated AngularJS to React incrementally with a simple rule: "New code? React. Refactoring old code? React maybe."

Backend rewrites are far more dangerous because they involve APIs, data models, persistent state, and business logic that has accumulated edge-case handling for years. The consequences of getting it wrong include data loss, broken integrations, and violated SLAs.

Database Migrations: Almost Always Evolutionary

Database schema changes are widely considered the riskiest area in application development. The Expand and Contract pattern - deploy new schema alongside old, dual-write during transition, migrate data, switch reads, then remove old schema - exists specifically because big-bang database migrations fail in ways that are hard to recover from. Martin Fowler and Pramod Sadalage pioneered evolutionary database design at ThoughtWorks, treating schema changes as small, reversible, version-controlled migrations. The rule is straightforward: evolve the schema unless you're moving to a fundamentally different data model.

The Monolith Question

Martin Fowler's observation from 2015 holds up: "Almost all the successful microservice stories have started with a monolith that got too big and was broken up." The modular monolith - single deployment with enforced module boundaries - has emerged as a middle ground that Shopify, StackOverflow, and Basecamp all chose over full decomposition. Start monolithic, add boundaries when complexity demands it, decompose only the pieces that genuinely need independent deployment.

The AI Variable

The most significant recent development is AI's impact on migration timelines. Things that were previously too expensive to do incrementally - tedious file-by-file migrations, mechanical syntax transformations, boilerplate modernization - are now feasible at speeds that change the math entirely.

Airbnb migrated roughly 3,500 React test files from Enzyme to React Testing Library in six weeks. The old estimate was a year and a half. Their pipeline used AI-powered processing with state machines, configurable retry loops, and rich context injection - not a developer manually converting each file, but not a blind find-and-replace either. Salesforce compressed a two-year legacy code migration to four months using dependency-graph-driven leaf-to-root AI refactoring. Qonto's Ember-to-React migration with Claude reported output "by orders of magnitude" above their 200 lines-per-day manual target.

These results have a pattern worth noting: AI favors structured incremental migration over big-bang rewrites. The most successful approaches embed AI within disciplined pipelines - file-by-file processing with validation gates, automated test execution, and human review for the remaining edge cases. The tool handles the mechanical grinding. The humans handle the judgment calls about architecture and design.

A critical caveat: up to 76% of initial LLM refactoring suggestions can be hallucinations when validation is absent. Speed without verification is just faster mistakes. I wrote about the practical side of maintaining code quality with AI tools if you're navigating that tradeoff.

AI makes the incremental path cheaper, which means the "no incremental path exists" exception to the "don't rewrite" rule triggers less often than it used to. But it doesn't eliminate the need for architectural judgment about what to migrate and in what order. The tools have changed. The thinking hasn't.

Decision Signals

The case studies and frameworks converge on a set of concrete signals. Not rules - signals. Each one shifts the probability, and you're looking for which direction the weight of evidence points.

Signals Favoring Refactor

The system is in production serving real users. The team that built it still maintains it. Test coverage is reasonable, or achievable through characterization tests without heroic effort. The problems are localized to specific modules rather than spread across the entire codebase. The technology stack is still supported and receiving security patches. Business constraints prohibit a feature freeze. And critically - the organization hasn't addressed the practices that created the mess in the first place. If the same team with the same habits rewrites the system, they'll reproduce the same architecture in slightly different syntax. Software structure mirrors org structure - if you rewrite without restructuring teams, you get the same design back.

Signals Favoring Rewrite

The technology stack is genuinely obsolete - not unfashionable, but unsupported. No security patches, no ecosystem, a shrinking talent pool that makes hiring painful. The architecture fundamentally cannot support required capabilities. Not "it's slow" (optimize the bottleneck) or "it's ugly" (refactor it), but "the data model is wrong for what this system needs to become." The system is small enough and well-understood enough that rebuild complexity is low. The original team is doing the rewrite, so institutional knowledge transfers with the people rather than being lost with the code.

Red Flags That a Rewrite Will Fail

Some patterns predict failure reliably. No articulated reason beyond "the code is ugly" or "we want to use a newer framework." No migration plan for data, integrations, or customer transition. The old system continues receiving features during the rewrite, creating a moving target that the new system can never catch. The rewrite is bundled with a product pivot or experimental launch. The team plans to switch languages, architectures, and infrastructure all at once. Product managers see only the visible 10% of the system's complexity and estimate accordingly.

And the most reliable red flag of all: the primary motivation is developer boredom or resume-building rather than a business problem that the current architecture genuinely cannot solve.

The Real Question

The rewrite-versus-refactor framing is a false binary. The real question is: how incrementally can you replace this system while continuing to deliver value?

The Strangler Fig pattern has become the dominant migration strategy not because it's elegant - it isn't. Compatibility layers are ugly. Transitional architecture feels wasteful. Running old and new side by side is operationally painful. But it converts a high-risk, all-or-nothing gamble into a series of small, reversible, value-delivering steps. Every successful case study in this article used some version of it. Every failure skipped it.

Facebook didn't rewrite their app - they evolved the language. Shopify didn't decompose into microservices - they enforced boundaries within their monolith. Twitter didn't rebuild all at once - they replaced components over five years. The few legitimate exceptions to the "never rewrite" rule involve small, well-understood systems with stable teams, obsolete technology with no migration path, or fundamentally mismatched architectures. And even then, the incremental approach usually outperforms a clean-slate rebuild.

Twenty-five years of evidence. Billions of dollars in lessons. The pattern is clear. If your team is standing at the crossroads, the burden of proof is on the rewrite - not on the incremental path.

In December 2022, I wrote an article about using AI to clean data. I was excited. GPT-3 had just become accessible, text-davinci-003 could parse messy CSV rows, and I built a demo with 6 rows of sample data. Six rows. I showed how the model could normalize dates, fix capitalization, and extract structured fields from free text. It worked beautifully.

It also didn't scale past a demo.

The model cost $0.02 per 1,000 tokens. Rate limits capped throughput at a few hundred requests per minute. Every API call returned slightly different formatting. There was no validation, no error handling, no way to reproduce results. I was essentially asking the world's most expensive intern to hand-clean each row one at a time.

Three years later, I build data ingestion systems that process millions of records from dozens of sources, each arriving in its own special flavor of broken. County tax rolls with names like "SMITH JOHN A JR & MARY B SMITH-JONES TTEE OF THE SMITH FAMILY TRUST." Address files where column B switches meaning halfway through. CSV exports with mixed encodings, missing headers, and creative interpretations of what "null" means.

AI is central to how these systems work. But not the way I imagined in 2022. The LLM doesn't touch every row. It looks at a sample, figures out what's wrong, creates a transformation plan, and hands it off to pandas and Pydantic to execute and validate. The expensive, intelligent part happens once. The cheap, deterministic part runs on every row.

This is the article I should have written three years ago. Here's what actually works when you're cleaning data at production scale - not with 6 rows, but with millions.

Every Customer's Data Is Broken in Its Own Way

Clean datasets are all alike; every dirty dataset is dirty in its own way.

I've processed data from county governments, mortgage companies, title agencies, and tax assessors across 14 states. Not one source arrived clean. The problems fall into predictable categories, but the specific combination is unique every time.

Start with the structural problems. CSV files with no headers, mixed delimiters (tabs in one section, commas in another), multiline fields that break naive parsers. One county sends an Excel workbook with 6 tabs. Another sends a ZIP of CSVs named EXPORT_FINAL_v2_CORRECTED(1).csv.

Then there's encoding. Latin-1 data in a UTF-8 world. Windows line endings mixed with Unix. BOM markers that show up as invisible characters. I once spent a day debugging why 200 property addresses contained é instead of é - classic UTF-8 double-encoding from an intermediate system that converted the file twice.

Semantic ambiguity is worse. "John Smith" in one source, "SMITH, JOHN A JR" in another, "SMITH JOHN & MARY TTEE" in a third. Same person, three representations. Dates arrive in 15+ formats: 01/02/2024, 2024-01-02, Jan 2, 2024, 20240102, and my favorite, 1/2/24 (is that January 2nd or February 1st?).

Missing data has personality. Empty string, "N/A", "n/a", "NA", "NULL", "None", "-", "0", and a single space character - I've seen all of these mean "this field has no value" within the same file. Then there are the blanket deletions where an entire column is empty because the export process silently dropped it.

And domain-specific landmines. Addresses like "123 MAIN ST APT 4B C/O J SMITH" that need to be split into street, unit, and care-of fields. Property class codes that mean different things in different counties ("01" is "Single Family" in Lee County, "Residential" in Broward, and "Vacant Land" in Marion).

The problems are predictable by category but unique in combination. You can't write one script that handles all sources. And you can't hire enough people to manually map every new dataset - not when each one arrives with its own encoding, its own column names, its own creative interpretation of how to represent a null value.

The question is what you do about it. People have tried a lot of things.

What People Try Before AI

Every approach to data cleaning exists for a reason, and most of them work fine within their limits. The trouble starts when you outgrow those limits.

Manual cleanup in Excel. Power Query, VLOOKUP, find-and-replace. This handles one-off tasks with small files perfectly well. If you're fixing 200 rows once a quarter, Excel is the right tool. It's not reproducible, though. Next quarter, when the same file arrives with a new encoding issue, you start from scratch.

One-off Python scripts. A developer writes a pandas pipeline per data source. I've done this dozens of times: read CSV, rename columns, parse dates, strip whitespace, export. Works great for 5-10 sources. At 50+, you're maintaining 50 scripts, each encoding tribal knowledge that walks out the door when the developer leaves.

ETL platforms like dbt, Fivetran, or Airbyte are excellent for structured sources: APIs, databases, well-defined schemas. They struggle with truly messy data. The CSV with no headers. The Excel file where column B is sometimes "Owner Name" and sometimes "Property Address" depending on which county clerk exported it.

Data quality frameworks (Great Expectations, Soda, dbt tests) are superb at detecting problems: "15% of addresses failed to geocode." They don't fix anything. You still need a human to figure out why and write the fix.

Pasting into ChatGPT is where I was in 2022. Send 50 rows, ask for clean output, marvel at the result. It genuinely works for small batches. It doesn't scale: Osmos measured $2,250 for 100K rows through GPT-4 before Microsoft acquired them. Inconsistent output between calls, no validation, and rate limits that kill any automated pipeline.

AI-assisted pipeline is what I'll spend the rest of this article on. AI analyzes samples, generates transformation configs, quality gates verify the output. It combines deterministic execution from scripts, config-driven structure from ETL, automated checks from quality frameworks, and LLMs' ability to understand messy, ambiguous data.

Here's how they compare:

None of these approaches is universally wrong. Excel is perfect for one-off cleanup. Scripts are perfect for stable, well-understood sources. ETL platforms are perfect for structured pipelines. The AI-assisted approach fills the gap between them: understanding messy, ambiguous data and generating the transformation logic that traditional tools then execute.

## AI as Planner, Not Processor

The pattern that works in production is simple to describe: AI looks at a sample of your data, creates a transformation plan, and the system executes that plan with traditional tools. The LLM touches 100 rows. pandas processes 10 million.

Here's the concrete version. Five steps, each doing one thing.

Step 1: Define the target shape. Before any AI gets involved, you need a contract - what should clean data look like? I use Pydantic models for this because they double as validation:

class PropertyRecord(BaseModel):
    parcel_id: str = Field(description="Unique parcel identifier, e.g. '12-34-56-789'")
    owner_first: str
    owner_last: str
    address: str
    city: str
    state: str = Field(max_length=2)
    zip_code: str = Field(pattern=r"^\d{5}(-\d{4})?$")
    assessed_value: float = Field(ge=0)
    is_homestead: bool

This schema isn't just documentation. It's the success criterion. After cleaning, every row must validate against it.

Step 2: Describe expectations. Give the AI context about the data - not the schema (that's step 1), but what the data represents. "This is a property tax roll export from a county government. Each row is a parcel. Owners can be individuals, trusts, or corporations. Addresses include both mailing and property-site addresses."

Step 3: Feed AI sample data plus a toolkit. Send the LLM 50-100 representative rows, the target schema, and a list of available transformation operations: rename_column, parse_date, split_name, filter_rows, map_values, convert_type. The LLM doesn't need to implement these - it just needs to know they exist.

Step 4: AI generates the plan. The LLM responds with an ordered list of transformations. In our production system, this is a JSON config:

{
  "transforms": [
    {"op": "rename", "mapping": {"OWNERNAME1": "owner_raw", "SITUS_ADDR": "address"}},
    {"op": "map_values", "column": "prop_class",
     "mapping": {"01": "Single Family", "02": "Mobile Home", "03": "Condo"}},
    {"op": "split_name", "source": "owner_raw",
     "targets": ["owner_first", "owner_last", "trust_flag"]},
    {"op": "filter", "expr": "address_type == 'situs'"},
    {"op": "convert", "column": "assessed_value", "to": "float", "null_value": 0}
  ]
}

This config could be a Python script, a SQL query, or a dbt model. The format matters less than the pattern: it's declarative, versionable, and deterministic. Same input always produces same output.

Step 5: Validate, execute, monitor. Run the plan on a subset. Check quality gates (more on those next). If gates fail, send the failures back to the AI and let it adjust the plan. Once the subset passes, run on the full dataset. Check gates again.

This is why it scales. The expensive part (LLM analysis) happens once per data source. The cheap part pandas applying the config) runs on every row. When the same county sends next month's data in the same format, the config runs without any AI involvement at all. When a new county arrives with a completely different layout, AI generates a new config.

In our production system, we run 100+ county configurations. Each one was originally created by a human engineer studying the data. But the process that human follows - look at samples, understand the schema, write the mapping - is exactly what an AI agent can automate. The engineer's job shifts from "write the config" to "review the AI's config and approve it."

## How Do You Know the Data Is Actually Clean?

You don't trust AI-generated code without tests. You shouldn't trust AI-generated transformation plans without quality gates either. The principle is the same: automate the work, verify the result.

In our production pipeline, every data load runs through four layers of checks before anything hits the database.

Schema gates are the baseline. Every field matches its expected type. IDs are unique. Required fields aren't null. Dates parse to ISO format. Foreign keys point at existing records. If your Pydantic model says zip_code must match ^\d{5}$ and 3% of rows have "N/A" in that field, the gate catches it before those rows propagate.

Shape gates catch structural problems. On a refresh (re-processing the same data source), parcel count shouldn't change by more than 10%. If it does, someone uploaded the wrong file, or the schema changed, or data got truncated. We flag anything above 5% as a warning and above 10% as an error that blocks ingestion.

Distribution gates are where it gets interesting. After parsing 50,000 owner names, we check the top 20 most common first names. If "SMITH" appears as a first name, the parser swapped first and last name columns. If more than 5 of the top 20 first names are uncommon (not in a standard frequency list), the parser is splitting names wrong - probably treating "JOHN MICHAEL" as first="JOHN MICHAEL" instead of first="JOHN", middle="MICHAEL."

We run similar checks on address geocoding: after standardizing addresses through an address validation API (we use SmartyStreets), at least 60% should validate. Below 40% means a systematic parsing problem - wrong column, encoding mismatch, or format the parser doesn't recognize. On exemption rates, 25-40% of residential parcels typically have a homestead exemption. Below 10% means exemption data is missing entirely. Above 60% means the parser is incorrectly flagging non-exempt parcels.

Business logic gates check relationships across tables. Every owner record must reference an existing parcel. Every transaction must reference an existing parcel. Exemption codes must exist in the reference table. These catch join errors and mapping mistakes that look fine in isolation but break when you query across tables.

The thresholds (10%, 40%, 60%) come from processing 100+ data sources over several years. They encode what "normal" looks like. Data is never 100% accurate - there's always noise, always edge cases. The goal isn't perfection. It's catching systematic problems early, before bad data propagates through the system and produces wrong results downstream.

Here's the part that connects to the AI pipeline: when a gate fails, the system doesn't crash. It reports which gate failed, which rows caused the failure, and what looks wrong. That report feeds back to the AI agent, which can analyze the failures and adjust the transformation plan. Human reviews the adjustment, approves, and the pipeline reruns. This is the feedback loop that makes the system trustworthy: AI generates the plan, gates verify it, failures get fixed, and each iteration improves the result.

This isn't academic idealism. [Research on production agent deployments](https://arxiv.org/pdf/2512.04123) shows that 68% need human intervention within 10 steps. The best AI systems aren't fully autonomous - they're semi-autonomous with clear checkpoints. Quality gates are those checkpoints for data. I wrote up why agents declare victory too early.

Name Parsing - Where the Hybrid Pattern Clicks

Let me show you the "AI as planner" pattern with a concrete case study.

Our data has a full_name field. It contains everything from "John Smith" to "SMITH, JOHN A JR & MARY B SMITH-JONES TTEE OF THE SMITH FAMILY TRUST DATED 01/01/2020." We need to extract: first name, last name, trust flag, corporate flag. For every record.

The traditional approach is a regex-based parser. Ours handles the common patterns well: "JOHN SMITH," "SMITH, JOHN A JR," "DR JOHN MICHAEL SMITH," Hispanic compounds like "DE LA CRUZ." It scores each parse result with a confidence value based on how well the tokens match known name patterns.

About 80% of names parse cleanly with high confidence. That's 40,000 out of 50,000 records handled instantly by deterministic code. No AI involved, no API cost, no latency.

The remaining 20% is where regex breaks down. Trust names: "DEBBIE DAVIDSON REVOCABLE TRUST." Multiple owners in a trust: "WALSH JOSEPH T & NHUNG REVOCABLE TRUST." Corporate entities: "M & M MANAGEMENT LLC." Names where context matters: is "LE" a Vietnamese surname or a legal abbreviation?

These low-confidence results get routed to an LLM. The prompt includes the raw name string, the target schema (first_name, last_name, trust_flag, is_corporate), and a few examples of correct parsing. The LLM handles the ambiguity that regex can't.

Out of those 10,000 LLM-routed names, roughly 9,500 resolve cleanly. The remaining 500 get flagged for human review - a queue that a person can clear in a few hours instead of manually fixing all 10,000.

The cost math: the LLM portion costs about $2 for 10,000 names. The alternative is hiring someone to manually parse 10,000 complex names, which takes days.

The pattern generalizes beyond names. Any data cleaning task where 80% is predictable and 20% is ambiguous fits this model: let traditional tools handle the easy cases, route the hard cases to AI, validate everything against the schema, and flag what neither can handle. You're not replacing the regex parser. You're adding a second tier that handles what the regex parser can't.

Stop Copy-Pasting Into ChatGPT

There's a difference between using AI as a chat tool and using AI as a worker. Most "AI data cleaning" tutorials show the chat version: open ChatGPT, paste 50 rows, ask it to clean them, copy the result back. That works for one-off tasks the way manual Excel cleanup works - fine for small, non-recurring jobs.

The production version looks completely different from the user's perspective. Here's what actually happens when someone uploads data to our system:

A user receives a link. They drop their files - a ZIP of CSVs, an Excel workbook, a folder of PDFs. They don't know or care about our schema. They just know they need to get their data into the system.

Behind the scenes: the system unpacks the archive and identifies file formats. An AI agent examines samples from each file, detects what each column likely represents, and generates a transformation config. The system runs that config, quality gates verify the output, and if something fails, the agent adjusts and retries. If it still can't resolve the issue, a human gets a specific report: "45% of addresses didn't geocode. Likely cause: PO boxes mixed with street addresses. 1,247 rows affected."

The user who uploaded never sees any of this. They get "success" or "we need to clarify a few things."

The key difference from the chat approach: an AI agent uses tools. It doesn't try to hold a million rows in its context window. It calls pandas to read files, runs SQL to check distributions, executes validation functions, and writes configs. It iterates - try, check, adjust, retry - without a human in the loop for each step.

An agent processing 1,000 files doesn't pipe each file through the LLM. It categorizes files by format and problem type, creates a plan per category, and executes plans with traditional tools. The LLM might make 20 API calls total for 1,000 files. The rest is pandas, Pydantic, and SQL doing what they do best: processing data fast and deterministically.

This isn't a chatbot. It doesn't need a conversation UI. It doesn't need your attention. It reads messy files, figures out what's wrong, writes the fix, tests it, and leaves a report. When it can't fix something, it tells you exactly what's wrong and why. That's the kind of AI I'm excited about - not the one that talks to you, but the one that works for you.

Production Realities

Let me be honest about the parts that aren't elegant.

Sending every row through an LLM doesn't survive contact with the finance team. One million CSV rows at 500 tokens each through a budget API (\(0.10/M tokens) costs \)50 per run. Through GPT-4, it's closer to $5,000. Run that weekly across 50 data sources and you're looking at a six-figure annual line item for data cleaning alone. The config-generation approach costs a few dollars per data source (one-time LLM analysis of samples) plus pennies for pandas execution on every run. That's 3-4 orders of magnitude cheaper. One practical tip: if you are sending data to an LLM, send it as CSV, not JSON. CSV uses 50-56% fewer tokens for the same data and LLMs parse it just as accurately.

Rate limits compound the cost problem. We've had API providers throttle us during burst processing. Fifty thousand records hitting an LLM endpoint floods the rate limit within minutes. Batch APIs help (50% discount, higher throughput), but the latency goes from seconds to hours. If you need data cleaned before a morning deadline, that's a problem.

LLMs also aren't deterministic. Same prompt, different run, slightly different output. For planning this is fine - the generated config is deterministic once it exists. For row-level processing it's a deal-breaker. I've seen the same name parsed as "John Smith Jr" in one call and "Smith, John Junior" in the next. Configs don't have this problem. They run the same way every time.

And there are things AI simply can't do yet. Deduplication across records is still hard - a [2025 evaluation](https://arxiv.org/html/2511.21708) found that LLMs excel at standardization and profiling but fail at non-exact deduplication. "J. Smith at 123 Main" vs "John Smith at 123 Main St Apt 4" requires comparing millions of pairs, and LLMs are both too slow and too inconsistent for this. Use dedupe or recordlinkage for fuzzy matching. AI can help define the matching rules, but the execution needs traditional algorithms.

For high-volume or privacy-sensitive pipelines, self-hosting (Ollama for development, vLLM for production, cloud GPUs for burst) eliminates rate limits and API costs. The break-even is roughly 2 million tokens per day at 70%+ utilization. Below that, APIs are cheaper when you factor in infrastructure overhead.

The upside that makes all of this worthwhile is resilience. Traditional pipelines are brittle - one unexpected column name, one new encoding, one schema change, and the pipeline crashes at 3 AM. AI-assisted pipelines degrade gracefully. New format arrives? The agent analyzes the sample and generates a new config. Encoding changed? Detected and handled. Unseen edge case? Flagged for review instead of silently producing garbage. Each new data source makes the system smarter, not more fragile. The library of configs grows, the quality gate thresholds get refined, and the percentage of cases handled automatically increases over time.

The Silent Helper

Everyone is tired of chatbots. "Talk to AI" has become the default pitch for every product, every feature, every startup. But there's another kind of AI that doesn't get the hype and does most of the useful work.

It doesn't have a chat interface. It doesn't need your attention. It reads 500 messy files while you sleep, figures out what's wrong with each one, writes the transformation config, tests it against quality gates, and leaves a report on your desk in the morning. When it can't fix something, it tells you exactly what's wrong and why. When a new format arrives that it's never seen before, it adapts instead of crashing.

What previously required a week of an onboarding engineer's time - staring at spreadsheets, guessing at column meanings, writing one-off scripts, debugging encoding issues - now takes an afternoon of supervised automation. The human becomes the reviewer, not the laborer. That's a better use of judgment.

The three-year lesson is simple. AI is best as a planner and edge-case resolver, not as a row-level processor. Define your target schema. Sample the data. Let AI generate the transformation plan. Verify with quality gates. Execute at scale with traditional tools. Handle exceptions. That's the pattern, and it works.

This approach isn't revolutionary - the industry has converged on it independently. Every major data platform shipped some version of "AI suggests, human approves" in 2025. What's still missing is the practitioner knowledge: what thresholds to set, how to structure the feedback loop, when to route to AI versus when traditional tools are sufficient. That's what I tried to share here.

As agents get better at using tools, this pattern will become standard infrastructure. Today it's an engineering pattern you build. Soon it'll be a checkbox in your data platform. But the fundamentals - target schemas, quality gates, confidence-based routing - those don't change. They're the boring, essential foundation that makes the AI part trustworthy.

Originally published at potapov.dev.