I'm training a model for my PhD, and recently I found out that my test set was not a test set. It was a replay of part of my training data. This post is about how that happened, what I did about it, and the part I still can't explain.
What happened
My training and test data come from generators, not fixed files. Both generators were seeded with the same value, and their draw sequences lined up. So my "independent" test set was the first 10% of my training set, regenerated sample for sample.
At the lowest noise level the inputs were byte-identical. I trained without noise, and each test set adds its own noise when it's generated, so the higher noise levels were the same underlying samples with noise on top.
That makes the evaluation useless as a measure of generalisation. It could tell me the model fits data it was trained on. It couldn't tell me anything about data it hadn't seen, which is the whole point of a test set.
How I found it
By accident. I was looking into an unrelated question about my data distribution, which led me to the project's seeding convention. That convention named a single shared seed for "test sets". When I checked which functions used that seed, the training generator did too.
Reading the code made me suspect it. To confirm, I reproduced both generators in a standalone script and compared them sample by sample. They were identical.
What I changed
The cause was simple: one seed shared by two jobs. I gave the test generator its own seed, different from every other seed in the project, and regenerated only the test sets. The code change was one line, plus a new seed constant.
I didn't retrain. I re-evaluated the model I already had on the new test data. That was deliberate. The model was unchanged, so any difference in the numbers had to come from what it was scored on.
What I still don't know
The honest numbers came out better, not worse.
I expected the opposite. If the old test set was leaked, the old score should have been flattering. Instead, the failure rate went down and the utility score went up at every noise level I tested.
I have guesses, but I haven't tested them, so I won't present them as findings. This is also one checkpoint and one new test seed. I haven't measured how much a fresh test seed alone moves the numbers, so I can't say how much of the shift is more than noise.
What I can say is that a bug that moves a number in the direction I didn't expect is still a bug. I'd rather report that than invent a tidy reason.
What's still open
The validation set I used during training came from a copy of the same bug, and I haven't fixed it. Fixing it would change nothing without a retrain, and the run it fed is finished. A later stage of my pipeline also has its own copy. Until I fix those, I treat anything scored on that data as in-sample.
What to check if you're building evals
This is the checklist I wish I'd run on day one:
- Does the evaluation data come from a different stream than the training data? If you use generators, check that the seeds differ, and that they differ on purpose. Deriving every stream from one parent seed (NumPy's
SeedSequence.spawn, for example) is safer than picking integers by hand. - Can you show the overlap is zero, rather than assume it? Different seeds don't guarantee disjoint data, especially in small or discrete spaces. Compare test items against training items directly, and put that check in a test that fails loudly, so it's repeatable and not a one-off.
- Which splits does each evaluation draw from? If a number is computed on data that also fed training or validation, it isn't held out, whatever it's called.
- Do your numbers ever look surprisingly good, or surprisingly bad? Either is a reason to check the pipeline before you trust the model.
- Can you re-run an evaluation on its own? Mine could be re-run without retraining, which is what made this cheap to fix.
Why this matters beyond my own work
I'm new to AI safety, and I don't have a view on the field. Most claims about how a model behaves rest on an evaluation. If the evaluation is broken, the claim is broken, and nobody can tell from the outside.
My mistake was a small version of the test-data leakage that affects language-model benchmarks. It stayed hidden because the number didn't look wrong. So the lesson for me is small and practical: check how the numbers were produced, and don't wait for a bad number before doing it.