I Doubled My Training Data and Poisoned Half of It

For UltimateFightingStats.com I train an XGBoost model that predicts UFC fight winners from public fight statistics. Somewhere in the middle of that project I shipped a bug that I have been thinking about ever since, because it is the purest example I know of how machine learning fails in practice. Nothing crashed. No test went red. The model trained, exported, deployed, and produced sensible-looking probabilities. It was simply learning from data where the single most informative feature had been silently corrupted on exactly half the training rows.

This is the story of that bug, how it got there, what fixing it was worth, and why, at the end of it all, I still refused to publish the one headline number everyone wants from a sports model.

Why every fight goes into the training set twice

A fight between fighter A and fighter B is the same fight as one between fighter B and fighter A. But a tabular model does not know that. The feature vector has a fighter-one slot and a fighter-two slot, and whichever fighter the data pipeline happens to list first gets slot one. If you train naively, the model can pick up meaningless artifacts of that ordering.

The standard fix is symmetry augmentation. Every row in the training table is written in twice: once as-is, and once with the two fighters swapped. The label flips with it. The model sees each matchup from both sides, the dataset doubles, and the learned function is pushed toward order invariance. On a dataset of a few thousand fights, doubling your data for free is a genuinely good trade.

Here is the part that the textbook description glosses over: "swap the fighters" is not one operation. It is a different operation for every feature, and my feature contract has 81 of them.

Every feature needs its own flip rule

When you swap fighter one and fighter two, each feature transforms according to what it means:

• Symmetric pairs swap. Fighter one's strikes per minute and fighter two's strikes per minute trade places. Simple.
• Signed differentials negate. A reach difference of +8 cm becomes -8 cm. Same for height difference, age difference, rank difference, and the head-to-head record.
• Probabilities complement. If the market implies a 70% chance fighter one wins, then after the swap it implies a 30% chance. The value 0.70 must become 0.30, which is 1 minus p, not minus p.
• Signed categorical features flip sign. A style-clash encoding of -1, 0, or +1 mirrors to +1, 0, or -1.

The training code makes these classes explicit. This is the core of add_symmetric_flips() in train.py, essentially as it appears in the repo:

# Swap each f1_*/f2_* pair
for a, b in SYMMETRIC_PAIRS:
    flipped[a] = df[b]
    flipped[b] = df[a]
# Negate features
for col in NEGATING_FEATURES:
    if col in flipped.columns:
        flipped[col] = -df[col]
# Complement probability features (0.7 -> 0.3); NaN stays NaN
for col in COMPLEMENT_FEATURES:
    if col in flipped.columns:
        flipped[col] = 1.0 - pd.to_numeric(df[col], errors="coerce")
# Style clash flips (-1 -> 1, 0 -> 0, 1 -> -1)
for col in STYLE_CLASH_FEATURES:
    if col in flipped.columns:
        flipped[col] = -df[col]
# Flip target
if "target_f1_wins" in flipped.columns:
    flipped["target_f1_wins"] = ~df["target_f1_wins"].astype(bool)

Four transform classes, each applied to an explicit list of feature names. That structure exists because of the bug. The bug existed because, originally, it did not.

The bug: treating a probability like a differential

In version 1.0, the market's implied probability for fighter one, f1_vegas_implied_prob, sat in the negation list. On every flipped row, 0.70 became not 0.30 but -0.70. A comment in feature_names.py now memorializes it: v1.0 and v1.1 "wrongly negated f1_vegas_implied_prob (p -> -p), feeding a corrupted market signal to every flipped training row. Fixed in v1.2." The ML README says the same thing more bluntly: "v1.0 wrongly negated them, corrupting the odds signal on every flipped row."

Think about what this did. The betting market's implied probability is, unsurprisingly, the most predictive single feature a fight model has access to. The market aggregates information the box-score statistics never see. And on exactly half of the training set, that feature was garbage: a negative number in a column that should live between 0 and 1, with a target label that had been correctly flipped. The model was being told, on half its data, that the market signal pointed one way while the outcome pointed wherever it pleased.

The insidious part is that nothing failed loudly. Gradient-boosted trees are robust. XGBoost happily learned to partially discount the odds feature, hedged its splits, and still produced a model that looked fine on the surface. Standard augmentation intuition says doubling your data is safe. What it does not say is that augmentation multiplies your ways to corrupt data at exactly the same rate it multiplies the data itself. Every transform you apply is a new place to encode a wrong assumption, and a wrong transform is worse than no augmentation, because it manufactures confident, systematic noise.

The fix, measured

The fix itself was small: move one feature from the negation list into a new COMPLEMENT_FEATURES list and apply 1 minus p instead of minus p. It shipped in v1.2 of the winner model, alongside a few other changes the training config records: monotone constraints on the market probability and rank difference (the prediction should never move against those features), time-decay sample weights with an 8-year half-life, and retuned depth-4 trees. So the honest framing is that the numbers below measure the v1.2 release as a whole, with the flip fix as its headline ingredient, not the fix in isolation.

On the held-out test window (October 2025 through June 2026, 393 fights the model never trained on), the project's training runbook records overall winner accuracy going from 63.5% in v1.0 to 65.8% in v1.2, driven mainly by fights without betting lines, where the model records a move from 57.9% to 61.4%. The deployed model records an expected calibration error of 0.023, comfortably inside the promotion gate. Those figures are the deployed model's own recorded backtest, verified to reproduce bit-for-bit against the model registry, not a number I re-measured for this post.

Two points on that. First, 2.3 percentage points is a large amount of accuracy in this domain, and it was sitting inside a bug fix, not inside a bigger model. Second, the no-line fights improved most, which fits the diagnosis: once the market feature stopped being poisonous on half the rows, the trees could trust it properly where it existed and lean on the fight statistics more coherently where it did not.

I also wanted to know what my new features were worth separately from all of this, so the repo keeps a controlled A/B script (ab_v1_v2.py) that trains the v1 feature set (75 features) against the v2 feature set (all 81) on the same fights, the same temporal split, and the same baseline XGBoost configuration with Platt calibration, so the only difference is the feature columns. After the flip bug, I stopped trusting any comparison where more than one thing varies at a time.

The number I refused to publish

Here is the kicker, and the reason this story is really about restraint rather than debugging.

On the 170 held-out fights that had betting lines, the model records 71.5% accuracy against 69.7% for the vig-free market favorite, with lower log-loss (0.575 versus 0.594). If I were writing marketing copy, that is "our AI beats Vegas," and that phrase prints money in this niche.

I did not publish it, and the reason is written directly into the code. The comment block in model-benchmarks.ts, the file that ships the public numbers, says the accuracy gap is within sampling noise on 170 bouts, and instructs whoever touches the file to keep the public copy at "holds its own with the market," not a confident claim of beating it. It goes further: when the model disagrees with the market, it is roughly a coin flip, so a "model edge" is a talking point, not a bet signal. A 1.8 point gap on a sample of 170 is exactly the kind of result that evaporates on the next 170.

The same file carries a note on the method model (KO/TKO versus submission versus decision), which records 48.2% on a three-way problem: decisions are about half of all fights, so always guessing "decision" already scores around 50%. The comment literally says do not tout this number. Writing "do not tout" notes into the source next to the metrics turned out to be the most effective honesty mechanism I have, because the numbers and the caveats travel together, and future me cannot quote one without seeing the other.

The guardrails that outlived the bug

A bug like this changes how you build. The flip fix took minutes; the process changes it prompted are permanent.

A temporal leakage canary in CI. A unit test inserts a fixture fight at a known date, runs the entire feature pipeline, and asserts that no source row used in any aggregate is dated on or after the fight being predicted. CI blocks merges if it fails. Leakage is the other great silent corrupter of tabular ML, and it deserves a tripwire, not a code-review habit.

Calibration as a first-class metric. Every training run computes expected calibration error on the held-out test set, and Platt scaling parameters are stored in the model's metadata so the Node inference path applies the exact same calibration that was evaluated.

A promotion gate. A new model version is only promoted if ECE is below 0.08 and accuracy is at least the current baseline. Fail either check and the version still uploads, but with is_active=FALSE, and a Slack alert fires. The pipeline is allowed to produce a bad model; it is not allowed to quietly serve one.

An orientation sanity check, by construction. Because evaluation runs on the flip-symmetric test set, the recorded metrics are orientation-independent. A model that scored differently depending on which fighter the data listed first would show up as exactly the kind of asymmetry this whole exercise exists to remove.

Compute was never the problem

One last observation. The total training cost of this system is approximately zero. XGBoost on roughly 9 to 18 thousand rows trains in seconds on a laptop CPU; the runbook explicitly says not to spin up cloud training instances because it would be pointless at this scale. There is no neural network, because on small tabular data XGBoost meets or beats one and trains in seconds.

Every hard problem in this project was a correctness problem: a flip rule that was wrong, leakage that had to be designed out, calibration that had to survive the trip from Python training to Node inference, and a marketing claim that had to be resisted. None of them were compute problems. In small-data ML, the scarce resource is not GPUs. It is the discipline to notice that 0.70 became -0.70 on half your rows, and the honesty to say "holds its own with the market" when the sample size will not support anything stronger.