Ask an LLM for a Win Probability Twice and You Get Two Different Answers

Everyone ships "AI predictions" the same way now: paste the stats into a prompt, ask the model who wins, print whatever comes back. It demos beautifully. And the moment you charge money for it, it falls apart, because a language model cannot stand behind a number.

When I built the AI prediction suite for Ultimate Fighting Stats, the paid tier promised fight predictions with confidence scores, method and round probabilities, and a public historical accuracy dashboard. That last item is the trap. An accuracy dashboard is a standing, auditable claim about every prediction you have ever made. You cannot back that claim with a system that gives different probabilities on every run.

So the architecture splits in two. A calibrated XGBoost model produces every probability. Claude writes the prose that explains them. Neither one is ever allowed to do the other's job.

The line in the design doc that settled it

The internal implementation plan puts the rationale in one sentence, and I have not found a better way to say it since: "LLMs cannot produce calibrated, deterministic, backtestable probabilities."

Unpack those three words, because each one is a product requirement, not a technical nicety.

Calibrated means that when the system says 63%, the event happens about 63% of the time over many predictions. That is a measurable property. We measure it with Expected Calibration Error, and a model version cannot go live unless its ECE clears a threshold. There is no equivalent procedure for a chat completion.

Deterministic means the same fight with the same data yields the same number, today and in a regression test six months from now. Version-pinned model artifacts give you that. Sampling from a language model does not.

Backtestable means I can run the exact production model over three years of historical fights it never trained on and publish the result. That backtest is the entire basis of the marketing claim. If the number generator cannot be replayed, the claim cannot be substantiated.

The design doc is blunt about the alternative, noting that competitors have tried asking an LLM to pick winners and ended up with inconsistent probabilities across runs, nothing that can be calibrated or backtested, and an accuracy dashboard that is impossible to substantiate. And so the rule in the doc: "The LLM never sees a probability question; it only writes prose grounded in numbers we already computed."

Two layers, two jobs

The division of labor is strict.

The prediction engine is XGBoost, trained in Python on our fight dataset, producing the win probability, a method distribution across KO/TKO, submission, and decision, and a round distribution. Training happens offline; the Python code never deploys anywhere.

The narrative layer is Claude Sonnet 4.6 via AWS Bedrock. It receives the model's output, the underlying stats, and the top SHAP features, and produces the roughly 120-word preview a fan actually reads. This is what LLMs are genuinely great at: turning structured numeric input into readable, on-brand prose.

The boundary is enforced, not just documented. The prompt constrains Claude to the stats in its input JSON, and a validator parses every generated narrative for numeric claims that do not appear in that input. Fail once, regenerate. Fail twice, fall back to a template. The LLM physically cannot introduce a number into the product that the statistical model did not compute first.

The boring model is the correct model

Why XGBoost and not a neural network? Because of the shape of the data. After filtering to fights where both fighters have enough history, the usable training set is a few thousand fights, roughly doubled by writing each fight twice with the corners swapped so the model learns to be order-invariant. The ML readme states the decision plainly: "On 5-12k tabular rows, XGBoost meets or beats NNs and trains in seconds." Gradient-boosted trees on tabular data is the least exciting choice available, and it is the right one. A neural net and a stacked ensemble are explicitly listed as deferred, consciously excluded rather than forgotten.

That discipline got tested. In mid-2026 we evaluated a stacked blend of the model with the betting market and found it added less than 0.003 log-loss on bouts with odds. Not worth a contract change; the trees already absorb the market feature directly, with monotone constraints on the market-implied probability and ranking difference so the model cannot learn a backwards relationship to them.

The order-invariance trick also produced my favorite bug in the project. When you flip fighter one and fighter two, signed differentials like reach advantage negate cleanly. But a probability feature like the market-implied win probability has to complement instead: p becomes 1 minus p. The v1.0 pipeline negated it like every other feature, which corrupted the odds signal on every flipped training row. Half the dataset was quietly teaching the model nonsense about the market. Deterministic, versioned pipelines are what let you find and fix that kind of thing; the fix is now written into the training readme so nobody reintroduces it.

The promotion gate is product policy

Every training run ends with an evaluation on a strictly temporal holdout: train on the past, test on a later window the model never saw. Probabilities get Platt-scaled on the validation set, then accuracy, AUC, log-loss, Brier score, and ECE are computed on the held-out test.

Then the gate, straight from the readme: promotion requires ECE below 0.08 and accuracy at or above baseline. If either check fails, the new version still uploads to S3, but with its active flag set to false, and a Slack alert fires. A worse model can exist; it cannot serve traffic. Rollback is a one-row update in the model registry.

I think of that gate as product policy wearing an engineering costume. It is the mechanism that makes the public accuracy page honest by construction. Nobody has to remember to check the numbers before shipping a model, because an unchecked model does not ship.

Calibration discipline shows up in the UI too. The winner confidence users see is clamped to a display range of 0.52 to 0.82, while the raw value is retained internally for auditing. An overconfident 91% that loses is how you churn a subscriber; the clamp exists because a calibrated system should also be presented with humility.

No Python in the hot path

Production inference runs in Node, not Python. The training pipeline exports three artifacts to S3: the ONNX model, a scaler JSON with per-feature means, standard deviations, and medians, and a metadata JSON carrying the version info, class labels, metrics, and the Platt calibration parameters.

The inference wrapper in the API loads all three from S3 and runs the model with onnxruntime-node inside a Lambda. Loaded models are memoized per container with a one-hour TTL, so a warm Lambda serves predictions without touching S3 at all. Preprocessing in Node mirrors training exactly: nulls impute to the training median, then z-score normalization from the scaler file. Even calibration crosses the language boundary explicitly; ONNX does not encode the Platt step, so the Node code reads the a and b parameters from metadata and applies the sigmoid itself.

The part I would steal for any ML deployment is the handshake. At load time, the code verifies the scaler's feature name list against the feature names compiled into the TypeScript pipeline, element by element, and verifies the model's feature version against the code's feature version. Both are currently "v2", an 81-feature contract. Any mismatch throws immediately, with an error that says exactly which side is stale. The failure mode this prevents is the worst one in production ML: features silently feeding the wrong model inputs and producing confident garbage. Here, drift between training and serving is a loud crash, not a quiet accuracy decay.

What calibration buys you commercially

Here is the payoff. Because every prediction comes from a versioned, replayable model with a real held-out test set, the site publishes actual numbers with provenance instead of vibes.

The active winner model, v1.2, live since July 2026, records 65.8% accuracy on 393 unique held-out fights from October 2025 through June 2026, with an ECE of 0.023 against the 0.08 gate. The published constants in the codebase carry a provenance comment verifying they reproduce the model registry row bit for bit, and the subsets reconcile: 170 bouts that had betting lines at 71.5%, 223 bouts without lines at 61.4%, and the weighted combination lands on the same 65.8%.

On the bouts with betting lines, the model's 71.5% sits near the vig-free market favorite's 69.7%, with slightly better log-loss. The gap is within sampling noise on 170 bouts, so the copy we allow ourselves is that the model holds its own with the market. Not beats it. Holds its own. The source file says so in a comment right above the numbers, along with a sharper observation: when the model disagrees with the market, it is roughly a coin flip, which makes a "model edge" a talking point and not a bet signal. The benchmarks file is as much editorial policy as data, and I mean that as a compliment to whoever has to write marketing copy against it later, which is me.

That is what "calibrated" buys you. Not just better math. The ability to publish a number, put your product name next to it, and know it will survive someone checking.

The division of labor generalizes

I wrote before, in Beyond Chatbots, that the interesting AI systems do work instead of chatting. This project is what that principle looks like when the work is probabilistic and people are paying for it.

The pattern is not specific to fight predictions. Anywhere a product makes a quantitative claim, the same split applies: a boring, versioned, testable model owns the numbers, and the language model owns the explanation, fenced in so it can never invent a figure. If your product's central number comes out of a prompt, you do not have a prediction product. You have a very confident narrator.

Put the statistical model under the chatbot. Let each layer do the only job it can actually defend.