My ensemble scored 0.129 on the leaderboard, below the field median. I published it anyway, then tested six explanations for the gap.
Before submitting I wrote down a forecast: 0.143. The graded scores were 0.128 (single model) and 0.129 (ensemble).
Setup
804.5M raw market rows reduced to 292 BigQuery features, walk-forward validation with a one-month embargo, six months held out and read exactly once (+0.15171).
What I found, without spending another submission
- One hypothesis confirmed: my hold-out sat at the 83rd percentile of period difficulty. De-biasing for that gives +0.14084, within 0.00004 of the walk-forward mean computed a different way.
- Four falsified, including the two I liked most. One unsettled.
- Period difficulty explains 46% of the gap. The rest is documented, not explained away.
Five recorded forecasts, five overshoots, all in the same direction. That points to one cause, not five mistakes.
The rule I kept
An internal gain smaller than the fold-to-fold noise tells you which model to prefer, not what a leaderboard will show.
Links
- Case study: MSCapital market forecasting
- Notebook: My hold-out was a lucky stretch (Kaggle)
Top comments (7)
hold-outがperiod difficultyで83パーセンタイルだったという検証と、de-biasingで+0.14084に補正した点が印象的でした。私たちのラボの投稿でも、検証設計がモデル選択を左右するという話を書いたところです。de-biasingはスコアの補正として使ったのか、それともモデル選択の順位づけにも使ったのか、気になりました。walk-forwardのfoldごとのdifficulty分布で補正すると、モデルの順位が変わることはありましたか。
Thanks for the thoughtful question. In this write-up the de-biasing was used only to interpret the hold-out score, not to rank models. Model preference came from the walk-forward folds, and the hold-out was read once after that choice was made, so it could not feed back into selection.
I have not tested whether adjusting each fold by its own difficulty would reorder the models. It is a good check, and I would expect it to matter most when two models score this close.
Thanks for the detailed clarification — that separation makes the write-up much stronger: de-biasing as an interpretation tool for the hold-out, with selection locked in from the walk-forward folds before the hold-out was ever read. That discipline is exactly what keeps a 0.129 from turning into a story you tell yourself afterwards.
The fold-difficulty point is the one I keep coming back to. When two models are separated by less than the fold-to-fold difficulty spread, the ranking is partly a statement about which folds each model happened to face. A simple version of the check: for each fold, compute a difficulty baseline (e.g., the mean score across all candidate models on that fold), subtract it from each model's fold score, then re-aggregate. If the order survives that adjustment, you can trust it; if it flips, the honest conclusion is 'tied within fold noise.'
We see the same pattern in our own walk-forward runs, so I may test this adjustment on one of our recent competitions and write up whether the ranking moved. If I do, I'll link it here — I'd be curious whether your intuition (it matters most when scores are this close) holds on a second dataset too.
One thing I noticed while thinking it through: if the baseline is subtracted from every model on the same fold, the gap between any two models on that fold stays the same, so with a plain mean across folds the order cannot change. It can change when the adjustment rescales the folds, for example dividing by each fold's spread, or when the aggregation is not a simple mean.
For my case the more telling number is probably the per-fold difference itself: on how many folds the ensemble beat the single model, and how large the mean difference is compared to its variation across folds. With 0.128 against 0.129, I would expect that to land on "tied within fold noise", which is also what the leaderboard ended up saying.
I would be glad to read your write-up if you run it on your competitions. Thanks for pushing on this.
You're right, and it's a clean catch — subtracting the same fold baseline from every model leaves each fold's pairwise gaps untouched, so a plain mean can't reorder anything. My baseline-subtraction framing was sloppy there.
Where an adjustment can actually bite: rescaling each fold (dividing by its spread across models, so a quiet fold counts as much as a volatile one), or aggregating by rank or median instead of the mean. And your per-fold framing is sharper than either: the ensemble's win count against the single model, plus the mean difference set against its fold-to-fold variation — basically a paired test on the folds. For 0.128 vs 0.129 I'd bet on "tied within fold noise" too, and it's satisfying that the leaderboard independently landed there.
I'll build the check in that form — per-fold differences first, spread-rescaled scores second — and run it on a public dataset's walk-forward folds, then write it up and link it here. Thanks for pushing back; the check is better for it.
Writing the forecast down before seeing the graded score is the habit most leaderboard posts skip, and it makes the 46% explanation credible. The 83rd percentile difficulty finding is a good reminder that a single held-out window can mislead more than the model itself. Did you consider scoring across several rolling hold-out windows to see how wide the difficulty spread really is?
iin1006h02
Thanks, and yes, that is essentially where the 83rd percentile figure comes from. The walk-forward folds already act as rolling windows, so I ranked the hold-out period against the difficulty distribution of the other periods instead of trusting it on its own. De-biasing for that brought the hold-out score to +0.14084, within 0.00004 of the walk-forward mean computed a different way, which is what convinced me the window was the outlier, not the model.
What I did not do is re-read the final six-month hold-out in multiple slices. I wanted it read exactly once, so the spread estimate comes from the validation folds only. A proper rolling hold-out scheme is a fair next step, and I would expect it to show the spread is wider than a single window suggests.