# Why Peeking At Holdout Data Weakens The Split Slug: why-peeking-at-holdout-data-weakens-the-split. Canonical: https://tradingtune.com/blog/why-peeking-at-holdout-data-weakens-the-split ## Summary Explains how repeated decisions based on held-back data can turn validation into another tuning step. ## Key facts - Checking the holdout each time you adjust a parameter turns it into a second training set. - Fix the split and the stopping rule before the sweep starts, so what a single cycle shows cannot change the plan. - If a cycle looks weak, roll forward to the next window instead of reopening the current one and adjusting. - Reserve the final window as a true holdout that is checked once, after the candidate range is already chosen. ## Outline - What Repeated Holdout Peeks Do To Your Validation Split (#what-repeated-holdout-peeks-do-to-your-validation-split) - Setting Up A Sweep With A Real Walk-Forward Split (#setting-up-a-sweep-with-a-real-walk-forward-split) - What Changes As You Move Through The Parameter Range (#what-changes-as-you-move-through-the-parameter-range) - A Hypothetical Look At One Sweep (#a-hypothetical-look-at-one-sweep) - When A Setting Looked Solid Until The Holdout Leaked In (#when-a-setting-looked-solid-until-the-holdout-leaked-in) - How I Keep The Holdout Honest (#how-i-keep-the-holdout-honest) - The One Habit That Keeps A Split Meaningful (#the-one-habit-that-keeps-a-split-meaningful) - Sources (#sources) ## Code snippets - Anchor: #pine-snippet-1