09. Backtests Are Liars

Backtests Are Liars

The 80/20 Trap

One of the first things you learn in machine learning is "split your data into 80% and 20%." 80% for training, 20% for testing. Simple and intuitive. I used this approach from v2.1 through v2.5.

The problem is that a random 80/20 split on time-series data can leak information from the future.

Stock and crypto price data has a temporal order. If you train on January 2024 data and test on December 2023 data, the model has already "seen the future" when it's predicting the "past." Even with a temporal split that preserves order, relying on a single split point can overfit to the specific market conditions at that moment.

Through v2.5, I thought this wasn't a big deal. The accuracy was consistently in the 60% range, after all. But when I deployed it live, I took a -62.83% loss. It looked fine in backtests but blew up in production.

That's when I realized my backtests had been lying to me.

Walk-Forward: Walking Along Time

Walk-Forward validation is the honest answer to this problem. The core concept is simple: progressively expand the training window, and always test only on the "future."

Expanding Window vs Sliding Window

There are two main approaches to time-series cross-validation. Defined mathematically:

Expanding Window (the approach I used):

$$\text{Fold } k: \quad \text{Train} = \{x_1, \ldots, x_{t_k}\}, \quad \text{Test} = \{x_{t_k+1}, \ldots, x_{t_{k+1}}\}$$

Training data accumulates as folds progress. Fold 1's training data $\subset$ Fold 2's training data $\subset$ Fold 3's training data.

Sliding Window:

$$\text{Fold } k: \quad \text{Train} = \{x_{t_k - W}, \ldots, x_{t_k}\}, \quad \text{Test} = \{x_{t_k+1}, \ldots, x_{t_{k+1}}\}$$

Training data is maintained at a fixed width $W$. Old data drops off as new data comes in.

I chose the Expanding Window because of the nature of the crypto market. The 2018 bear market, the 2020-2021 bull run, the 2022 bear market. I wanted all of these different market regimes included in training. With a Sliding Window, information from older regimes gets lost. The Expanding Window does have the downside that older data may differ from recent patterns, but tree models like XGBoost have a natural ability to ignore irrelevant data.

Think of it like studying for exams. The Expanding Window is like reviewing everything from freshman year to the current semester before each test, while the Sliding Window is like only reviewing the most recent semester. If the subject keeps testing material from earlier semesters, the former approach wins.

The structure I used in v2.7:

Data: 2017-01 ~ 2026-01 (5-minute candles, ~950K rows)
Using most recent 200,000 samples

Fold 1: Train [~2023-07]      -> Test [2023-07 ~ 2023-12]
Fold 2: Train [~2024-01]      -> Test [2024-01 ~ 2024-06]
Fold 3: Train [~2024-07]      -> Test [2024-07 ~ 2024-12]

Final:  Train [~2025-07]      -> Test [2025-07 ~ 2026-01]

Training data grows with each fold (Expanding). Fold 1 trains on all data before July 2023 and tests on the second half of the year. Fold 2 trains on all data before January 2024 (including Fold 1's test period) and tests on the first half. And so on.

fold_boundaries = [
    ('2023-07-01', '2023-12-31'),
    ('2024-01-01', '2024-06-30'),
    ('2024-07-01', '2024-12-31'),
    ('2025-01-01', '2025-06-30'),
    ('2025-07-01', '2026-02-01'),
]

for test_start, test_end in fold_boundaries:
    train_mask = timestamps < test_start  # Always only the past
    test_mask = (timestamps >= test_start) & (timestamps <= test_end)

The key point: the test data is always in the future relative to the training data. The model can never peek ahead.

Mathematical Definitions of Evaluation Metrics

To interpret Walk-Forward results, you need to understand exactly what each evaluation metric measures.

Confusion Matrix and Derived Metrics

In 3-class classification (LONG, SHORT, NO_TRADE), prediction results for each class are summarized in a Confusion Matrix. From a binary classification perspective, for each class:

  • True Positive (TP): Predicted this class, and it actually was this class

  • False Positive (FP): Predicted this class, but it was actually a different class

  • True Negative (TN): Predicted not this class, and it actually wasn't

  • False Negative (FN): Predicted not this class, but it actually was

Precision and Recall

Precision:

$$\text{Precision} = \frac{TP}{TP + FP}$$

"Of everything I predicted as this class, how many were actually correct?" In a trading context, high LONG Precision means "when the model predicted LONG, the price actually went up most of the time." Low Precision means too many false entry signals, which translates to unnecessary losses in live trading.

Recall:

$$\text{Recall} = \frac{TP}{TP + FN}$$

"Of all actual instances of this class, how many did I correctly predict?" High LONG Recall means "of all the times the price actually went up, the model predicted LONG for most of them." Low Recall means many missed profit opportunities.

Precision and Recall are in a tradeoff relationship. Lowering the threshold increases Recall but decreases Precision (more trades, but lower accuracy). Raising the threshold increases Precision but decreases Recall (more selective trading, but more missed opportunities).

Think of airport security. Precision is "of all the people who triggered the alarm, what fraction were actual threats?" Recall is "of all actual threats, what fraction did the checkpoint catch?" Loosening security catches more threats (high Recall) but flags more innocent travelers (low Precision). Tightening security reduces false alarms but lets more threats slip through.

F1 Score

$$F1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

F1 is the harmonic mean of Precision and Recall. The reason for using the harmonic mean instead of the arithmetic mean is to ensure that F1 drops significantly if either metric is extremely low. With Precision 0.99 and Recall 0.01, the arithmetic mean is 0.50, but F1 is 0.0198. The harmonic mean enforces a stricter requirement for balance between the two.

Imagine grading PE class on two tests: the 100-meter sprint and sit-ups. With an arithmetic mean, scoring 100 on the sprint and 0 on sit-ups still gets you an average of 50. But with a harmonic mean, having a zero on either test drives the overall score toward 0. "You can't just be good at one thing. You have to do both." That strict grading approach is what F1 does.

F1(macro)

$$F1_{macro} = \frac{1}{K} \sum_{k=1}^{K} F1_k$$

This is the simple average of F1 across all $K$ classes. In our case, $K = 3$ (LONG, SHORT, NO_TRADE). Macro averaging matters because it isn't fooled by class imbalance. In our data, NO_TRADE makes up 62.9%, so if the model predicted NO_TRADE for every sample, Accuracy would be 62.9%. But F1(macro) would be very low because the F1 for LONG and SHORT would be 0. F1(macro) is the right metric for checking whether the model performs well across all three classes in a balanced way.

To use a school analogy, Accuracy is like your overall class rank (how many total questions did you get right), while F1(macro) is like the average of your grades across individual subjects. If you got an A in English but F's in Math and Science, your class rank might still be passable, but your subject-level average would be dismal. Colleges look at the balance across subjects.

What Each Fold Revealed

v2.7's Walk-Forward results:

FoldTest PeriodAccuracyF1(macro)Train AccOverfit Gap12023 H280.71%0.690494.64%+13.93%22024 H184.54%0.700090.31%+5.77%32024 H288.02%0.689689.07%+1.05%Average84.42%0.69336.92%

The number to focus on here is the Overfit Gap. This is the difference between training accuracy and test accuracy, a quantitative measure of overfitting.

$$\text{Overfit Gap} = \text{Acc}_{train} - \text{Acc}_{test}$$

A large gap means the model memorized noise in the training data; a small gap means the model learned generalizable patterns. In machine learning theory, this directly ties to the Bias-Variance tradeoff.

  • High Bias, Low Variance (underfitting): Both training and test accuracy are low. Gap is small, but absolute performance is also low.

  • Low Bias, High Variance (overfitting): Training accuracy is high but test accuracy is low. Large gap.

  • Low Bias, Low Variance (ideal): Both training and test accuracy are high. Small gap with high absolute performance.

Think of it like chefs. A High Bias chef barely learned the recipe, so the food tastes bad no matter which kitchen they're in. A High Variance chef makes perfect dishes only in their own kitchen but falls apart in someone else's. The ideal chef makes consistently great food wherever they go.

Fold 1 has a gap of 13.93%. Not enough training data, so overfitting is severe, a classic High Variance state. But as folds progress and more training data accumulates, Fold 3's gap shrinks to 1.05%. Training 89.07%, test 88.02%. Virtually no difference.

This is the real value of Walk-Forward. You can verify whether the model performs consistently. With a single 80/20 split, you'd only see Fold 3 and celebrate: "Wow, 88% accuracy!" But Walk-Forward also shows you Fold 1's 80.71%.

Interpreting this through the lens of learning curves: going from Fold 1 to Fold 3, training accuracy decreases from 94.64% to 89.07% while test accuracy increases from 80.71% to 88.02%. This is the classic pattern of "the model stabilizing as data grows." When the two curves (train, test) converge on a learning curve, it signals that the model's capacity is appropriate for the data.

F1(macro) Stability

Even more important than Accuracy is F1(macro), the metric that captures balanced performance across all 3 classes (LONG/SHORT/NO_TRADE).

Fold 1: 0.6904
Fold 2: 0.7000
Fold 3: 0.6896
std:    0.005

Standard deviation of 0.005. All three folds are stable around 0.69. This means the model isn't depending on specific market conditions during a particular period. It's learning consistent patterns. Computing the coefficient of variation (CV) of F1(macro) across folds:

$$CV = \frac{\sigma}{\mu} = \frac{0.005}{0.6933} \approx 0.72\%$$

A coefficient of variation under 1% indicates the model is highly stable.

What a Single Split Can't Show You

When I was using a single 80/20 split in v2.5, I was satisfied with a test accuracy of 60.16%. But hiding behind that number was:

  • The model might only work well in certain market regimes

  • It could nail the strong trending market of early 2024 but fall apart in the sideways market of late 2024

  • A single test window can't reveal this

Walk-Forward exposes these biases. Each fold tests a different market environment.

If I had tested v2.5 with Walk-Forward, the performance gap between folds would likely have been extreme. The -62.83% loss in live trading happened because the live market environment differed from the training data. Walk-Forward simulates exactly this kind of scenario in advance.

Final Model: The Most Recent 6 Months

After confirming consistency with Walk-Forward, the final production model was trained like this:

Training: ~ July 2025 (all historical data)
Testing: July 2025 ~ January 2026 (most recent 6 months)

Reserving the most recent data for testing is deliberate. The model going into production must work in the most current market environment. There's no guarantee that a model performing well on 2023 data will still work in late 2025.

Final test results:

  • Accuracy: 88.02%

  • F1(macro): 0.6896

  • Overfit Gap: +1.05%

An Overfit Gap of 1.05% is excellent. 89.07% on training, 88.02% on test. Almost no overfitting.

A Critical Detail in Scaling: Information Leakage

There's a commonly overlooked mistake in Walk-Forward validation: you must re-fit the scaler for each fold.

The StandardScaler transformation formula is:

$$z = \frac{x - \mu}{\sigma}$$

Here, $\mu$ is the mean and $\sigma$ is the standard deviation. This transforms the data to have mean 0 and standard deviation 1.

The problem is where you compute $\mu$ and $\sigma$.

for fold_idx, (train_idx, test_idx) in enumerate(folds):
    # Correct: fit on train for each fold, only transform test
    scaler = StandardScaler()
    X_train = scaler.fit_transform(X_train_raw)  # Uses train's mu, sigma
    X_test = scaler.transform(X_test_raw)         # Same mu, sigma for transform

What happens if you fit_transform on the entire dataset once? The test data's $\mu$ and $\sigma$ bleed into the training process. If BTC surged during the test period, the overall $\mu$ shifts higher, altering the scaling of training data. This is Information Leakage.

Mathematically, the correct approach transforms test data as:

$$z_{test} = \frac{x_{test} - \mu_{train}}{\sigma_{train}}$$

The wrong approach uses:

$$z_{test} = \frac{x_{test} - \mu_{all}}{\sigma_{all}}$$

Since $\mu_{all}$ and $\sigma_{all}$ include test data statistics, the model gains indirect access to future information. I made this mistake initially, and the suspiciously high accuracy tipped me off. Unrealistically good results are almost always a sign of a bug.

A perfect analogy: it's like knowing the exam scope in advance. You're supposed to study without knowing what the test covers (test data), but scaling on the entire dataset is like getting a hint that "calculus will be heavily weighted on this final." You'll score well, but that's not skill. It's cheating.

The Takeaway: Be Skeptical of Your Backtests

Even after going through Walk-Forward, I still don't trust backtest results 100%. v2.7's backtest return of +6,214% is certainly impressive, but live performance will be lower. Slippage, liquidity gaps, API latency, changes in market microstructure, none of these are reflected in backtests.

But Walk-Forward at least provides an answer to "does this model perform consistently across different market environments?" A single 80/20 split only answers "does it work in this specific 6-month window?"

Backtest returns can lie, but Walk-Forward is honest.

Honest doesn't mean "always right." It means "if there's a problem, it shows you." A single split hides problems, while Walk-Forward reveals them through performance differences across folds.

When I deployed v2.7 to production, I had far more confidence than before. Not because of the 88% accuracy, but because I had confirmed it performed consistently across three different market environments. F1(macro) standard deviation of 0.005, Overfit Gap converging to 1.05% by Fold 3. Those numbers are the real value of Walk-Forward.

Beyond Walk-Forward: Dry-Run as a Third Layer of Validation

But even Walk-Forward wasn't enough. Backtests are, at the end of the day, simulations on historical data. I needed an environment that called real APIs, received live data, ran the model's predictions, and went right up to the point of placing orders, identical to production except no actual money on the line. So I built a Dry-Run system.

The 3-Stage Validation Pipeline

My validation framework ultimately settled into three stages:

Stage 1: Walk-Forward (historical data, offline)
  -> "Does it perform consistently across different market environments?"

Stage 2: Dry-Run (live data, no orders)
  -> "Does the model make sound decisions in a live environment?"

Stage 3: Live (live data, real orders)
  -> "Can it actually make money?"

Stage 2, the Dry-Run, fetches live 5-minute candles from the Binance API, computes features, runs the model's predictions, and logs virtual trades to the database as if the system had entered at that point. It's the complete pipeline minus actual order execution.

What Dry-Run Revealed

Running the Dry-Run exposed things that backtests couldn't show.

First, the model's entry filters were too strict. The initial setup included multi-condition filters with MACD requirements, but in real-time, trade signals were too rare, only 1-2 per day. Backtests showed hundreds of trades, but subtle timing differences in the live market caused many signals to miss the filter thresholds.

I ended up minimizing entry filters and removing the MACD condition. The idea was to trust the model's Triple-Barrier prediction itself and strip away auxiliary filters. It was a bold call, but the Dry-Run data backed it up, many of the trades filtered out by MACD had actually been profitable.

Second, there were timeframe data synchronization issues. When the model attempted a prediction before a 4-hour candle had finished forming, it used the previous 4-hour candle's values. In backtests, all candles are complete, so this never happens. Discovering this in Dry-Run led me to adjust feature computation timing.

Third, the prediction latency on the Celeron J4125 became tangible. The model lightweighting discussed in Part 8 (removing TensorFlow, keeping only XGB+LGB) was validated as necessary during the Dry-Run phase. The original model with LSTM took 3-5 seconds per prediction; after lightweighting, it dropped to 0.3 seconds.

If Walk-Forward is "checking the route on a map," Dry-Run is "taking the car out for a test drive on actual roads." Construction zones, traffic light timing, rush-hour congestion. You only discover these by actually driving.

Validation Never Ends

Walk-Forward -> Dry-Run -> Live. Going through these three stages taught me that validation isn't about "passing" a model. It's about "revealing" its flaws. Each stage surfaced problems invisible to the previous one. Walk-Forward revealed overfitting. Dry-Run revealed the friction of live environments. Live trading revealed the psychological pressure.

There's no such thing as perfect validation. But each additional layer of verification adds a small increment of confidence that "this model will work in production." Those small increments are ultimately what give you the fortitude to not shut the system off during live operation.