04. SMOTE Was Poison

2026-04-22

The Temptation of Class Imbalance

When model v2.1 hit 60.74% accuracy, I was feeling pretty good. The LSTM + XGBoost + LightGBM ensemble was running nicely, and a 32,687% return in backtesting wasn't too shabby either. But one thing bugged me. The Recall for upward predictions was only 55%. It was catching downward moves at 66% just fine, but missing the upward ones.

The cause was obvious. In the training data: 4,883 downward samples vs. 4,668 upward samples. A tiny difference, but the model was reacting sensitively to this imbalance. To quantify the degree of class imbalance, there's a metric called the Imbalance Ratio (IR).

$$IR = \frac{N_{majority}}{N_{minority}} = \frac{4883}{4668} \approx 1.046$$

An IR of 1.0 means perfect balance, and typically you'd start worrying about imbalance at 1.5 or above. At 1.046, it's practically balanced. But I wanted to fix even this tiny imbalance, and the phrase "solving class imbalance" kept swirling around in my head. Naturally, SMOTE (Synthetic Minority Over-sampling Technique) came to mind.

The SMOTE Algorithm: A Mathematical Deep Dive

The core idea of SMOTE (Chawla et al., 2002) is to create new synthetic samples by linearly interpolating between minority class samples. The mathematical definition goes like this.

Given a minority class sample $x_i$ and one of its K-nearest neighbors $x_{nn}$, a synthetic sample $x_{new}$ is generated as follows:

$$x_{new} = x_i + \lambda \cdot (x_{nn} - x_i), \quad \lambda \sim U(0, 1)$$

If I had to compare SMOTE to cooking, it's like taking doenjang-jjigae recipe A and recipe B and mixing ingredients at some random ratio to create a new recipe. The assumption is that since both are doenjang-jjigae, the in-between recipe should also be doenjang-jjigae -- and for data like images, this actually works pretty well. The problem is that with time series, if you blend "yesterday's doenjang-jjigae" with "last month's doenjang-jjigae," the temporal context becomes a total mess.

Here, $\lambda$ is a random number drawn from a uniform distribution between 0 and 1. Geometrically, it means picking an arbitrary point on the line segment connecting $x_i$ and $x_{nn}$. Repeat this process across the entire minority class, and synthetic samples accumulate until they match the count of the majority class.

In domains where each sample is independent (i.i.d.), like image classification or text classification, this is a highly effective technique. But my data wasn't from that kind of domain.

v2.2: Four Improvements All at Once

I got greedy with v2.2. I changed four things simultaneously.

Ensemble weight optimization -- LSTM weight slashed from 35% to 20%
Stronger LSTM regularization -- Dropout from 0.3 to 0.4, added L2 regularization
Learning rate optimization -- Introduced CosineDecay scheduling
SMOTE class balancing -- Minority class oversampling

from imblearn.over_sampling import SMOTE

# Flatten LSTM sequences to 2D and combine with XGB features
X_combined = np.concatenate([X_train_seq_flat, X_train_xgb], axis=1)

smote = SMOTE(random_state=42, k_neighbors=5)
X_combined_balanced, y_train_seq = smote.fit_resample(X_combined, y_train_seq)

# Split back and reshape LSTM input to 3D
X_train_seq = X_train_seq_flat.reshape(-1, 90, 21)

After applying SMOTE, the class distribution went from 16,550:16,229 to a perfectly balanced 16,550:16,550. It looked clean on the surface. But the results were devastating.

The Numbers Tell the Story of Failure

Ensemble accuracy: 60.74% -> 60.48% (-0.26%) Backtest returns: 32,687% -> 24,806% (-24%)

Accuracy only dropped by 0.26%, yet returns plummeted by 24%. That's roughly a 780 million won difference. And that upward Recall improvement SMOTE was supposed to deliver? It barely went from 55% to 56% -- a measly 1% gain. The target was 7%.

What was even more alarming was that the returns drop was 57 times larger than the accuracy drop. The model hadn't just become "slightly wrong" -- it had become "catastrophically wrong at the moments that mattered most." This phenomenon can be understood through the lens of probability calibration. Even with similar accuracy, errors at high-confidence moments have a far greater impact on returns than errors at low-confidence moments. SMOTE had injected noise that disrupted the model's high-confidence regions.

v2.2.1: Trying to Treat a Fracture with a Band-Aid

I didn't want to admit SMOTE was the problem. "It must be the ensemble weights I adjusted wrong." So I built v2.2.1. Bumped the LSTM weight from 20% back to 25% and reduced LightGBM from 35% to 30%. An experiment that should've taken 5 minutes.

# v2.2.1: Weight readjustment attempt
self.lstm_weight = 0.25  # Trying to restore from 0.20
self.xgb_weight = 0.45   # Unchanged
self.lgb_weight = 0.30   # Reduced from 0.35

On paper, it should've yielded 60.67%. The actual result was 60.25%. It got even worse. There was a nonlinear effect caused by the probability threshold (0.5) between a simple weighted average calculation and the actual ensemble prediction, and my intuition had overlooked that.

Ensemble prediction isn't simply a continuous value of $\hat{y} = w_1 \cdot p_1 + w_2 \cdot p_2 + w_3 \cdot p_3$. Since binary classification is determined by whether this value exceeds the threshold of 0.5, small changes in weights can flip samples near the threshold. This nonlinearity is why the "calculated expected accuracy" and "actual accuracy" diverged so significantly.

This failure actually gave me conviction.

The problem with v2.2 isn't the weights -- it's SMOTE. The root cause needs to be eliminated.

Why SMOTE Is Poison for Time Series

Let's look at SMOTE's synthesis formula $x_{new} = x_i + \lambda \cdot (x_{nn} - x_i)$ again. What this formula implicitly assumes is that "the space between $x_i$ and $x_{nn}$ belongs to the same class." In images, the midpoint between two cat photos is still likely a cat. But in financial time series, this assumption breaks down for three reasons.

1. Destruction of Autocorrelation Structure

One of the key characteristics of financial time series is autocorrelation. The lag-$k$ autocorrelation of a time series $\{x_t\}$ is defined as:

$$\rho_k = \frac{\text{Cov}(x_t, x_{t-k})}{\text{Var}(x_t)} = \frac{E[(x_t - \mu)(x_{t-k} - \mu)]}{\sigma^2}$$

Autocorrelation is like "previously on..." in a TV drama. Today's episode (current value) is built on the context of yesterday's episode (past value). If SMOTE interpolates a scene from episode 100 with a scene from episode 500 to create a synthetic scene, you get something that makes no sense -- with the entire middle storyline missing.

In other words, there's a statistical dependency between the current value and the value from $k$ time steps ago. When SMOTE interpolates between $x_i$ (at t=100) and $x_{nn}$ (at t=500), it completely ignores the autocorrelation structure spanning 400 bars between these two points. The synthetic sample becomes a data point that never existed at any point in time, with broken autocorrelation relationships.

2. Future Data Contamination (Look-Ahead Bias)

When SMOTE searches for K-nearest neighbors, it selects the closest neighbors based on Euclidean distance:

$$d(x_i, x_j) = \sqrt{\sum_{f=1}^{F} (x_i^{(f)} - x_j^{(f)})^2}$$

This distance calculation doesn't consider the time axis at all. To augment an upward sample at t=100, an upward sample at t=300 could be selected as the nearest neighbor. The resulting interpolated synthetic sample implicitly absorbs future information. As a result, look-ahead bias seeps into the training data, and the model shows inflated performance in backtesting compared to live trading.

3. Non-Convexity of the Feature Space

Financial market patterns are extremely context-dependent. Take an upward sample $x_i$ with RSI=30 and high volume, and an upward sample $x_{nn}$ with RSI=70 and low volume. The midpoint -- a synthetic sample with RSI=50 and medium volume -- doesn't necessarily represent a meaningful upward pattern. Instead, it becomes ambiguous noise that belongs to neither camp.

Mathematically speaking, this means the class region is not a convex set. For a set $S$ to be convex, for any two points $x, y \in S$, it must hold that $\lambda x + (1-\lambda)y \in S$. The "upward" class in financial data frequently violates this condition.

What Succeeded in v2.2: The Math Behind Regularization

SMOTE was a failure, but the stronger regularization applied simultaneously in v2.2 was a success. LSTM accuracy went from 56.47% to 57.77%, a 1.30% improvement. To understand this success in isolation, we need to look at the mathematical principles behind each regularization technique.

L2 Regularization (Ridge Regularization)

This adds an L2 norm penalty on the weights to the original Cross-Entropy loss function:

$$\mathcal{L}_{total} = \mathcal{L}_{CE} + \lambda \sum_{i} w_i^2$$

Here, $\mathcal{L}_{CE} = -\sum_{c} y_c \log(\hat{y}_c)$ is the standard Cross-Entropy loss, $\lambda$ is the regularization strength (0.01 in our code), and $w_i$ represents all the weights in the network.

L2 regularization is like packing for a trip. Imagine a campsite that charges admission proportional to your backpack weight. The more stuff (weights) you bring, the higher the cost (penalty), so you end up packing only what you truly need. Instead of the model relying on a few heavy items, it's nudged to pack essential things evenly.

The intuition is straightforward. Since larger weights incur larger penalties, the model is encouraged to rely on many small weights that learn general patterns, rather than a few large weights. From a Bayesian perspective, L2 regularization is equivalent to placing a Gaussian prior $w_i \sim \mathcal{N}(0, 1/\lambda)$ on the weights. A larger $\lambda$ expresses a stronger prior belief that weights should be concentrated near zero.

from tensorflow.keras.regularizers import l2
x = Dense(64, activation='relu', kernel_regularizer=l2(0.01))(x)
x = Dense(32, activation='relu', kernel_regularizer=l2(0.01))(x)

The Probabilistic Interpretation of Dropout

Dropout independently "turns off" each neuron at every training step. Mathematically, it's Bernoulli masking:

$$\tilde{h}_i = m_i \cdot h_i, \quad m_i \sim \text{Bernoulli}(1 - p)$$

Bernoulli masking in Dropout is like randomly making team members absent from group projects. Every class (training step), each team member (neuron) has a 40% chance of being absent, so any team that relies on one star player won't survive. Eventually, every member has to build their own skills, and that's the mechanism that prevents overfitting.

Here, $h_i$ is the activation of neuron $i$, $p$ is the dropout probability (0.4 in v2.2), and $m_i$ is a mask variable that's either 0 or 1. In expectation:

$$E[\tilde{h}_i] = (1 - p) \cdot h_i$$

What does this mean? A different subnetwork is activated at each training step. It effectively trains $2^n$ subnetworks simultaneously (Srivastava et al., 2014). At inference time, all neurons are activated but outputs are scaled by $(1-p)$ to match the expected value during training.

Increasing Dropout from 0.3 to 0.4 between v2.1 and v2.2 was intended to suppress LSTM overfitting through stronger regularization. The result was a reduced train-validation accuracy gap and a 1.30% improvement in validation accuracy.

# v2.1: Weak Dropout
x = Dropout(0.3)(x)  # 30% drop

# v2.2: Stronger Dropout
x = Dropout(0.4)(x)  # 40% drop -> better overfitting suppression

Cosine Annealing Learning Rate

Learning rate scheduling directly affects the optimization trajectory. The CosineDecay introduced in v2.2 follows this formula:

$$\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$$

Here, $\eta_{max} = 0.0002$ (initial learning rate), $\eta_{min} = 0.00001$ (minimum learning rate), $T = 10000$ (total decay steps), and $t$ is the current step.

Cosine Annealing is like searching for a great restaurant in an unfamiliar neighborhood. At first, you take big strides to scan the entire area (high learning rate), and once you spot a promising alley, you gradually shorten your steps to pinpoint the exact location (low learning rate). The key is decreasing smoothly like a cosine curve -- if you suddenly halve your stride in a staircase fashion, you might walk right past a great spot.

Why is this schedule better than a fixed learning rate or step decay? According to Loshchilov & Hutter's (2017) SGDR paper, the smooth decrease of the cosine curve helps avoid sharp minima in the loss landscape and converge toward broad, flat minima instead. Flat minima are known to generalize better -- because small perturbations in parameter space don't cause large increases in loss.

from tensorflow.keras.optimizers.schedules import CosineDecay

lr_schedule = CosineDecay(
    initial_learning_rate=0.0002,   # η_max
    decay_steps=10000,              # T
    alpha=0.00001                   # η_min
)
optimizer = Adam(learning_rate=lr_schedule, clipnorm=1.0)

v2.3: Remove SMOTE, and Rise Again

The strategy for v2.3 was simple. "Keep what works, remove what doesn't."

Remove SMOTE (the root cause)
Restore v2.1's ensemble weights (LSTM 0.35 / XGB 0.40 / LGB 0.25)
Keep v2.2's regularization improvements (Dropout + L2 + CosineDecay)

# v2.3: Remove SMOTE, keep the rest
use_smote = False
self.lstm_weight = 0.35  # Restore v2.1 weights
self.xgb_weight = 0.40
self.lgb_weight = 0.25
# Dropout, L2, CosineDecay kept as in v2.2

After 2 hours of training and 10 minutes of backtesting, the results exceeded expectations.

Metric	v2.1	v2.2 (SMOTE)	v2.3 (SMOTE removed)
Accuracy	60.74%	60.48%	60.67%
Returns	32,687%	24,806%	37,970%
Win Rate	54.63%	54.46%	55.10%
Sharpe Ratio	31.23	29.78	32.00
MDD	41.2%	40.88%	40.41%

Accuracy was ever so slightly lower than v2.1, but returns were 16.2% higher. An all-time best. The paradox of accuracy being 0.07% lower while returns are 16% higher is explained by regularization improving the model's distribution of prediction quality.

The stronger regularization in v2.2 reduced LSTM overfitting (56.47% -> 57.75%), resulting in a model that "better captured high-profit trades." Once the noise that was SMOTE was removed, this effect fully materialized.

Lessons in Experimental Design: Variable Control and Elimination

The most painful lesson from this experience was changing multiple things at once. By modifying four things simultaneously in v2.2, it was difficult to isolate that the regularization was a success and SMOTE was a failure. It took a failed hypothesis test (v2.2.1) before I could be confident that SMOTE was the culprit.

In scientific experiments, this is called a controlled variable. You change only one independent variable while holding everything else constant to establish causation. v2.2 was a poorly designed experiment that changed four independent variables at once. Fortunately, through the partial elimination experiment of v2.2.1 (changing only weights) and the decisive elimination experiment of v2.3 (removing only SMOTE), I was able to isolate the cause.

In the end, the natural ablation study that emerged from the three versions can be summarized like this:

Component	v2.1	v2.2	v2.3	Effect
Weights	0.35/0.40/0.25	0.20/0.45/0.35	0.35/0.40/0.25	Restored to v2.1
Regularization	Basic	Stronger	Stronger	Positive
SMOTE	Off	On	Off	Negative
Returns	32,687%	24,806%	37,970%

With time series data, you need to respect the nature of the data rather than reaching for statistical tricks. Just because there's class imbalance doesn't mean you should fill the gap with synthetic data -- that's like stacking fake bricks in the empty spots. It looks convincing from a distance, but crumbles under pressure. Especially applying SMOTE to data with an IR of around 1.046 -- nearly balanced to begin with -- was a case of forcing a solution where there was no problem to solve.

v2.3 etched an important principle into my mind.

Failed experiments have value too. But only if you can precisely isolate the cause of failure.

A 5-minute experiment (v2.2.1) determined the direction for a 2-hour training run (v2.3). Fail fast, learn fast. I think that's a principle that applies to all engineering, not just model development.