03. The Rise of the Transformer and the Illusion of 32,687%

Attention Is All You Need

October 27, 2025. After laying the groundwork with log returns in v2.0, I pulled out my next card: Transformer.

What if I applied the Transformer -- the architecture that revolutionized natural language processing -- to time series? LSTMs process data sequentially and gradually forget earlier information. But Self-Attention can directly reference any point in the sequence. It can hold onto the fact that "trading volume exploded 3 hours ago" and reflect that in the current prediction.

The Math Behind Self-Attention

The core of the Transformer is Scaled Dot-Product Attention. The formula is surprisingly concise:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Where:

  • $Q$ (Query): "What information am I looking for?"
  • $K$ (Key): "What information do I have?"
  • $V$ (Value): "The actual information I'll pass along"
  • $d_k$: Dimension of the Key vector

Let's think concretely about what this means for my time series data. Each time step in the sequence is a token. The current time step (Query) computes its relevance to past time steps (Keys), and retrieves information (Values) proportional to that relevance.

The $(i,j)$ element of $QK^T$ is the similarity between time step $i$ and time step $j$. After applying softmax, this becomes a probability distribution -- an attention weight that says "how much should time step $i$ refer to time step $j$'s information for its prediction."

The reason we divide by $\sqrt{d_k}$ is mathematically clear. If the elements of $Q$ and $K$ are independent random variables with mean 0 and variance 1, the variance of the dot product $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$ becomes $d_k$. When $d_k$ is large, the dot product values become extreme, causing the softmax gradients to approach near zero (saturation). Scaling prevents this.

Think of Scaled Dot-Product Attention as a library visit. I walk into a library with the question (Query) "Bitcoin volatility." I compute the relevance between my question and each book title (Key) on the shelves, then read heavily from highly relevant books' contents (Values) and barely skim the less relevant ones. Dividing by $\sqrt{d_k}$ is like a volume knob -- it prevents the relevance scores from becoming so extreme that I end up reading just one book and ignoring everything else.

Multi-Head Attention

My model used 4 heads. Multi-Head Attention performs multiple attention operations in parallel, extracting information from different perspectives:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$

$$\text{where } \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$

$W_i^Q \in \mathbb{R}^{d_{model} \times d_k}$, $W_i^K \in \mathbb{R}^{d_{model} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{model} \times d_v}$ are the linear transformation matrices for each head.

In my system, the 4 heads could each learn different types of temporal relationships. For example:

  • Head 1: Micro-trends over the past few minutes
  • Head 2: Volume patterns around the 1-hour mark
  • Head 3: Interactions between specific technical indicators
  • Head 4: Long-term volatility patterns

Of course, it's hard to interpret exactly what each head actually learns. But the fact that parallel extraction from multiple perspectives produces richer representations than a single attention has been empirically proven.

Positional Encoding

Transformers inherently have no concept of order. Self-Attention operates on sets, not sequences. To provide temporal ordering, we add Positional Encoding to the input:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

$$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

Here, $pos$ is the position in the sequence and $i$ is the dimension index. The sine/cosine periods are set differently for each dimension, giving each position a unique vector.

Why trigonometric functions? Because for any fixed offset $k$, $PE_{pos+k}$ can be expressed as a linear function of $PE_{pos}$. This makes it easier for the model to learn relative positions. In financial time series, the relative time relationship "30 minutes ago" matters more than the absolute time "2:30 PM."

Positional Encoding is like numbering the questions on an exam. The Transformer originally sees all questions at once, but without numbers, it doesn't know the order between question 1 and question 50. The position vectors made from sine/cosine act as a coordinate system that naturally expresses relative distances like "question 3 is two steps after question 1."

That said, in my implementation I used a hybrid architecture stacking Transformer on top of LSTM, so the LSTM was already learning sequential information. I didn't explicitly add separate Positional Encoding. The LSTM's sequential processing itself served as an implicit positional encoding.

# v2.1: Stacking a Transformer layer on top of LSTM
Bidirectional LSTM(128)
  -> Bidirectional LSTM(64)
  -> Multi-Head Attention (4 heads, key_dim=32)  # NEW
  -> LayerNormalization                           # NEW
  -> LSTM(32)
  -> Dense(64) -> Dense(32) -> Dense(2, softmax)

It wasn't just the model architecture that changed. I also significantly expanded the data.

The World of Microstructure

In v2.1, I stepped into the new world of market microstructure features.

Traditional technical indicators like RSI and MACD show "how the price moved." But microstructure features get closer to "why the price moved."

# Microstructure features added in v2.1
new_features = [
    "Order_Imbalance",      # Buy/sell pressure imbalance
    "Price_Acceleration",   # Acceleration of price change (2nd derivative)
    "Volume_Acceleration",  # Acceleration of volume change
    "Price_Efficiency",     # Efficiency of price movement
]

Order_Imbalance measures the imbalance between buying and selling pressure. If the order book is tilted toward the buy side, it means there's upward pressure that hasn't been reflected in the price yet.

Price_Efficiency looks at how "efficiently" the price moved. If it went up in a straight line, that's efficient; if it zigzagged its way up, that's inefficient. Efficient movement signals that the trend has strong momentum.

I also expanded the data from 20,000 to 48,205 samples -- a 2.4x increase. The LSTM had 21 features and XGBoost had 42. The world the model could see had gotten much wider.

60.74%

The training results came in.

Model Accuracy
LSTM 56.47%
XGBoost 60.35%
LightGBM 60.33%
Ensemble 60.74%

60.74%. I had broken through the 60% wall for the first time. Starting from v1.0's 55.58%, I'd made it here through log returns (v2.0) and Transformer + microstructure (v2.1).

In binary classification, 60% is a more meaningful number than it might seem. In a market as noisy as crypto, it means the model is making judgments over 10% better than a coin flip.

But the real shock came next.

32,687%

I ran the backtest. Here were the conditions:

  • Initial capital: 10,000,000 KRW
  • Position size: 20% of current capital
  • Leverage: 1x
  • Long/Short bidirectional trading
  • Stop loss: 10%
  • Commission: 0.1%

When the results printed on the terminal, I read the numbers three times.

Final capital: 3,278,665,832 KRW
Total return: 32,686.66%
Win rate: 55.0%
Sharpe Ratio: 31.23

3.2 billion won. 10 million turned into 3.2 billion. A 326x return. With 1x leverage.

My heart started racing. My brain was saying "this is just a backtest," but my heart was already punching numbers into a calculator. If I put in 100 million, that's 32.6 billion? What about a billion?

I took a moment to collect myself.

The Math of Backtest Returns

Let me break down mathematically how this 32,687% figure was produced.

In a backtest, compound returns are calculated as:

$$\text{Final Capital} = C_0 \cdot \prod_{i=1}^{N}(1 + r_i \cdot w)$$

Where $C_0$ is the initial capital, $r_i$ is the return of the $i$-th trade, $w$ is the position ratio (20% in my case), and $N$ is the total number of trades (9,539). Expressed as total return:

$$R_{total} = \prod_{i=1}^{N}(1 + r_i \cdot w) - 1$$

If it had been simple interest instead:

$$R_{simple} = \sum_{i=1}^{N} r_i \cdot w$$

The magic (or illusion) of compounding lies in the exponential growth of $(1 + x)^n$. With an average return per trade of $\bar{r} = 0.314\%$ and position ratio $w = 0.2$, the capital growth per trade is roughly $1 + 0.00314 \times 0.2 = 1.000628$. Repeat this 9,539 times:

$$1.000628^{9539} \approx 400$$

With simple interest, it would be $9539 \times 0.000628 \approx 5.99$, or about 600% return. The same trade record yields 600% with simple interest and 40,000% with compounding. Compounding amplifies a small edge into a fantastical number. This is the first trap you absolutely must know when reading backtest returns.

Illusion or Skill?

Once the excitement died down, the doubt kicked in. A 326x return -- does this even make sense?

I dug into the trade statistics in detail.

Total trades: 9,539
Win rate: 55.0% (5,245 wins / 4,294 losses)
Average return per trade: 0.314%
Stop-loss triggered: 11 times (0.12%)

0.314% per trade. It wasn't about hitting a jackpot on any single trade -- it was the result of stacking a tiny edge consistently over more than 9,500 trades. A 55% win rate compounding at 0.3% -- yes, mathematically, that's how you get there.

But this was also the illusion of compounding.

Doubt 1: Overfitting

Could the model have "memorized" patterns in the test data? I did split the training/validation/test data chronologically, but a single split can overfit to a specific market regime. I hadn't done Walk-Forward validation.

Statistical Methods for Detecting Overfitting

Overfitting is the gap between "performance on training data" and "performance on unseen data." There are ways to quantify this.

Train-Validation Gap: The most intuitive metric.

$$\Delta_{gap} = \text{Acc}_{train} - \text{Acc}_{val}$$

The Train-Val Gap is like the difference between your mock exam score and your actual exam score. If you scored 95 on the mock (training data) but 80 on the real thing (validation data), that 15-point gap is the size of your overfitting. It means you memorized the mock exam questions rather than truly understanding the material.

The larger this value, the worse the overfitting. As a rule of thumb, $\Delta_{gap} > 5\%$ should raise serious suspicions. My v2.1 model showed about 3-4% -- a gray zone that was warning-level but not immediately dangerous.

Learning Curve Analysis: Gradually increase the training data size and observe train/validation performance.

  • If the two curves converge: sufficiently trained (low variance)
  • If a large gap persists: overfitting (high variance)
  • If both curves are low: underfitting (high bias)

In v2.1, I didn't do any of this systematic analysis. I looked at one number from one split and made my judgment. This was one of the reasons I ended up blindly trusting the 32,687% figure.

Doubt 2: Limitations of Backtesting

There are realities that backtests don't capture.

  • Slippage: In practice, orders don't fill at the price you want
  • Liquidity: Your own orders can move the market
  • Microstructure regime changes: There's no guarantee that past patterns will hold in the future

The assumption that I'm paying a flat 0.1% commission on each of 9,539 trades is optimistic in itself. In live trading, slippage can exceed the commission.

Doubt 3: MDD of 41.2%

The Maximum Drawdown was 41.2%. That means there was a stretch where my capital dropped by 41%.

Mathematical Definition of Maximum Drawdown

MDD is defined as:

$$MDD = \max_{t \in [0,T]} \left( \max_{s \in [0,t]} V_s - V_t \right) / \max_{s \in [0,t]} V_s$$

Written out, it's the largest ratio of decline from the peak up to time $t$ relative to that peak. More intuitively:

$$MDD = \max_{0 \leq s \leq t \leq T} \frac{V_s - V_t}{V_s}$$

Where $V_t$ is the portfolio value at time $t$. An MDD of 41.2% means 41.2% of the capital evaporated from peak to trough.

If you think of MDD in terms of hiking, it measures how far you tumbled from a summit down to the deepest valley. It finds the worst fall along the entire trail. At 41.2%, you've slid nearly halfway down from the peak, and climbing back up requires more effort than the original ascent.

And here lies the trap of asymmetry. To recover from a 41% loss:

$$\frac{1}{1 - 0.412} - 1 = \frac{1}{0.588} - 1 \approx 0.70 = 70\%$$

In other words, lose 41% and you need a 70% gain just to break even. This asymmetry becomes dramatically worse as losses grow. A 50% loss requires 100% gains to recover, and an 80% loss requires 400%.

The standalone LSTM model had an MDD of 84.5%. To recover from that, you'd need $1/(1-0.845) - 1 \approx 545\%$ returns. Without the ensemble, the account would have been effectively destroyed.

Sharpe Ratio: What 31.23 Really Means

The backtest showed a Sharpe Ratio of 31.23. The Sharpe Ratio is defined as:

$$S = \frac{E[R_p - R_f]}{\sigma_p} = \frac{\bar{R}_p - R_f}{\sigma_p}$$

Where $\bar{R}_p$ is the portfolio's average return, $R_f$ is the risk-free rate, and $\sigma_p$ is the standard deviation of portfolio returns.

The Sharpe Ratio measures "how much excess return did you earn per unit of risk."

If you think of the Sharpe Ratio as a restaurant review, it's not just "the food is good" (high returns) -- it's "the food is good for the price" (returns relative to risk). An expensive prix fixe being delicious is expected, but a cheap hole-in-the-wall being mind-blowingly good means a high Sharpe Ratio. An SR of 31.23 is like claiming a $5 lunch spot deserves three Michelin stars -- the kind of number that makes you think "are these reviews fake?"

Generally:

  • $S < 1$: Poor
  • $1 \leq S < 2$: Acceptable
  • $2 \leq S < 3$: Very good
  • $S \geq 3$: Be suspicious

31.23 is a virtually impossible number in reality. Renaissance Technologies' Medallion Fund runs at an annual Sharpe Ratio of roughly 6-7. A value of 31.23 is strong evidence that this backtest result is massively overstating reality.

Analyzing why such an abnormally high Sharpe Ratio emerged:

  1. The extremely high trading frequency (9,539 trades) artificially lowered the variance of daily returns
  2. Idealized backtest assumptions (ignoring slippage, etc.) inflated returns
  3. A single-period backtest failed to account for changing market conditions

What 32,687% Taught Me

I decided to understand this result on three levels.

Level 1: The model actually learned something. A 55% win rate isn't a fluke. If you trade both directions with 60.74% accuracy, profitability over the long run is mathematically inevitable.

Level 2: But the number 32,687% is an illusion. A backtest is a simulation under ideal conditions. You can't reproduce this return in live trading. Long-term backtests where compounding works its magic are especially prone to overestimating results.

Level 3: What matters is the direction, not the magnitude. Even if it were 3,268% or 326% instead of 32,687%, a profitable strategy has value. The real question is whether you've secured a statistical edge in the market.

At this point, I stood at a crossroads.

Path A: Deploy this model live immediately and start making money. Path B: Dig deeper into the doubts and improve the model.

I chose B. And that choice was the beginning of a long journey that would lead through v2.2's failure, v2.3's rebound, and eventually v2.7's complete redesign.

The 60% Accuracy Plateau

After v2.1, a long plateau set in. v2.2 adjusted SMOTE and ensemble weights but accuracy actually dropped to 60.48%. v2.3 barely recovered to 60.67%. v2.4 hit 60.90%. v2.5 fell to 60.16%.

I couldn't break out of the 60% range. I'd spend hours running training sessions to squeeze out a 0.1% improvement, forming and testing countless hypotheses, but the wall wouldn't budge.

I realized later that the wall hadn't refused to break -- I'd been knocking on the wrong wall. Instead of pushing accuracy higher, I needed to change what I was predicting. But that story is for the next series.

Looking Back

v2.1's 32,687% taught me two things.

First, don't be fooled by spectacular numbers. Backtest returns aren't reality. Especially in long-term simulations where compounding works over extended periods, even a small edge gets amplified into astronomical figures. Look at the soundness of the structure behind the number, not the number's magnitude. The Sharpe Ratio of 31.23 was the warning sign. If I had known that even the world's best quant funds struggle to exceed an SR of 7, I would have been skeptical sooner.

Second, skepticism is a virtue. The ability to ask "is this for real?" can be more important than the ability to build models. If I had jumped into live trading right at v2.1, I never would have reached v2.7's genuine performance improvements (73.5% win rate, PF 5.01, MDD 3.62%).

The Transformer was a powerful tool. But no matter how sharp the tool, it's useless if you're answering the wrong question. The real breakthrough wasn't in making the model more complex. It was in changing the question from "will the price go up?" to "will it hit TP first, or SL first?"

But it would take three more months of experimentation and failure before I figured that out.