02. Birth of the First Model and the First Wall
Before Writing a Single Line of Code
Before I wrote a single line of code, I spent quite a bit of time just studying. This series covers the development process from v1.0 to v2.7, but before all that, there was a preparation phase too tedious to write about in detail.
Fortunately, I wasn't starting from scratch. Experience with time-series ML at my day job was a huge help. Why stationarity matters, why you can't carelessly split train/test in time series, why feature engineering matters more than model architecture -- these were instincts I'd built through hands-on work.
On top of that, I needed to build domain knowledge in finance. Marcos Lopez de Prado's Advances in Financial Machine Learning was a book that changed the way I think. If you're interested in financial ML, I highly recommend it. (Not sponsored.) It taught me how financial ML differs from general ML, how insidiously look-ahead bias can creep in, and introduced me to the concept of Triple-Barrier labeling.
I read a fair number of papers too. Initially, I sought out "predicting stock prices with LSTM" type papers, but most only showed unrealistically good results with no discussion of real-world application. What actually helped were critical studies on the non-stationarity of financial time series and backtest biases. Knowing "why this doesn't work" turned out to be far more useful in practice.
Even so, v1.0 was full of mistakes. There's an unbridgeable gap between studying something and actually doing it. What I've written in this series is just the highlights of the many trials and errors from v1.0 to v2.7. In reality, there were far more missteps that didn't make the cut.
55.58%
October 23, 2025. When the first epoch of my first model finished, this was the number printed on the terminal.
Epoch 1/80
- Train Accuracy: 55.58%
- Validation Accuracy: 55.38%
55.58%. A coin flip is 50%, so my model was 5.58% better than a coin. I didn't know whether to laugh or cry. But at the same time, it was hopeful. If it's this good on the first epoch, wouldn't it get better as training progresses?
I brewed some coffee while waiting for all 80 epochs to finish.
The Architecture of v1.0
v1.0 was the crystallization of naivety. Looking back now, I made several textbook mistakes, but at the time I thought it was the best I could do.
LSTM + XGBoost + LightGBM Ensemble
I decided to combine three models. Each had its own role.
LSTM: Looks at a sequence of 90 candles (7.5 hours) to learn time-series patterns
XGBoost: Captures nonlinear relationships among 33 technical indicators
LightGBM: Ensures diversity through a different boosting approach than XGBoost
I set the ensemble weights to LSTM 35% : XGBoost 40% : LightGBM 25%. The reason XGBoost got the highest weight was the empirical evidence that gradient boosting outperforms LSTM on tabular data.
Feature Composition
# v1.0 LSTM features (18)
lstm_features = [
"Close", "Volume", # Raw price/volume
"RSI", "MACD", "BB_Position", # Technical indicators
"Momentum_5", "Momentum_10", # Momentum
"Return_5m", "Return_15m", # Returns
"Volatility_5", "Volatility_10" # Volatility
# ... etc.
]
See the problem? Close and Volume are sitting right at the top of the feature list. I was feeding raw prices directly into the model. This was the original sin of v1.0.
The First Wall: Non-Stationarity
After training finished and I started analyzing the results, something felt off. The validation accuracy wasn't keeping up with the training accuracy. It was a sign of overfitting.
I started digging for the cause. And I hit the wall of non-stationarity.
The Heart of the Problem
Bitcoin price is a non-stationary time series. Its mean and variance change constantly over time.
# Raw price time series
Close = [30000, 31000, 50000, 45000, 80000, ...]
# Mean? Variance? They keep changing over time.
My model was learning absolute-value patterns like "if the price is $45,000, it goes up." But $45,000 six months ago and $45,000 today exist in completely different market contexts. The patterns the model learned were fake patterns that only worked in the past.
The Mathematical Definition of Stationarity
Let me pause here and clarify what "stationarity" actually means. What time-series analysis calls weak stationarity is a stochastic process satisfying the following three conditions:
Condition 1. The mean is constant:
$$E[X_t] = \mu \quad \text{for all } t$$
The mean must be the same regardless of the time point. Bitcoin's price moved from $30,000 to $80,000. This clearly violates this condition.
Condition 2. The variance is a finite constant:
$$\text{Var}(X_t) = E[(X_t - \mu)^2] = \sigma^2 < \infty \quad \text{for all } t$$
Variance must not change over time either. But Bitcoin's daily fluctuation was $500 in 2023 and sometimes $3,000 in 2024.
Condition 3. Autocovariance depends only on the lag:
$$\text{Cov}(X_t, X_{t+h}) = \gamma(h) \quad \text{for all } t$$
The covariance between time $t$ and $t+h$ must depend only on the lag $h$, not on the time point $t$ itself.
If I were to compare the three conditions of weak stationarity to school exams: Condition 1 says "the average difficulty of every exam must be the same," Condition 2 says "the spread of scores must be similar every time," and Condition 3 says "the correlation between midterms and finals must be the same whether it's the first semester or the second." Bitcoin prices are like an exam where all three rules keep changing.
Raw price data violates all three conditions. Feeding such data directly into an LSTM is like training a game under rules that constantly change. The rules the model learned are already yesterday's rules.
Scalers Aren't Enough
You might push back: "But I normalized with RobustScaler?" I thought the same.
RobustScaler().fit_transform(df['Close'])
# Even after normalization, the absolute values are meaningless
# $30,000 -> $60,000 (100% increase) and
# $60,000 -> $90,000 (50% increase) are
# the same $30,000 increase in dollar terms,
# but they mean completely different things
A scaler only adjusts the range of values; it doesn't resolve the fundamental non-stationarity of the data. Even normalized prices still change in meaning over time.
The Returns Calculation Was Wrong Too
The volatility features were also problematic.
# v1: Standard deviation based on raw prices
Volatility_5 = df['Close'].rolling(5).std()
# $1,000 fluctuation at $30,000 vs $1,000 fluctuation at $60,000
# The former is 3.3%, the latter is 1.7%. The actual volatility differs, but std is the same.
Even simple returns (pct_change) were mathematically incomplete. If you go +10% then -10%, you should return to your original capital, but with simple returns: 100 * 1.1 * 0.9 = 99. 1% just vanishes.
The realization hit me. What the model should be learning isn't "price" but "patterns of change."
v2.0: Introducing Log Returns
I turned that realization into code.
The Key Change: Raw Prices OUT, Log Returns IN
# v1 (simple returns) -- non-stationary, asymmetric
Return_5m = df['Close'].pct_change(1)
# v2 (log returns) -- stationary, symmetric
Log_Return_1 = np.log(df['Close'] / df['Close'].shift(1))
Mathematical Definition of Log Returns
The definition of log returns is simple:
$$r_t = \ln\left(\frac{P_t}{P_{t-1}}\right) = \ln(P_t) - \ln(P_{t-1})$$
To use an analogy, log returns are like measuring "how many floors you went up." Going from floor 1 to floor 2 (100%) or from floor 5 to floor 10 (100%) -- if the ratio is the same, you get the same value. You're measuring relative change, not absolute height (price).
Here $P_t$ is the price at time $t$. Its relationship to simple returns $R_t = (P_t - P_{t-1})/P_{t-1}$ is:
$$r_t = \ln(1 + R_t)$$
When $R_t$ is small (e.g., $|R_t| < 0.1$), by Taylor expansion $r_t \approx R_t$, so the two are nearly identical. But when fluctuations are large, log returns have several mathematically superior properties.
The Problems Log Returns Solve
Let me explain one by one, with math, why log returns are the standard in financial time-series analysis.
1. Additivity
When aggregating returns over multiple periods, simple returns must be multiplied. But log returns can simply be added:
$$r_{t:t+n} = \ln\left(\frac{P_{t+n}}{P_t}\right) = \ln\left(\frac{P_{t+1}}{P_t} \cdot \frac{P_{t+2}}{P_{t+1}} \cdots \frac{P_{t+n}}{P_{t+n-1}}\right)$$
$$= \sum_{k=1}^{n} \ln\left(\frac{P_{t+k}}{P_{t+k-1}}\right) = \sum_{k=1}^{n} r_{t+k}$$
If I compare additivity to LEGO: simple returns are like "assembling blocks through multiplication" -- a complex build -- while log returns are like "just lining blocks up side by side" -- a straight-line assembly. If you want the total change over one hour, you just add up twelve 5-minute log returns and you're done.
Why does this property matter? In my system, the LSTM takes features from 90 time steps as input. With log returns, "the total change over the past hour" can be computed by simply adding the log returns of 12 candles. This linearity makes it much easier for the model to learn patterns.
2. Symmetry
Let's see how log returns solve the asymmetry problem of simple returns:
Price goes from 100 to 110: simple $R = +10\%$, log $r = \ln(1.1) = +0.0953$
Price goes from 110 to 100: simple $R = -9.09\%$, log $r = \ln(1/1.1) = -0.0953$
With simple returns, "going back down by the same amount" gives $+10\%$ and $-9.09\%$ -- asymmetric. But with log returns, it's $+0.0953$ and $-0.0953$ -- perfectly symmetric. Mathematically:
$$\ln\left(\frac{P_t}{P_{t-1}}\right) = -\ln\left(\frac{P_{t-1}}{P_t}\right)$$
This symmetry enables balanced learning of Long/Short directions in a binary classification model.
3. Achieving Stationarity and the ADF Test
Converting to log returns brings a non-stationary time series closer to stationarity. The way to verify this statistically is the ADF test (Augmented Dickey-Fuller Test).
The regression model for the ADF test is:
$$\Delta y_t = \alpha + \beta t + \gamma y_{t-1} + \sum_{i=1}^{p} \delta_i \Delta y_{t-i} + \epsilon_t$$
Where:
$\Delta y_t = y_t - y_{t-1}$ (first difference)
$\gamma$ is the key parameter being tested
$\sum \delta_i \Delta y_{t-i}$ are lag terms to absorb higher-order autocorrelation
Null hypothesis $H_0$: $\gamma = 0$ (a unit root exists, i.e., non-stationary) Alternative hypothesis $H_1$: $\gamma < 0$ (no unit root, i.e., stationary)
Running the ADF test on raw prices typically gives $p > 0.05$, failing to reject the null hypothesis (non-stationary). But running the ADF test on log returns gives $p \approx 0$, rejecting the null hypothesis (stationary). This statistically confirms my intuition that "log returns are a stationary time series."
The ADF test is similar to a blood test during a health checkup. You're running a test asking, "Does this time series suffer from the disease called a unit root?" If the p-value is low, the diagnosis is "No, it's healthy (stationary)." If it's high, then "Non-stationarity suspected -- treatment (differencing/log transformation) is needed."
4. Connection to the Central Limit Theorem (CLT)
Why "log" specifically? This connects to the Central Limit Theorem.
Think about the price formation process. The price at time $T$ is the result of each period's returns being multiplied together:
$$P_T = P_0 \cdot \prod_{t=1}^{T}(1 + R_t)$$
Taking the log of both sides:
$$\ln(P_T) = \ln(P_0) + \sum_{t=1}^{T} \ln(1 + R_t) = \ln(P_0) + \sum_{t=1}^{T} r_t$$
If the log returns $r_t$ for each period are independent and identically distributed (i.i.d.), then by the CLT:
$$\frac{\sum_{t=1}^{T} r_t - T\mu}{\sigma\sqrt{T}} \xrightarrow{d} N(0,1) \quad \text{as } T \to \infty$$
In other words, the sum of log returns over a sufficiently large number of periods converges to a normal distribution. This is the theoretical basis for assuming log returns follow a normal distribution in financial modeling, and the Geometric Brownian Motion underlying the Black-Scholes model is also built on this assumption.
5. Black-Scholes and Geometric Brownian Motion
Let me take a brief detour to mention just how fundamental log returns are in financial theory.
In the Black-Scholes model, asset prices are assumed to follow Geometric Brownian Motion (GBM):
$$dS_t = \mu S_t \, dt + \sigma S_t \, dW_t$$
where $\mu$ is the expected return, $\sigma$ is the volatility, and $W_t$ is a Wiener process. Applying Ito's Lemma:
$$d(\ln S_t) = \left(\mu - \frac{\sigma^2}{2}\right) dt + \sigma \, dW_t$$
If I use a bar analogy for GBM and Ito's Lemma: the original stock price movement (GBM) is like "a drunk person walking on a slope" -- drift (direction) and staggering (volatility) are all mixed together. Ito's Lemma is the mathematical tool that puts a "log lens" on this staggering path, converting it into a random walk on a flat surface. It becomes much easier to analyze.
That is, the log price follows an arithmetic Brownian motion with drift and diffusion. This is equivalent to saying log returns follow a normal distribution:
$$r_t = \ln\left(\frac{S_t}{S_{t-1}}\right) \sim N\left(\left(\mu - \frac{\sigma^2}{2}\right)\Delta t, \, \sigma^2 \Delta t\right)$$
Of course, real financial market returns exhibit fatter tails than a normal distribution. It's not a perfect model. But the essential property of log returns -- measuring "relative change independent of price level" -- remains valid even when the theoretical assumptions aren't perfect.
My switch to log returns in v2.0 wasn't just because "people say it improves performance." It stood on this mathematical foundation.
Complete Feature Overhaul
# v2.0 LSTM features (16) -- raw prices completely removed
lstm_features = [
"Log_Return_1", # 5-min log return
"Log_Return_5", # 25-min
"Log_Return_12", # 1-hour
"Log_Volume_Change", # Volume rate of change
"RSI", "BB_Position", "ATR_Percent",
"MACD_Normalized", # MACD divided by ATR (NEW)
"Volatility_5", # Volatility based on log returns (NEW)
# ...
]
Close and Volume disappeared. In their place came Log_Return_1 and Log_Volume_Change. What the model sees is no longer "Bitcoin is at $45,000" but "it went up 0.3% in the last 5 minutes."
I also normalized MACD by ATR. This made it possible to measure momentum strength regardless of price level. Mathematically:
$$\text{MACD\_Normalized} = \frac{\text{MACD}}{\text{ATR} + \epsilon}$$
Here $\epsilon = 10^{-8}$ is a small constant to prevent division by zero. Since ATR (Average True Range) is itself a measure of volatility, dividing MACD by ATR gives you "the relative strength of momentum compared to current volatility."
So What Were the Results?
When I saw v2.0's first epoch results, I nearly jumped out of my chair.
v1.0 Validation Accuracy: 55.38%
v2.0 Validation Accuracy: 57.35% (+1.97%)
You might think 1.97% is nothing. But in binary classification, 2% is significant. Especially in a domain as noisy as financial data, a 2% improvement at the first epoch is a strong signal that you're on the right track.
And what mattered more than the numbers was this: the very nature of what the model was learning had changed. Instead of fake patterns dependent on absolute price levels, it was now learning relative change patterns that are invariant across time.
Looking Back
The transition from v1.0 to v2.0 can be technically summarized in one line: "I replaced raw prices with log returns." But the lesson I learned went much deeper than that.
In financial time series, how you represent your data is just as important as how you design your model architecture -- maybe even more so. No matter how complex an LSTM you stack, if you feed it non-stationary data, you get non-stationary results.
The concept of log returns is the most basic of basics in financial theory. Both the Black-Scholes model and the GARCH model use log returns. But because I started from the machine learning side, I missed this fundamental. You need to understand both the language of ML and the language of finance. That's the biggest lesson I learned between v1.0 and v2.0.
v2.0 was just the starting line. I had cleared the first wall of non-stationarity, but higher walls lay ahead. The temptation of Transformers, the fantastical number of 32,687%, and the fundamental question: "If accuracy is high, can you actually make money?"