I built a platform that trains 72 machine learning models to predict the direction of the S&P 500. It runs a daily data pipeline, retrains every model once a month, and backtests every prediction with real transaction costs. It is deployed in production and you can use it right now.
None of the models beat the baseline.
I am publishing that result instead of hiding it, because the way I got there is more interesting than any number I could have reported if they had worked.
What "the baseline" actually is
The baseline is deliberately stupid. It ignores every feature — every volatility measure, every momentum indicator, every macro variable — and always predicts the average return observed during training.
That sounds trivial to beat. It isn't, for one reason: the stock market goes up. Over the 6,641 trading days in my dataset, 54.1% closed positive. A model that always predicts "up" and never looks at anything is right 54% of the time before you even start.
In practice, always predicting a positive average return means holding a long position permanently. So the baseline is, effectively, buy and hold — which is why the title of this piece says so, even though the technical name for it in my code is baseline_mean.
Here is how hard that bar is, measured in Sharpe ratio:
| Training cadence | 1 period ahead | 5 periods | 21 periods |
|---|---|---|---|
| Daily | 0.721 | 0.781 | 0.797 |
| Weekly | 1.620 | 1.724 | 2.168 |
| Monthly | 3.825 | 4.653 | — |
The Sharpe ratio measures return per unit of risk: how much you earned, divided by how much the value of your portfolio bounced around while you earned it, annualized. Higher is better. A Sharpe around 1 is considered good for a real strategy. A Sharpe of 2 is very good.
So why does the monthly baseline show 3.825? Because it trades almost never. It takes one position per month and holds it in an index that drifts upward over time. Few decisions, low measured volatility, permanent exposure to a rising market. That number is not a sign of skill — it is a sign that the metric rewards doing nothing in a market that drifts upward. Any model competing in that slot has to be dramatically better, not marginally better, to justify its existence.
How the models were tested
72 models means nine algorithms — ridge and lasso regression, random forests, XGBoost, a stacking ensemble, LSTM and GRU neural networks, and classifier variants — across eight combinations of training cadence and prediction horizon.
The evaluation method matters more than the algorithms, so here is what it does:
Walk-forward validation. You cannot shuffle financial data and split it randomly. If you train on 2020 and test on 2015, the model has seen the future. So the data is walked forward in time: train on an early block, test on the block immediately after, slide forward, repeat. Every test period comes strictly after the training data that produced the prediction.
A gap between train and test. When you predict the sum of returns over the next 21 days, the target for the last day of training overlaps with the first days of the test period. A model can learn from that overlap without anyone noticing. So the split forces a gap equal to the prediction horizon.
A hard failure, not a warning. Every single fold checks that the last training date is strictly earlier than the first test date, and raises a runtime error if it is not. Not a unit test — an invariant that runs in production. Data leakage is the single most common reason a financial backtest looks brilliant and then loses money, and I wanted it to be impossible to ship by accident rather than unlikely.
Real costs. Every trade is charged 5 basis points of commission and 2 basis points of slippage — 0.07% per operation. These are subtracted inside the Sharpe ratio itself, so a strategy that trades constantly is penalized for it automatically. Assuming you execute at the mid-price is a fantasy, even for the most liquid ETF in the world.
No cherry-picking. When multiple training runs of the same model exist, the code picks between them by run type and timestamp — never by the out-of-sample score. Choosing the run that scored best is how you fool yourself into publishing noise.
Across the 27 non-baseline daily slots, not one model beat the baseline.
The two that looked like they worked
Two slots did come out ahead:
| Slot | Model | Sharpe | vs. baseline | R² out-of-sample |
|---|---|---|---|---|
| Weekly, 1 period | LSTM (default params) | 1.832 | +0.212 (+13%) | −0.038 |
| Monthly, 1 period | Stacking classifier | 3.904 | +0.079 (+2%) | — |
A 13% improvement over the benchmark from a neural network sounds like a finding. It isn't, and there are two reasons why.
The R² is negative. R² measures how much of the variation in the outcome your model explains. Zero means you do no better than always guessing the average. Negative means you do worse than always guessing the average. So this LSTM predicts the size of weekly returns worse than a constant would, while still landing on the right direction often enough to produce a decent Sharpe. That is the signature of a model that has caught a faint directional tilt and nothing else.
The sample is tiny. These are weekly and monthly models, so the out-of-sample windows contain roughly 50 to 100 observations. Not 50,000 — fifty. With that little data, an excess of 2% to 13% over a benchmark is entirely compatible with random sampling variation. If you flip a slightly biased coin a hundred times, you will sometimes get a run that looks like skill.
The monthly case is even weaker: a 2% edge over a baseline whose Sharpe is already 3.825, measured across a few dozen months.
When you evaluate 72 models, some of them will look good by chance. That is not a flaw in the models — it is arithmetic. The honest response is to treat a small edge on a small sample as noise until proven otherwise, not to write a press release about your neural network.
The part that nobody publishes
Here is the finding that surprised me most, and it has nothing to do with predictive accuracy.
I checked what the deployed models actually output, by having each one generate predictions over the last 252 observations. Eight of them produce a single constant value. Standard deviation zero. Same number every day, forever.
The worst case: XGBoost, daily cadence, 21-day horizon. Its nested out-of-sample Sharpe was 0.606 — a perfectly respectable-looking number. But the final model, retrained on the complete dataset for deployment, collapsed. Its L1 regularization parameter was strong enough to flatten every tree into a single leaf once it saw the full data, even though it hadn't done so on the smaller training blocks used during validation.
The lasso models did the same thing, for a cleaner reason: with their tuned penalty, every single coefficient goes to exactly zero on the full dataset. The model is mathematically a constant.
The lesson is uncomfortable and worth stating plainly: out-of-sample validation metrics do not tell you what your deployed model does. The model you validate and the model you ship are trained on different amounts of data, and that difference is enough to change the behaviour completely. Checking the final artifact for degeneracy is not optional. I found this only because I went looking.
So what does this mean?
It means the weak-form efficient market hypothesis held up, which is the boring and expected answer. SPY is among the most liquid instruments on earth. My features are public: technical indicators anyone can compute, macro series anyone can download, news sentiment from a public API. The idea that a few thousand training rows of public data would reveal an exploitable pattern in the most heavily analyzed index in the world was never plausible.
The literature warns exactly about this. Most positive backtests in financial machine learning are artifacts of overfitting or of testing many things and reporting the best one. A pipeline with proper temporal isolation and honest cost accounting produces the result that theory predicts, and mine did.
That is not a failed project. The goal was never to demonstrate abnormal returns — it was to build infrastructure that could measure the question honestly and then actually tell you the answer. A system that reports "this doesn't work" is more valuable than one that reports a beautiful number you cannot reproduce.
The uncomfortable truth about most machine learning results in finance is not that the models are bad. It's that the evaluation is.
STAIR is a research platform built as a final degree project at the Universidade da Coruña. It does not constitute financial advice. Every number in this article comes from the system's own model registry and can be inspected on the platform.