Rule one: don't fool yourself (the backtest is lying)

I generated 200 trading strategies that were pure random noise, with zero skill between them. The best one returned 27% a year for five years. Then it made 3%.

If I’d built that strategy myself, tested it on five years of history, and watched it compound $10,000 into $33,000, I would have believed it. I’d have written a very excited blog post. Then I’d have put real money in and found out what you’re about to see.

Richard Feynman said it better than I can: “The first principle is that you must not fool yourself - and you are the easiest person to fool.” In trading, the tool you fool yourself with is the backtest, and microcaps make it especially easy. Here are the four ways it happens, and what I’m building to guard against each one.

Lie one: you picked the lucky one

Here’s the experiment. Each of the 200 strategies earns a random monthly return that drifts up a little, like the market does, with no skill involved at all. Run them for five years and pick the best one, which is exactly what you do when you tweak a strategy until the backtest looks good. Then keep watching it for three more years. To make sure I wasn’t just showing you one lucky draw, I ran the whole experiment 500 times.

• Backtest winner
• Winner, afterward
• Typical strategy

$10,000$15,000$20,000$25,000$30,000$35,000$40,000

Backtest ends$35,789 Winner, afterward$12,961 Typical strategy

020406080

Simulated. 500 repeats, seed 2026, reproducible with scripts/sims.mjs. X axis is months.

Show the data

x	Backtest winner	Winner, afterward	Typical strategy
0	$10,000		$10,000
1	$10,195		$10,051
2	$10,464		$10,037
3	$10,661		$10,107
4	$10,842		$10,081
5	$11,078		$10,115
6	$11,222		$10,095
7	$11,451		$10,211
8	$11,686		$10,155
9	$11,868		$10,217
10	$12,148		$10,305
11	$12,539		$10,323
12	$12,760		$10,248
13	$13,055		$10,219
14	$13,308		$10,329
15	$13,636		$10,488
16	$13,895		$10,523
17	$14,206		$10,514
18	$14,457		$10,562
19	$14,686		$10,604
20	$15,015		$10,627
21	$15,214		$10,490
22	$15,635		$10,647
23	$15,866		$10,669
24	$16,120		$10,704
25	$16,499		$10,683
26	$16,900		$10,741
27	$17,153		$10,810
28	$17,714		$10,919
29	$17,867		$10,886
30	$18,204		$10,862
31	$18,601		$10,922
32	$18,921		$10,992
33	$19,200		$11,050
34	$19,488		$11,059
35	$19,904		$11,009
36	$20,066		$11,103
37	$20,835		$11,037
38	$21,393		$11,150
39	$21,824		$11,163
40	$22,210		$11,103
41	$22,615		$11,065
42	$23,194		$11,172
43	$23,725		$11,213
44	$24,185		$11,227
45	$24,419		$11,195
46	$24,859		$11,307
47	$25,332		$11,315
48	$25,867		$11,282
49	$26,391		$11,319
50	$27,066		$11,426
51	$27,581		$11,440
52	$28,250		$11,479
53	$28,731		$11,548
54	$29,146		$11,549
55	$29,828		$11,591
56	$30,315		$11,660
57	$30,877		$11,640
58	$31,821		$11,727
59	$32,539		$11,732
60	$32,940	$32,940	$11,780
61		$33,341	$11,816
62		$33,360	$11,818
63		$33,361	$11,854
64		$33,662	$11,858
65		$33,691	$11,967
66		$33,751	$12,015
67		$34,087	$12,091
68		$34,084	$12,138
69		$34,138	$12,185
70		$34,426	$12,148
71		$34,423	$12,294
72		$34,457	$12,307
73		$34,469	$12,351
74		$34,472	$12,388
75		$34,465	$12,496
76		$34,850	$12,605
77		$35,235	$12,664
78		$34,969	$12,657
79		$34,932	$12,697
80		$35,075	$12,640
81		$34,400	$12,722
82		$34,777	$12,695
83		$34,946	$12,701
84		$35,036	$12,816
85		$34,905	$12,863
86		$35,210	$12,896
87		$34,939	$12,909
88		$34,922	$12,933
89		$34,947	$12,893
90		$34,989	$12,819
91		$35,058	$12,820
92		$35,095	$13,073
93		$34,908	$13,065
94		$35,401	$13,029
95		$35,302	$12,883
96		$35,789	$12,961

In the backtest, the winner compounded at 26.9% a year. Afterward it made 2.8% a year, while a typical strategy made 3.3%. The winner never had an edge. It had five lucky years, and I picked it because it had five lucky years. The moment the luck stopped being selected for, it became average.

That’s overfitting, and it’s the default outcome of trying lots of ideas against the same history. Every tweak you make after peeking at the results (“let’s skip January,” “what if the threshold were 7 instead of 5”) is another coin flip, and the backtest quietly rewards you for every one that lands heads.

What I’m doing about it: I write down the question and the test before I look at the answer. A chunk of history is locked away as a holdout, and the system records every time anything touches it. Data in the holdout gets looked at once, at the end. If I tune something after seeing holdout results, that holdout is burned and I have to say so.

Lie two: the dead stocks disappeared

Most stock databases make it easy to grab “all the stocks” and test a strategy on them. The trap is that “all the stocks” often means all the stocks that exist today. The ones that went bankrupt, got delisted, or quietly faded away aren’t there anymore, and they’re exactly the ones your strategy would have lost money on.

In microcaps this is a big deal, because small companies die a lot. Researchers found that when stocks get delisted for bad reasons, the final losses are usually a surprise, and they’re often missing from the data entirely.¹ A follow-up study suggested assuming a -55% return for Nasdaq stocks whose delisting returns are missing.²

Here’s a simulation of how much that matters:

Average annual return, survivors only+6.6%The 594 stocks still listed after 5 years

Average annual return, all stocks+0.7%All 1,000, dead ones included

Stocks that vanished41%Delisted at a -55% final return

Same stocks, same period, same luck. The only difference is whether you counted the dead ones. Leave them out and an asset class that went nowhere looks like it returned 6.6% a year.

What I’m doing about it: the system builds the universe for each historical day from what was listed on that day, including companies that later disappeared. When a stock delists and I can’t find its final price, the backtest assumes a bad outcome, not a neutral one. Missing data is never allowed to make results look better.

Lie three: you knew the future

This one is subtle, and it’s the bug I worry about most. A backtest is a simulation of the past, and it’s shockingly easy to let information from the future leak into it.

A few ways it happens:

• Revised numbers. Companies restate earnings. If your database stores only the latest version, your backtest “knew” the corrected figure on a day when everyone else was looking at the wrong one.
• Timestamps that lie. Something dated Tuesday might not have actually been public until Tuesday night, or Wednesday morning. Trade on it Tuesday at 10 a.m. in your backtest and you’ve given yourself a superpower nobody had.
• Today’s lists. Using today’s index membership or today’s industry labels to decide what you’d have bought five years ago.

Each of these makes a backtest look better, sometimes a lot better, and none of them will show up as an error. The code runs fine. The returns are beautiful. They’re just impossible.

What I’m doing about it: I treat this as the foundational design rule of the whole system. Everything is stored as it looked at the moment it became public, and nothing is ever overwritten. When a figure gets revised, the old version stays, and the backtest only sees the version that existed on the simulated date. Every piece of information carries a “public as of” time that’s computed conservatively. I spent a surprising number of hours on timestamps alone, which I’ll write about in the build log.

Lie four: trading was free

This is the lie that’s specific to microcaps, and it’s the one that kills the most strategies.

As I showed in the last post, the median quoted spread for stocks under $100 million was about 2%, compared to 0.07% for $2-5 billion companies.³ A backtest that buys at the last traded price and sells at the last traded price ignores all of that.

Here’s what it does to an otherwise decent strategy:

$2B - $5B

+26.0%

$1B - $2B

+25.1%

$500M - $1B

+23.1%

$250M - $500M

+18.1%

$100M - $250M

+7.1%

Under $100M

−23.8%

−30%−20%−10%0%10%20%30%

Arithmetic, not data: (1 + 1% - spread)^25 - 1. Spreads from Collver, SEC (2014), stocks priced $10-$19.99.

Show the data

	Net annual return
$2B - $5B	+26.0%
$1B - $2B	+25.1%
$500M - $1B	+23.1%
$250M - $500M	+18.1%
$100M - $250M	+7.1%
Under $100M	−23.8%

Nothing about the strategy changed. Just the toll. And this chart is the optimistic version: it ignores commissions, it ignores the fact that buying pushes the price up in a thin stock, and it assumes you can always trade at the quote.

What I’m doing about it: every simulated trade pays the spread, plus an estimate of how much my own order would move the price, plus a delay between when information becomes public and when I could realistically act on it. Trades are capped at a small fraction of the stock’s typical daily volume, and anything that doesn’t trade enough is out. When I’m unsure what something costs, I assume it costs more.

The uncomfortable part

Put all four guards in place and most backtests die. That’s the point. A strategy that only works in a simulation where you picked the lucky variant, ignored the dead stocks, peeked at the future and traded for free isn’t a strategy. It’s fool’s gold.

I’d rather find that out now, for the price of some electricity, than later for the price of $50,000.

What happens next

The backtesting engine is what I’m building right now. When it produces its first real answer, holdout and all, I’ll publish what it says, even if what it says is “no.” Before that, the next post covers what got built in the first two weeks, including a lot of work that exists purely so the backtest can’t lie to me.

Footnotes

1. Tyler Shumway, “The Delisting Bias in CRSP Data,” Journal of Finance (1997). Abstract. ↩

2. Tyler Shumway and Vincent Warther, Journal of Finance (1999). Abstract. ↩

3. Charles Collver, SEC Division of Trading and Markets (2014). PDF. ↩