Home
/
Today's Focus
Backtesting and Statistical Testing
Hard
·
22 min read
Statistical / ML for Quants
Backtesting
Sharpe Ratio
Multiple Hypothesis Testing
Overfitting
1
Article
2
Notebook
3
Quiz
Quick Quiz
1.
Daily excess returns have mean
r
ˉ
=
0.05
%
\bar r=0.05\%
r
ˉ
=
0.05%
, sd
σ
^
=
1
%
\hat\sigma=1\%
σ
^
=
1%
over
T
=
1260
T=1260
T
=
1260
days. Assuming Gaussian returns, the annualised Sharpe and the t-statistic for
H
0
:
SR
=
0
H_0:\text{SR}=0
H
0
:
SR
=
0
are:
SR = 0.79; t
≈
\approx
≈
1.26
SR = 0.79; t
≈
\approx
≈
2.00
SR = 0.79; t
≈
\approx
≈
1.77
SR = 1.26; t
≈
\approx
≈
2.00
2.
Testing 100 strategies, you apply Benjamini-Hochberg at FDR = 5%. What is the BH threshold for the
k
k
k
-th ranked p-value, and what does BH guarantee under the global null?
Threshold
k
×
0.05
/
100
=
k
/
2000
k\times0.05/100=k/2000
k
×
0.05/100
=
k
/2000
; BH keeps the false-discovery rate
≤
5
%
\le5\%
≤
5%
Threshold
k
×
0.05
/
100
k\times0.05/100
k
×
0.05/100
; under the null BH rejects exactly 0
Threshold
k
/
(
100
×
0.05
)
=
k
/
5
k/(100\times0.05)=k/5
k
/
(
100
×
0.05
)
=
k
/5
; expect 5 rejections under the null
Threshold
k
/
(
100
×
0.05
)
k/(100\times0.05)
k
/
(
100
×
0.05
)
; expect 100 rejections since all tests are run
3.
A 3-year backtest reports SR = 2.5 after trying 80 parameter sets with average pairwise correlation 0.6. Estimating the effective number of independent tests, what should you conclude?
N
eff
≈
1.6
N_{\text{eff}}\approx1.6
N
eff
≈
1.6
, so the search is harmless and SR = 2.5 is automatically credible
N
eff
=
80
/
(
1
+
79
×
0.6
)
≈
1.6
N_{\text{eff}}=80/(1+79\times0.6)\approx1.6
N
eff
=
80/
(
1
+
79
×
0.6
)
≈
1.6
: essentially one strategy, so 2.5 still needs the full sample to clear significance
N
eff
=
80
N_{\text{eff}}=80
N
eff
=
80
; SR = 2.5 is credible as it exceeds typical noise
N
eff
=
80
N_{\text{eff}}=80
N
eff
=
80
; expected max SR
≈
2
ln
80
≈
3.0
\approx\sqrt{2\ln80}\approx3.0
≈
2
ln
80
≈
3.0
, so 2.5 is not significant
4.
Simulating a strategy that rebalances at the close using only that day's closing prices (no later data) is automatically free of look-ahead bias.
True
False
5.
The Deflated Sharpe Ratio needs
ρ
ˉ
\bar\rho
ρ
ˉ
(average pairwise correlation) in addition to
N
N
N
. What does
ρ
ˉ
\bar\rho
ρ
ˉ
capture that
N
N
N
alone does not?
It is needed for the Shanken errors-in-variables correction
It corrects for correlated strategies sharing training data, which inflates the backtest SR
It sets the effective number of independent tests; higher
ρ
ˉ
\bar\rho
ρ
ˉ
lowers the max SR by chance
It measures the strategies' correlation with the market, controlling for beta
6.
A walk-forward backtest gives OOS Sharpe 0.8 over 5 years (IS Sharpe 1.5). Before calling the strategy viable, what should a rigorous quant additionally do?
OOS t-stat, autocorrelation, sub-period, net-of-cost SR, and DSR over trials
Verify IS SR > OOS SR, which confirms there is no look-ahead bias
Only compute the t-statistic; walk-forward already controls multiple testing
Nothing — OOS SR > 0 is sufficient to declare it viable
Submit
←
Notebook