samar-gulati/Algorithm-tester
A Python framework for testing trading strategies against the ways backtests mislead: look-ahead audits, matched-exposure controls, and block-bootstrap significance tests. The tester is itself test...
0 stars · 0 forks · Python
README
Algorithm-tester
A small Python framework for testing trading strategies honestly — built because the backtests I used were not accurate enough. I'm a high-school student who runs a systematic trading strategy on a paper-trading account. Early on, I realised backtest harnesses I used were both naive and did not account for enough variables and extremes. The problem wasn't the strategy — it was me. A backtest is a machine for telling you what you want to hear. So I built the thing this repo contains: a testing pipeline whose whole job is to kill bad ideas before they cost money — including, repeatedly, my own. Three of those funerals are documented in case_studies/, with every number reproducible from the code here. (The live strategy itself is private and intentionally not described anywhere in this repository — the framework is tested against a deliberately generic textbook demo instead.)
Contents
The idea in 30 seconds The three guardrails Quickstart The demo strategy Case studies: three ideas this pipeline killed What's in the box Honest limitations Never used GitHub before?
The idea in 30 seconds Every strategy idea goes through the same gauntlet, in order:
flowchart LR A[idea] --> B["1 backtest<br/>(look-ahead-safe engine)"] B --> C["2 look-ahead audit<br/>(corrupt the future, re-run)"] C --> D["3 matched-exposure control<br/>(beat a brainless twin, not the base)"] D --> E["4 block bootstrap<br/>(does the edge survive resampling?)"] E --> F["5 leave-one-crisis-out<br/>(or is it one episode in disguise?)"] F --> G{verdict} G -->|rare| H[ACCEPTED] G -->|usual| I[REJECTED]
Loading
Most ideas die. That is the point. A pipeline that never rejects anything isn't testing — it's decorating. The three guardrails These are the framework's contribution — each one targets a specific way backtests lie:
Guardrail The lie it catches Module
Look-ahead audit — corrupt every price on/after the decision date with noise, re-ask the strategy for weights; any change proves it peeked "My strategy is brilliant" (it read tomorrow's newspaper) backtest_lab/audit.py
Matched-exposure control — compare a filter against a twin holding the same average exposure constantly, with zero timing "My safety filter cuts drawdowns" (it just holds more cash; a coin flip would too) backtest_lab/controls.py
Block-bootstrap significance — resample paired return blocks 2,000× and demand the metric difference excludes zero; plus a leave-one-crisis-out check "It beat the benchmark in both halves" (two lucky draws; the whole edge was 2008) backtest_lab/significance.py
The engine itself enforces a fourth, quieter honesty rule: strategies receive a DataView that only exposes prices strictly before the decision date, entry costs are charged from day one, and leverage/shorts are rejected — so the easy ways to cheat are structurally unavailable. Who tests the tester? The guardrails above judge strategies — but why trust the judge? A second layer tests the framework itself:
Layer-2 check What it proves Where
Property fuzzer — 40 randomized markets (random drifts, vols, crash days) × random strategies, run through the engine Six invariants hold no matter the input: finite returns, weight/gross/turnover bounds, costs can only hurt, and every honest strategy passes the look-ahead audit tests/test_fuzz_engine.py (runs inside the 73-test suite and in CI)
Mutation test — plant four realistic bugs in the engine, one at a time: a cost sign flip, a free first trade, a one-day look-ahead, a disabled leverage guard The test suite catches every planted bug — a green suite that can't turn red proves nothing. Current score: 4/4 killed, including the look-ahead audit catching a corrupted version of the very engine it ships with mutation_test.py (runs in CI on every push)
Quickstart Requires Python 3.10+. Install (clone first — you'll want the examples and tests too): git clone https://github.com/gulat70984-del/Algorithm-tester.git cd Algorithm-tester pip install -r requirements.txt # or: pip install -e . (installs backtest_lab as a package) Or install the library alone, straight from GitHub, no clone needed: pip install git+https://github.com/gulat70984-del/Algorithm-tester.git (While this repository is private, the direct install requires GitHub authentication on your machine; after cloning, pip install -e . needs nothing extra.) Run everything: python -m pytest tests/ -q # 73 tests incl. a property fuzzer — offline, seconds python mutation_test.py # plant 4 real bugs in the engine; all must be caught python examples/run_demo.py # backtest the demo strategy (downloads + caches data) python examples/vet_a_filter.py # run a real candidate idea through the full pipeline The same checks run automatically on every push via GitHub Actions. Full function-level documentation: docs/api.md. Writing your own strategy is one function — anything callable that maps a data view to weights: from backtest_lab.engine import run
def my_strategy(view): px = view.prices # ONLY data strictly before the rebalance date if len(px) < 127: return {} # not enough history yet -> stay in cash momentum = px.iloc[-1] / px.iloc[-127] - 1.0 best = momentum.idxmax() return {best: 1.0} # long-only weights, must sum to <= 1
result = run(prices, my_strategy, cost_per_side=0.0020, cash="SHY") print(result.returns.tail()) # daily net returns; also .weights, .gross, .turnover The demo strategy The framework needs something concrete to test, so it ships with a deliberately generic textbook strategy (strategies/demo.py): monthly, hold the top 2 of 5 asset-class ETFs by 3-month momentum, only while above their 200-day average, equal-weighted, idle cash in short-term Treasuries. These are illustration parameters from the public tactical asset allocation literature — not a live configuration, and not tuned to look good. The point of this repo is the tester, not the demo's performance.
net of costs, 2007–2026 CAGR Sharpe max drawdown
demo strategy 8.7% 0.65 −23.2%
S&P 500 buy & hold 11.0% 0.63 −55.2%
An honest reading, because that's the house style: the plain textbook demo gives up return versus buy-and-hold in exchange for less than half the worst-case loss — and these are backtest numbers (see limitations). Every figure regenerates from python examples/run_demo.py; the numbers land in results/demo_headline.txt. Case studies: three ideas this pipeline killed Real ideas, tested with real rigor, rejected by their own evidence:
# The idea How it died Writeup
1 "Don't buy near the 52-week high" (my own idea — it felt safer) Halved returns without reducing drawdown; a brainless twin with the same average exposure beat it everywhere, and the bootstrap confirmed the damage was real, not noise case study 1
2 Vol-targeting & turbulence "crash protectors" (both literature-backed) Their entire apparent protection was just holding more cash — the matched twin did the same thing better with zero intelligence case study 2
3 Absolute-momentum floor — the one that passed the first four gates Leave-2008-out flipped its in-sample edge negative; the bootstrap showed its out-of-sample edge reverses under resampling. Two lucky draws, not a mechanism case study 3
Case study 1 can be re-run end-to-end: python examples/vet_a_filter.py prints the pipeline's full reasoning and verdict (saved artifact). What's in the box backtest_lab/ the framework engine.py monthly-rebalance engine; DataView look-ahead protection; cost accounting metrics.py CAGR, Sharpe, Sortino, max drawdown, Calmar, comparison tables audit.py guardrail 1: the look-ahead audit controls.py guardrail 2: matched-exposure twins significance.py guardrail 3: paired block bootstrap + leave-one-window-out data.py cached data loading (reproducible, offline after first download) report.py charts + text artifacts (every README number traces to a file) strategies/demo.py generic textbook demo strategy + two real (rejected) candidate filters examples/ run_demo.py, vet_a_filter.py — both fully reproducible tests/ 73 pytest tests, all synthetic/offline (the guardrails are themselves tested: a deliberately cheating strategy must be caught, a pure de-grossing filter must not beat its twin, a fake single-window edge must be exposed) test_fuzz_engine.py property fuzzer: 40 randomized markets x random strategies, asserting 6 engine invariants (finite returns, weight/gross/turnover bounds, costs-only-hurt, honest strategies always pass the look-ahead audit) mutation_test.py who tests the tests? plants 4 realistic bugs in the engine one at a time - the suite must catch every one (current score: 4/4 killed) case_studies/ the three funerals, with numbers docs/api.md function-level API reference results/, assets/ generated artifacts and charts
Honest limitations
Backtests ≠ live results. Costs are modeled as a flat 20 bps/side; there's no slippage model, no intramonth cash drag, no taxes. Real-world results would be somewhat worse. One rebalance frequency, long-only. The engine is monthly and unlevered by design (that's what it was built to test). It is not a general event-driven backtester — use zipline/vectorbt/backtesting.py if you need one. The bootstrap is a noise filter, not an oracle. Passing it means "not obviously luck", never "guaranteed edge". The framework's asymmetry is deliberate: it's much better at rejecting bad ideas than certifying good ones. Survivorship note: the demo's ETF universe is fixed and all funds still exist, so the usual delisting bias is limited here — but apply this framework to individual stocks and you must handle it yourself.
Never used GitHub before? No problem — nothing here requires programming to evaluate. The two charts above summarize the demo; the case studies are short plain-English documents; and each blue file link shows code with comments explaining what it does. The one-sentence summary of this repo: I built the machine that stops me from fooling myself, and I'm showing you the ideas it caught — including mine. Author Built by a high-school student as part of a systematic-trading research project (paper-trading only; no real money is managed by this code, and no live strategy's rules appear in this repository). License & disclaimer MIT. Nothing in this repository is investment advice; it is an educational research tool. Backtested performance does not predict future results.