DESIGN — QU100 experimentation substrate (A/B framework)

Source: DESIGN-qu100-ab-testing.md · Rendered: 2026-07-04 06:07 UTC · agents read the .md, humans read the .html.

1. The problem (plain English)

Every day, rainier scores ~100 QU100 stocks and feeds the top few to an LLM. The score blends money-flow rank, sector, and chart pattern with hand-set weights — and the first portfolio review caught them pointing the wrong way (bought nothing while the market rose). We want to change those weights, but today the only way is to edit config and push it live: a blind flip. If it's worse, we find out weeks later, with no way to roll back or prove cause.

We froze the current config as v0.2.0 — the "champion." We lack a way to take a proposed change (a "challenger"), measure it head-to-head on real history, and promote it only on a measured win.

Two things make this more than a one-off:

  1. The same measurement machine is needed twice. A separate auto-research engine (DESIGN-qu100-llm-backtest, src/rainier/research/) is being built to search for better LLM strategies. It needs the exact same parts — a reward registry, a scoring harness, an archive, walk-forward validation. Today those are empty stubs. Building them separately for the screener and the LLM search guarantees drift (we already hit this: the old qu100_portfolio backtest diverged from the live ranker and measured the wrong thing).

  2. We want to experiment at three grains, not one. Swap a whole config; vary one signal's computation (e.g. how "momentum" is calculated); and try different objectives (total return vs Sharpe vs dodged-loss). A whole-config-only A/B can't do the latter two.

So this design is one generic experimentation substrate that both the screener and the auto-research engine ride on.

2. How it works today

config/settings.yaml ─► StockScreenerConfig ─► screen_stocks() ─► top-N ─► LLM ─► Discord
   (one config, hand-edited, hot-reloaded each scan)

3. What goes wrong

operator edits a weight ─► pushes live ─► ??? ─► maybe worse, found out late
                                          ▲  no baseline · no measurement · no rollback · no record
two engines (screener A/B  ▼  built separately) ─► reward registry / archive / CV duplicated ─► DRIFT
  1. No way to run a config without going live — scoring needs a faithful replay of the real ranker; none exists.
  2. A config isn't a versioned, comparable, revertible object.
  3. No pluggable rewards (can't ask "best on dodged-loss?"), no parameter-level experiments (can't isolate one knob), no record of winners/losers.
  4. Duplicating the substrate across screener + auto-research → divergence.

4. The fix — one generic substrate

A config (or an LLM skill) is a candidate. Score candidates on the same history, by a chosen reward, under honest validation. Promote on a measured win. Everything is candidate-agnostic so the screener and the LLM search share it.

            ┌──────────────────── shared substrate (built ONCE) ─────────────────────┐
 candidate  │  screener-config ──┐                                                     │
 (champion/ │  LLM skill        ─┼─► evaluator ─► BASE scorecard ─► results-registry    │
  chall.)   │                    │   screener: cheap, NO LLM        + walk-fwd validation│
            │  reward registry: pluggable objective(s), tagged by input type,           │
            │                    role = primary | guardrail | secondary                  │
            │  experiment spec (YAML): champion + challengers + which knob + reward keys  │
            └────────────────────────────────────────────────────────────────────────┘
              candidate #1 = screener-config (cheap, proves the substrate)
              candidate #2 = LLM-skill (expensive; reuses everything)

4.1 Candidate abstraction

A candidate is anything scorable: a StockScreenerConfig (#1) or an LLM skill (#2). The champion is a candidate (config/model/champion.yaml); challengers differ from it by a declared delta. The evaluator and registry key on a candidate id + type, never on the candidate's internals — so adding a new candidate type doesn't touch them.

4.2 Pluggable reward functions — copy Eppo's "fact + aggregation + role"

A reward = a fact (the scored data) + an aggregation/objective (total return, Sharpe, hit-rate, drawdown-adjusted, dodged-loss…), declared as config-as-data, decoupled from the engine [Eppo docs]. Roles are declarative: primary (the objective being optimized), guardrail (is_guardrail: true + direction + threshold — a challenger that wins on primary but breaches a guardrail like max-drawdown or turnover is not promoted), secondary. Ratio objectives (return/drawdown, return/turnover) compose two aggregations [Eppo ratio metrics].

Input-typed registry. Rewards are tagged by the candidate type / input shape they accept: screener rewards consume BasketOutcomes (per-day basket + forward returns); LLM rewards consume TradeRecord series. The registry refuses to score a candidate with a reward registered for a different input type (no silent mis-scoring). This generalizes the as-built research/rewards/ stub (whose TradeRecord→float is the LLM-input case).

4.3 Parameter-level experiments — Statsig "layers" + PlanOut "experiment-as-spec"

A layer is one contended knob (e.g. how momentum is computed). A challenger varies that one parameter, read as config; mutual exclusion keeps concurrent knob-experiments from entangling [Statsig Layers]. Each experiment is a versioned YAML spec — champion + challengers, the knob(s) overridden, which reward keys are primary/guardrail, the promotion gate — that the replay engine interprets; no engine code per experiment [PlanOut]. A new experiment (or your momentum A/B) is a YAML file, not a code change.

4.4 Replay + score evaluator (screener-first; composes the shipped pattern replay)

For each trading day t in the corpus, replay the live ranker as-of t (_screen_money_flow_as_of + as-of sector via the shipped analyze_sectors_at + as-of capital-flow + the shipped pattern_replay chain emission_at → replay_composite → replay_screen), emit the selected basket + forward returns at H∈{5,10,20}, and reduce to a base scorecard via the chosen reward(s). Net-new here is the as-of money-flow leg, the corpus driver, and BasketOutcomes emission — the pattern layer and forward-return machinery already exist (paper/pattern_replay.py, paper/pattern_audit.py). Screener scoring uses no LLM (cheap → proves the substrate).

Base scorecard + optional extension (resolves the schema collision). Both evaluators emit a candidate-agnostic base scorecard — candidate_id, candidate_type, window, n_selection_days, corpus_hash (§4.5), the primary/guardrail/secondary reward values, per-regime sub-scores (regime_scores: reward values sliced by the corpus's regime tags — 13 months is mostly one regime, and a challenger can win overall while losing the minority regime; slices also enable an optional "no regime bucket below X" guardrail), deflated_sharpe (emitted as null until the §11-task-6 DSR spec lands — never a naively computed number), evaluator_sha. LLM-specific fields (valid_thesis_rate, cost_usd, filled_R, filled_rate, tqqq_bh_R, skill_yaml_sha) move to an optional LLM extension, not required of the screener evaluator. As-built, output_schema.py validate() is already generic over named schemas (cost_pilot, survivorship, backtest) — so this is additive: register a new base-scorecard schema + an optional LLM-extension schema in output_schema.yaml, plus a composition helper; strict validate() is untouched and the LLM-shaped backtest entry stays as-is for its existing consumer (§6).

4.5 Results-registry + statistical rigor

Co-evaluation rule (no stored-score comparisons): every evaluator run re-scores the champion alongside its challengers over the same corpus snapshot. The scorecard carries a corpus_hash (content hash of the (symbol, date, close) rows scored), and the registry refuses champion-vs-challenger comparisons across mismatched hashes. Stored scores (champion.yaml score:, registry rows) are provenance, never a comparison baseline — the corpus is a moving window, so comparing a fresh challenger against a frozen champion number is apples-to-oranges by construction. Screener evaluation is no-LLM, so re-scoring the champion every run is nearly free.

Every run (winner and loser) is appended to a results-registry — (candidate_id, candidate_type, reward, window, base-scorecard) — a flat append-only Parquet store (feature-store convention; not Neon — avoids the two-DATABASE_URL footgun). This is distinct from the auto-research engine's research/archive/ (an L4 MAP-Elites quality-diversity store that keeps the elite per behavior-niche and discards dominated entries): the MAP-Elites archive is the LLM-search bandit's concern and is fed from the registry later — the substrate ships the registry, not the archive.

Validation is walk-forward with a purged embargo + a locked holdout (built generically here). Efficiency toggles — CUPED/MLRATE regression adjustment, regime stratification (source: compute_market_regime, SPY vs 200-SMA, llm_thesis/research.py), sequential testing — are configurable, not engine rewrites [Statsig/MLRATE]. Caveat (load-bearing): those guarantees assume iid randomization over users; we randomize over autocorrelated time, so they don't transfer unmodified — see §5.

What this is NOT (this build)

5. The promotion bar — OPEN (flagged, provisional interim)

This section is the explicit research gap. The grounding research established the registry + layers + spec patterns, but not the trading-overfitting math. The honest bar for ~250 overlapping days, when scoring many challengers, requires methods we must specify in a committed follow-up: Deflated Sharpe Ratio and Probability of Backtest Overfitting (Bailey & López de Prado), White's Reality Check / Hansen's SPA for multiple-strategy selection, purged/combinatorial cross-validation, and stationary block bootstrap (Politis-White) for the effective-sample-size shrinkage that overlapping holding horizons cause. Web-A/B variance tricks (CUPED/MLRATE, always-valid p-values) assume iid-over-users and break under autocorrelated-over-time data — their analogue is block/cluster-robust resampling.

Interim provisional bar (accepted 2026-06-27; NOT final — replaced by the §11-Task-6 follow-up): - Split the ~13-month corpus: select/tune challengers on the first ~10 months; hold out the trailing ~3 months (~63 trading days) untouched. - Primary reward on the held-out window must beat champion by a margin, with no guardrail breach. - n ≥ 40 selection-days is a RAW-count gate, not an effective-sample-size gate. With H=20 overlapping horizons, ~63 holdout days give only ~3 independent blocks — so this raw count is explicitly a placeholder until the DSR / stationary-block-bootstrap ESS gate lands. - A challenger that only wins in-sample is discarded. - Number of challengers scored is recorded so the deflation correction (DSR/PBO) applies once specified — selection across many trials is the central overfitting risk on a short history.

6. Integration with the auto-research engine (as-built seams)

The engine's parts exist as stubs; we build them generically here. Concrete seams:

Part As-built This design
research/rewards/ register(name, fn) + empty REGISTRY, TradeRecord→float becomes a decorator factory @register(name, input_type, role, …); TradeRecord is the LLM-input case; add BasketOutcomes screener rewards
research/output_schema.yaml + output_schema.py generic validate()/format_block() over named schemas; only the backtest schema is LLM-shaped ADD a base-scorecard schema + optional LLM extension (additive new schema entries + a composition helper — not a relaxation of strict validate)
research/evaluator/ stub (Slice 1) add screener evaluator (no-LLM) emitting the base scorecard; LLM evaluator reuses base + extension
champion loader SHIPPED — core/champion.py deep-merge into load_settings, tested consumed as-is; challengers = champion deep-merged via existing merge_stock_screener_config, after interpreter-side override-key validation (§10.4 — merge does no validation itself)
results-registry SHIPPED (narrow) — core/champion.append_registry_entry → registry.parquet (version, window, metrics_json, recorded_at) extend schema with candidate_id/candidate_type/reward; add the net-new operator-gated promote flow
paper/pattern_replay.py + pattern_audit.py SHIPPED — pattern-layer as-of replay + forward-return corpus composed by the evaluator driver; net-new = as-of money-flow leg + driver + BasketOutcomes
research/archive/ (MAP-Elites) stub (Slice 2) NOT built here; the LLM-search bandit's elites store, fed from the registry later
walk-forward / embargo / holdout designed, not built build generically here
StockScreenerConfig unused by research becomes candidate type #1

7. Success metrics

8. Tradeoffs

Tradeoff Why accepted
Corpus bounded to ~13 months by stock_prices that's the price data we have; money-flow is deeper but returns need prices
Offline-first; LLM-in-the-loop shadow deferred (the no-LLM paper-shadow arm ships as task 5b) immediate read now; the 5b arm starts accruing out-of-sample corroboration from day 1; full LLM shadow remains Phase 2
Promotion bar provisional pending follow-up research the registry/layers/spec are independent of it and shippable; the bar is a config knob swapped in when specified
Build the substrate generically now (vs screener-only) avoids the duplicate-engine drift we already hit; the auto-research engine needs identical parts
Parity is "identical given identical bars", not "byte-identical vs live" live yfinance vs replay Postgres differ in source + adjustment basis; pinning the fixture is the only honest parity
Universe = scraped names over history (survivorship) historical QU100 membership isn't recorded; disclosed

9. Approval requests

  1. Substrate shape (§4) — accept the candidate/registry/spec/evaluator/results-registry design, built generically + shared with auto-research. [ACCEPTED 2026-06-27]
  2. Supersede — retire DESIGN-qu100-pattern-tuning.md §6.2/§6.3/WS1 into this doc (annotated in that doc). [ACCEPTED 2026-06-27]
  3. Promotion bar — accept the §5 interim bar now, with a committed follow-up research pass before any real promotion. [ACCEPTED 2026-06-27]
  4. Task split (revised 2026-07-03) — accept the §11 split, rebased on the as-built inventory (champion loader / registry / pattern replay / as-of sector all shipped and composed, not rebuilt), and proceed to per-task plans. [ACCEPTED 2026-07-03]
  5. Casing scope — replay-only normalization behind the normalize_casing flag; the existing sector_momentum live exposure stays as-is (documented; resolved later via the designated first experiment, §11). [ACCEPTED 2026-07-03]
  6. Substrate hardening additions — co-evaluation + corpus_hash (§4.5), regime-sliced scorecards (§4.4), paper-shadow arm (task 5b), casing as designated first experiment, deflated_sharpe null until specified. [ACCEPTED 2026-07-03]

10. Implementation detail (for engineers)

10.1 Corpus & data reality (verified 2026-06-26 against localhost/rainier)

10.2 As-of selectors

Generalize _screen_money_flow (analysis/stock_screener.py:180, already day-scoped on the QU-snapshot half) to as-of t, mirroring the #148 generation convention (sector_analyzer._latest_generation_filter):

as_of_date = max(data_date) where data_date <= t and ranking_type='top100'
as_of_ts   = max(captured_at) where data_date = as_of_date and ranking_type='top100'   # per-ranking_type generation
rows       = top100 where data_date=as_of_date and captured_at=as_of_ts
             and lower(long_short)='long in'          # normalized in the as-of accessor
             deduped per symbol                        # mirrors live dedup, stock_screener.py:262

10.3 Reward registry interface

@register("sharpe", input_type="basket", role="primary", direction="increase")
def sharpe(outcomes: BasketOutcomes) -> float: ...
# input_type tags the candidate the reward can score (basket=screener, trades=LLM).
# roles: primary | guardrail(threshold, direction) | secondary. Ratio rewards compose two aggregations.

BasketOutcomes (pinned; lands with the reward-registry task): a frozen dataclass —

BasketOutcomes:
  days: list[BasketDay]          # one per selection day t
BasketDay:
  date: date                     # t
  symbols: list[str]             # selected basket, rank order
  fwd_return: dict[int, dict[str, float | None]]   # H∈{5,10,20} → symbol → return; None near window end, never 0
  regime: str                    # from compute_market_regime

pattern_audit's corpus row (fwd_return 5/10/20 + regime + contribution) is the natural substrate — the evaluator maps corpus rows into BasketDays, no recomputation.

This is a signature change to the as-built register(name, fn) (it becomes a decorator factory taking input_type/role/direction), not merely widening the function's type. Rewards are pure, deterministic, side-effect-free.

10.4 Experiment spec (YAML, PlanOut-style)

id: layer-weights-rebalance
status: active                    # active | retired; specs live in config/experiments/*.yaml
champion: champion.yaml
layer: layer_weights              # the one contended knob (a named field set)
challengers:
  - id: mf35   override: {layer_weight_money_flow: 0.35, layer_weight_pattern: 0.55}
  - id: mf40   override: {layer_weight_money_flow: 0.40, layer_weight_pattern: 0.50}
primary: sharpe
guardrails: [max_drawdown, turnover]
window: {train: 2025-05-27..2026-03-31, holdout: 2026-04-01..2026-06-25, embargo_days: 20}

Interpreted by the replay engine; mutually-exclusive challengers over the same replay. embargo_days (optional, default 20 = max horizon H) is the purge/embargo carrier for walk-forward validation.

Override-key contract (task 3): override keys MUST be real StockScreenerConfig fields — flat fields verbatim (e.g. layer_weight_money_flow, strong_buy_threshold), plus pattern_weights.<pattern> dotted paths which the spec interpreter translates into the nested dict. The spec interpreter itself validates every override key against StockScreenerConfig.model_fields + known pattern names (reusing the check pattern from load_champion_overrides, core/champion.py:91-110) and rejects the spec on any unknown key — this validation cannot be delegated downstream: merge_stock_screener_config performs no key validation, and StockScreenerConfig(**merged) silently drops unknown kwargs (champion.py:86-88), so an unvalidated typo'd key would yield a challenger identical to the champion — a silent no-op A/B. Only after validation does the interpreter deep-merge via the existing merge_stock_screener_config. A layer: names a declared field set for mutual exclusion, not a config path. A knob must exist as a config field before it can be experimented on — the interpreter never invents config paths; new knobs (e.g. a signal.momentum.method) require a config-schema change first.

10.5 Base scorecard, champion.yaml, results-registry

10.6 Test plan (one line each; task ownership in §11)

11. Task split (preview — promoted after per-task plans)

Tasks 1–3 are independent and fully parallel; 4 composes them; 5 depends on 4. No task rebuilds shipped code (§2 as-built inventory). 1. as-of-money-flow-selectors — _screen_money_flow_as_of mirroring the #148 per-(data_date, ranking_type) generation convention + live symbol dedup; as-of capital-flow (flow_date≤t); casing normalization behind an explicit normalize_casing flag (live delegates with False — byte-identical behavior; replay passes True; §10.2), with the recorded §9.5 decision on the sector_momentum live exposure. (S/M) [depends: none] 2. reward-registry-generalize — decorator-factory register(input_type, role, direction, threshold); define BasketOutcomes (§10.3); basket rewards (return, Sharpe, max-drawdown, turnover, dodged-loss); input-type refusal. (M) [depends: none] 3. experiment-contracts — YAML spec interpreter (champion + challengers + layer override + reward keys + window; mutual exclusion; interpreter-side override-key validation per §10.4, then deep-merge via existing merge_stock_screener_config) + base-scorecard schema + optional LLM extension in output_schema.yaml (additive). (S/M) [depends: none] 4. replay-evaluator — corpus driver composing shipped pattern_replay (emission_at/replay_composite/replay_screen) with task-1 as-of selectors → per-day BasketOutcomes → task-2 rewards → task-3 base scorecard, driven by the task-3 spec; pinned-fixture parity test; walk-forward/embargo/holdout windows; look-ahead tests. (M) [depends: 1, 2, 3] 5. registry-extension-promote — extend the shipped registry schema (candidate_id/candidate_type/reward/corpus_hash/experiment_id, append-compatible); registry refuses cross-corpus_hash comparisons (§4.5 co-evaluation rule); net-new operator-gated promote command (history write, version++/parent, registry append, rollback = re-promote prior); held-out scoring with the interim §5 bar + recorded challenger count. (M) [depends: 4] 5b. paper-shadow-arm — after each live scan, run screen_stocks under each active challenger config (no LLM) and log its would-be basket to the paper-trade tables tagged candidate_id; live champion output untouched. Pulls Phase 2's shadow arm forward: true out-of-sample data accrues from the day a challenger is defined. Hook: pipeline/post_scrape.py. Requires a paper_trade migration — the shipped schema resists a no-LLM arm: add a candidate_id column; relax thesis_id (NOT NULL + FK analysis_results + UNIQUE, core/models.py:646-648,718 — a no-LLM basket has no thesis) to nullable or use synthetic rows; re-scope the one-active-per-symbol shadow unique index (idx_paper_trade_active_symbol_shadow, models.py:744-749) per candidate, else two challengers picking the same symbol — or overlap with the existing WS-A WATCH-shadow arm — collide. (M) [depends: 3; parallel with 4/5] 6. [follow-up] promotion-bar research + spec — DSR/PBO/SPA/purged-CV/stationary-bootstrap; replace the interim bar and populate deflated_sharpe. (research + S) [depends: 5, non-blocking]

Designated first experiment (dogfood, after tasks 1–5): live adoption of casing normalization, modeled as a new boolean StockScreenerConfig field normalize_long_short_casing (default False; adding the knob is a one-line config-schema change per the §10.4 contract). Its expected effect is hand-computable from the 406 affected 'Long In' days — if the substrate's measured result diverges from the hand computation, the pipeline is wrong, not the hypothesis. This resolves the §9.5 live exposure through the substrate and serves as its end-to-end validation.

12. Alternatives considered

A — Keep blind live flips.                                  (rejected: the failure mode we're fixing)
B — Reuse qu100_portfolio for scoring.                      (rejected: diverges from the live ranker)
C — Standalone screener A/B, separate from auto-research.   (rejected: duplicate engine → drift)
D — Whole-config-only A/B (no reward registry / layers).    (rejected: can't do signal-level or multi-objective)
E — One generic candidate-agnostic substrate (chosen).