TASK PLAN — ab-experiment-contracts-5a0f

• Parent design: docs/DESIGN-qu100-ab-testing.md (§10.4, §10.5, §11.3) — approved 2026-07-03
• Priority: P1 · Depends on: none · PR base: main · Effort: S/M

1. Problem¶

Running an A/B today would mean writing engine code per experiment. The design says an experiment is a YAML file — champion + challengers + the one contended knob + reward keys + window — interpreted by the replay engine. Nothing interprets such a file yet, and there is no candidate-agnostic scorecard schema for evaluators to emit (the only rich schema, backtest, is LLM/TQQQ-shaped).

One trap makes validation load-bearing: merge_stock_screener_config performs no key validation, and StockScreenerConfig(**merged) silently drops unknown kwargs (core/champion.py:86-88). Without interpreter-side validation, a typo'd override key yields a challenger identical to the champion — a silent no-op A/B that "finds" no difference.

2. The fix¶

experiment.yaml ──► interpreter ──► validate override keys ──► deep-merge ──► challenger configs
                        │              (model_fields +           (existing
                        │               pattern names;            merge_stock_
                        │               reject unknown)           screener_config)
                        └──► layer = declared field set → mutual exclusion

Spec interpreter (src/rainier/research/experiment.py): - Spec home + activation (pinned here; consumed by ab-paper-shadow-arm-cbcd and ab-replay-evaluator-edb5): experiment YAMLs live in config/experiments/*.yaml. Each spec carries status: active | retired; load_active_specs(dir=config/experiments) returns only active specs and is the single discovery entry point (cron shadow arm, evaluator CLI). Mutual exclusion is evaluated across the active set. - Parses the §10.4 YAML shape: id, status, champion, layer, challengers[{id, override}], primary, guardrails, window{train, holdout, embargo_days: 20} (embargo_days optional, default 20 = the max forward-return horizon H — the purge/embargo carrier for walk-forward validation). - Validates every override key against StockScreenerConfig.model_fields + known pattern names (reusing the load_champion_overrides check pattern, champion.py:91-110), translating pattern_weights.<pattern> dotted paths into the nested dict. Rejects the spec on any unknown key — validation cannot be delegated downstream (see trap above). - Only after validation: challenger = champion deep-merged via the existing merge_stock_screener_config. - layer: names a declared field set for mutual exclusion (two concurrent specs claiming the same layer → error), not a config path. - Reward keys are strings resolved at run time — no import of the rewards module needed here (keeps this task independent of ab-reward-registry-b3b8).

Scorecard schemas (additive — validate() is already generic over named schemas): - New output_schema.yaml entries: base_scorecard (candidate_id, candidate_type, window, n_selection_days, corpus_hash, reward values by role, regime_scores, deflated_sharpe nullable, evaluator_sha) + llm_extension (valid_thesis_rate, cost_usd, filled_R, filled_rate, tqqq_bh_R, skill_yaml_sha). - A composition helper validating base + optional extensions. The LLM-shaped backtest entry stays untouched for its existing consumer.

3. Expected files¶

• src/rainier/research/experiment.py — spec dataclasses + interpreter + override-key validation.
• src/rainier/research/output_schema.yaml — base_scorecard + llm_extension entries.
• src/rainier/research/output_schema.py — composition helper only (no change to strict validate()).
• tests/research/test_experiment_spec.py, extensions to tests/research/test_output_schema.py.

4. Test plan (one line each)¶

• Valid spec (design §10.4 example, real fields) → parsed; challengers materialize as deep-merged configs differing only in the overridden keys.
• Unknown override key (flat or pattern_weights.<typo>) → spec rejected with the key named.
• pattern_weights.<real> dotted path → translated to nested dict; merge preserves the other pattern weights.
• Two ACTIVE specs claiming the same layer → mutual-exclusion error; different layers → both load; a retired spec never conflicts.
• load_active_specs: returns only status: active; retired/malformed-status specs excluded (malformed → loud error).
• embargo_days omitted → defaults to 20; explicit value parsed.
• Base scorecard payload without LLM fields → validates; with llm_extension → composes; extension alone → rejected.
• deflated_sharpe: null validates (nullable in schema).
• Existing backtest/cost_pilot/survivorship validation → unchanged (regression).

5. Non-goals¶

• No replay/evaluator execution (ab-replay-evaluator-edb5 interprets the spec; this task only parses/validates/materializes).
• No reward implementations (reward keys are opaque strings here).
• No champion-loader changes — core/champion.py is consumed as-is.
• No promote flow.

6. Worker contract¶

Per the Subagent Dispatch Contract (~/.claude/CLAUDE.md): unit tests with the feature; codex + /review loops to clean; PR against main; PR URL in the return message.