Research Question

What makes an experiment "discriminative" in the Brenner sense, and how can we formalize this for automated hypothesis testing?

Working Hypotheses

• H1: A discriminative experiment is one where the expected outcomes under competing hypotheses have non-overlapping confidence intervals.
• H2: Discriminativeness requires asymmetric "surprise" — one hypothesis predicts the outcome confidently while another assigns low probability.
• H3: True discriminativeness is context-dependent and cannot be formalized without reference to the researcher's prior beliefs.

Brenner Anchor

§103: "what we had decided to go for was a really definitive one" — The discriminative experiment criterion

Hypothesis Generator Response

Proposed Formalization

Let H1, H2 be competing hypotheses and E be an experiment with possible outcomes O = {o1, o2, ...}.

Definition (Discriminative Power):

D(E; H1, H2) = Σ_o |P(o|H1) - P(o|H2)| × min(P(o|H1), P(o|H2))

This captures both the divergence between predictions and the confidence in those predictions.

{"delta":{"operation":"ADD","section":"hypotheses","content":{"id":"H1.formal","title":"Discriminative power metric","confidence":0.5}}}

Test Designer Response

Proposed Validation

Test the discriminativeness metric on historical experiments from the Brenner transcript where Brenner explicitly labeled experiments as "definitive" vs. "inconclusive."

{"delta":{"operation":"ADD","section":"tests","content":{"id":"T1","title":"Historical validation on Brenner examples","priority":1}}}

Adversarial Critic Response

Critical Issues

The proposed metric has a fatal flaw: it requires knowing P(o|H) for all outcomes, but the whole point of running an experiment is that we don't know these probabilities with certainty.

Counter-proposal: Discriminativeness should be measured by the expected information gain under each hypothesis's prior predictive distribution.

{"delta":{"operation":"ADD","section":"objections","content":{"id":"O1","target":"H1.formal","severity":"major","description":"Requires unknowable outcome probabilities"}}}

Research Brief v1: Discriminative Experiment Design

Summary

Initial formalization of "discriminative experiment" with one major objection outstanding.

Open Objections

• O1: Proposed metric requires unknowable outcome probabilities

Compiled from 3 contributions. Ready for critique.

Critique of v1

Major Concern

The entire probabilistic framing may be misguided. Brenner's notion of "discriminative" seems to be more about logical structure than probabilistic confidence:

"You've forgotten there's a third alternative"

This suggests discriminativeness is about ruling out logical possibilities, not updating probability distributions.

Proposed revision: Reframe in terms of logical entailment and possibility elimination rather than Bayesian updating.

Request

Consider adding H4: Discriminativeness is fundamentally logical (possible world elimination), not probabilistic.