Experimental Design

A flawed experiment can’t be saved by better statistics or fancier AI tools. The time to catch design problems is before you collect data — when changes are free. AI helps you stress-test your experimental design by simulating alternatives, identifying confounds, and calculating power requirements before a single participant is recruited or reagent is mixed.

🔄 Quick Recall: In the previous lesson, you learned to search and synthesize literature with AI. Those skills feed directly into experimental design — the gaps, contradictions, and methodological patterns you identified during literature review now inform your study design.

Hypothesis Refinement

A vague hypothesis leads to a vague experiment. AI helps sharpen your thinking.

Help me refine this research hypothesis:

Initial idea: [your rough hypothesis]
Background: [what the literature says — from your review]
Key gap: [what's missing or contradictory in existing research]

Generate:
1. A specific, testable hypothesis with clear IV, DV, and expected direction
2. The null hypothesis
3. Two alternative hypotheses (what else could explain the expected result?)
4. Assumptions that must hold for the hypothesis to be testable
5. The minimum evidence needed to support or reject it

Example — before AI refinement: “Social media affects teen mental health”

After AI refinement: “Adolescents (13-17) who use image-based social media (Instagram, TikTok) for >2 hours daily will report significantly higher scores on the PHQ-9 depression scale compared to those using <30 minutes daily, after controlling for pre-existing mental health conditions, socioeconomic status, and sleep quality.”

The refined version is testable. You know exactly what to measure, who to recruit, what to control for, and what “support” looks like.

✅ Quick Check: What makes the refined hypothesis better than the original? Identify at least three improvements. (Answer: (1) Specific population defined (13-17), (2) specific type of social media identified (image-based), (3) specific measure named (PHQ-9), (4) specific exposure threshold (>2 hrs vs <30 min), (5) confounds identified for control (mental health, SES, sleep). Each improvement makes the experiment more rigorous and the results more interpretable.)

Power Analysis with AI

Statistical power determines whether your study can detect an effect if one exists. Underpowered studies waste resources and add noise to the literature.

Calculate the required sample size for this study:

Design: [between-subjects / within-subjects / mixed]
Primary outcome: [what you're measuring]
Expected effect size: [small/medium/large, or specific d/r/f value]
  - Basis for estimate: [cite the study or meta-analysis]
Alpha: [typically 0.05]
Desired power: [typically 0.80 or 0.90]
Number of groups: [how many conditions]
Number of measurements: [if repeated measures]

Also calculate:
1. Required N per group
2. Total N with 15% attrition buffer
3. What power you'd have at 50% of that N (budget constraint scenario)
4. The minimum detectable effect size at your budget-limited N

Key principle: Your expected effect size should come from published data, not optimism. Use effect sizes from meta-analyses when available — they’re more stable than single studies.

Confound Identification

The experiments that survive peer review are the ones that anticipated criticisms.

Identify potential confounds for this experiment:

Study: [brief description]
IV: [your independent variable]
DV: [your dependent variable]
Population: [who you're studying]
Setting: [lab / field / online]

Analyze:
1. Known confounds from published studies on similar topics
2. Confounds commonly listed as "limitations" in related papers
3. Demographic variables that might interact with my IV
4. Temporal confounds (time of day, season, order effects)
5. Measurement confounds (instrument sensitivity, observer bias)
6. Environmental confounds (setting, noise, equipment variation)

For each confound, suggest:
- Whether to control, randomize, measure, or acknowledge
- How to address it in the design vs. the analysis

✅ Quick Check: You’re running a study comparing two teaching methods. Both groups are in the same school but taught by different teachers. What confound does this introduce? (Answer: Teacher effects — any difference in outcomes could be due to teacher quality rather than teaching method. Solutions: use the same teacher for both conditions (potential bias), use multiple teachers randomized across conditions (increases N needed), or measure and statistically control for teacher variables.)

Protocol Optimization

Once your design is solid, AI helps optimize the practical protocol.

Optimize this experimental protocol:

Study design: [description]
Procedure steps: [list your current protocol]
Duration per participant: [estimated time]
Equipment needed: [list]
Personnel required: [how many researchers]

Check for:
1. Logical flow — does each step follow naturally?
2. Timing — are any steps unnecessarily long?
3. Participant burden — can anything be simplified?
4. Data collection points — am I capturing everything I need?
5. Counterbalancing — if multiple conditions, is order balanced?
6. Blinding — can participants or researchers infer conditions?
7. Stopping rules — when would I stop the study early?
8. Missing controls — positive control, negative control, manipulation check?

Common protocol gaps AI catches:

• Missing manipulation checks (how do you know your IV worked?)
• No attention checks in surveys (how do you know participants were engaged?)
• Order effects in within-subjects designs without counterbalancing
• No plan for missing data or participant dropout
• Insufficient blinding (participants can guess their condition)

Practice Exercise

1. Take a research question from your current work and run it through the hypothesis refinement prompt
2. Calculate the sample size needed using the power analysis prompt — use a published effect size from your literature review
3. Run the confound identification prompt and categorize each confound as “control,” “measure,” or “acknowledge”

Key Takeaways

• Refine hypotheses with AI to specify population, variables, measures, and expected direction before designing the experiment
• Power analysis must use evidence-based effect sizes from published data — never inflate expected effects to reduce required sample size
• Confound identification is more thorough when AI scans hundreds of related studies than when you rely on memory alone
• Always pilot-test AI-generated protocols with real participants before running the full study — AI can’t anticipate practical lab issues
• Document all design decisions and their rationale — this becomes your preregistration and methods section

Up Next

In the next lesson, you’ll learn to analyze your data with AI — selecting the right statistical tests, generating visualizations, and identifying patterns in complex datasets.