Evaluating Claims and Evidence

The Claim That Cost Millions

A startup claimed its blood-testing device could run hundreds of tests from a single drop of blood. Investors poured in billions. Journalists wrote glowing profiles. Regulators granted approvals. Almost nobody asked the critical question: “Where’s the evidence that this actually works?”

The device didn’t work. The company was Theranos. The evidence—if anyone had demanded it—would have revealed the fraud years earlier.

By the end of this lesson, you’ll have a systematic method for evaluating any claim, from product promises to scientific findings to workplace proposals.

🔄 Quick Recall: In the previous lesson, we defined critical thinking as a set of learnable skills and identified AI as a reasoning partner. Can you name the six tools in the critical thinking toolkit? Today we sharpen the first tool: claim evaluation.

The Claim Evaluation Framework

Every claim can be evaluated using four questions:

Question 1: What Exactly Is Being Claimed?

Before evaluating evidence, make sure you understand the claim precisely. Vague claims are impossible to evaluate.

Vague: “Our product improves productivity.” Precise: “Our project management tool reduces time spent in status meetings by 30%, measured across 200 teams over 6 months.”

Here's a claim I want to evaluate:
[paste the claim]

Help me restate this claim precisely:
1. What specifically is being claimed?
2. What measurable outcome is implied?
3. What's the scope (who, when, where)?
4. What would need to be true for this claim
   to be valid?

Question 2: What Evidence Is Offered?

Not all evidence is equal. Here’s the evidence hierarchy from strongest to weakest:

Most persuasive claims rely on level 5-7 evidence. “Studies show…” without citation is level 7. “My friend tried it and…” is level 6.

✅ Quick Check: A supplement brand says “Doctors recommend our product.” Where does this fall on the evidence hierarchy? What additional information would you need?

Question 3: Is There a Gap Between Claim and Evidence?

The most common trick in misleading arguments is the gap between what the evidence shows and what’s claimed.

Evidence: “Participants who used our app reported feeling more productive.” Claim: “Our app makes you more productive.”

The gap: self-reported feelings of productivity are not the same as measured productivity. Feeling productive and being productive are different things.

Common gaps to watch for:

• Correlation vs. causation: The evidence shows a pattern, but the claim implies a cause
• Relative vs. absolute numbers: “50% improvement” might mean going from 2% to 3%
• Selected outcomes: The study measured 20 things; only the positive result is mentioned
• Sample vs. population: Results from college students may not apply to the general public

Question 4: Who Benefits?

Follow the incentives. Who benefits from you believing this claim?

• A pharmaceutical company claiming their drug is effective
• A politician claiming the economy is improving (or declining)
• A consultant claiming your company needs their services
• A news organization claiming the world is more dangerous than ever

Incentive doesn’t automatically make a claim false—but it means you should demand stronger evidence.

✅ Quick Check: You read that a study funded by a soda company found that sugar doesn’t contribute to obesity. How does the funding source affect your evaluation? Does it prove the finding is wrong?

The Five-Minute Claim Check

For everyday claims, this rapid evaluation catches most problems:

I want to quickly evaluate this claim:
[paste the claim]

Run a 5-minute claim check:
1. PRECISE CLAIM: Restate exactly what's being claimed
2. EVIDENCE LEVEL: What type of evidence is offered?
   (Meta-analysis? RCT? Anecdote? None?)
3. CLAIM-EVIDENCE GAP: Does the evidence actually
   support the specific claim being made?
4. ALTERNATIVE EXPLANATIONS: What else could explain
   this result?
5. INCENTIVES: Who benefits from this claim being
   believed?
6. VERDICT: On a scale of 1-10, how well-supported
   is this claim?

Practicing With Real Claims

Let’s apply the framework to common claim types:

Health claim: “This superfood boosts your immune system.”

• Evidence level? Usually testimonials or cherry-picked studies (level 5-6)
• Gap? “Boosts immune system” is vague and unmeasurable
• Incentive? The company selling the superfood

Business claim: “Companies that use our software grow 3x faster.”

• Evidence level? Likely observational correlation (level 3-4)
• Gap? Selection bias—faster-growing companies may adopt more tools
• Incentive? The software company’s revenue depends on this belief

News claim: “Crime is at an all-time high.”

• Evidence level? Depends on the data cited
• Gap? “All-time high” in what category? In what location? By what measure?
• Incentive? Fear-based claims drive engagement and clicks

Try It Yourself

Find a claim from today’s news, social media, or a product advertisement. Run it through the full evaluation framework using AI:

Evaluate this real-world claim for me:
[paste the claim with its source]

Perform a complete evaluation:
1. What precisely is being claimed?
2. What evidence is offered (and what level)?
3. What's the gap between evidence and claim?
4. What alternative explanations exist?
5. Who benefits from this claim?
6. What additional evidence would I need
   to accept or reject it?
7. Overall verdict with confidence level

Practice this with three different claims. You’ll quickly develop the habit of automatic evaluation.

Key Takeaways

• Every claim can be evaluated with four questions: What’s claimed? What’s the evidence? Is there a gap? Who benefits?
• Evidence exists on a hierarchy from meta-analyses (strongest) to unsupported assertions (weakest)
• The most common deception is the gap between what evidence shows and what’s claimed
• Correlation does not equal causation—always ask for the mechanism, not just the pattern
• Incentives don’t automatically invalidate claims but should increase your evidence requirements
• The five-minute claim check catches most problematic claims quickly

Up Next

In Lesson 3: Cognitive Biases That Distort Thinking, we’ll explore the mental shortcuts that cause smart people to make predictably bad judgments—and how to catch yourself before they do.