Evaluating Sources and Spotting Misinformation

The Confident Liar

A student once asked an AI to find research supporting a specific medical claim. The AI provided three citations: author names, journal titles, volume numbers, page ranges, years–everything looked perfect. The student included them in her paper.

Her professor couldn’t find any of the cited papers. They didn’t exist. The AI had fabricated citations that looked completely legitimate, down to realistic-sounding author names and journal formatting.

This is the challenge of the AI age: the information isn’t just wrong sometimes–it’s wrong with absolute confidence.

By the end of this lesson, you’ll be able to:

• Identify the most common ways AI generates false information
• Apply the CRAAP test to evaluate any source systematically
• Fact-check AI outputs using practical verification techniques
• Build a healthy skepticism that serves you in all information contexts

Recall: The Trust Spectrum

In Lesson 1, you learned the trust spectrum: high trust for general explanations, low trust for specific claims. This lesson gives you the practical tools to act on that spectrum–turning vague suspicion into systematic evaluation.

How AI Gets It Wrong

Understanding how AI generates errors helps you spot them. Here are the main failure modes:

1. Confident Fabrication (Hallucination)

AI generates information that sounds plausible but is entirely made up. Common examples:

• Non-existent research papers with realistic citations
• Historical events that didn’t happen
• Statistics with no basis in reality
• Quotes attributed to people who never said them

Red flag: Very specific claims (exact numbers, specific dates, named sources) that you can’t find elsewhere.

2. Outdated Information

AI models have training cutoff dates. They don’t know about events, research, or changes that happened after their training data was collected.

Red flag: Any claim about “current” trends, recent events, or latest research.

3. Oversimplification

When summarizing complex topics, AI can strip away important nuance, making things seem more settled or clear-cut than they actually are.

Red flag: Absolute statements like “research shows” or “experts agree” without acknowledging debates.

4. Bias Amplification

AI reflects patterns in its training data. If the training data overrepresents certain viewpoints, AI will too.

Red flag: One-sided presentations of controversial topics without acknowledging other perspectives.

Quick Check

Have you ever encountered one of these failure modes when using AI? Most people have experienced confident fabrication without even realizing it. That’s what makes it dangerous.

The CRAAP Test: Your Evaluation Framework

The CRAAP test is a widely-used framework for evaluating sources. It works for AI outputs, websites, articles, studies–any information source.

C - Currency

Ask: When was this information published or last updated?

• Is the topic one where recency matters? (Technology changes fast; ancient history doesn’t.)
• Does the source reflect current research or outdated understanding?

R - Relevance

Ask: Does this information directly address my research question?

• Is it at the right level of depth? (Too basic? Too advanced?)
• Does it cover the specific angle I need?

A - Authority

Ask: Who created this information, and are they qualified?

• What are the author’s credentials?
• Is it published by a reputable organization?
• Is there institutional backing or peer review?

A - Accuracy

Ask: Is this information supported by evidence?

• Are claims backed by data, citations, or reasoning?
• Can you verify key facts independently?
• Are there factual errors that undermine credibility?

P - Purpose

Ask: Why does this information exist?

• Is the purpose to inform, persuade, sell, or entertain?
• Is there potential bias due to funding or affiliation?
• Is it presented as fact when it’s actually opinion?

The CRAAP Test in Practice

Try this prompt to evaluate a source:

“I found this source for my research: [describe source or paste excerpt].

Help me evaluate it using the CRAAP test:

• Currency: How current is this?
• Relevance: Does it address my question about [topic]?
• Authority: What do we know about the author/publisher’s credibility?
• Accuracy: Are the claims well-supported? Any red flags?
• Purpose: What’s the likely intent behind this content?

Give me an overall reliability rating: High / Medium / Low / Unreliable. Explain your reasoning.”

Practical Fact-Checking Techniques

Technique 1: The Triangle Method

Never trust a single source. Verify every important claim with at least three independent sources.

“You mentioned that [specific claim from AI]. I want to verify this. Where specifically could I find authoritative evidence supporting or contradicting this claim? Suggest 3 types of sources I should check.”

Then actually check those sources. If you can’t find corroboration, treat the claim as unverified.

Technique 2: The Citation Check

When AI provides citations, always verify them.

“You cited [author name, paper title, year]. I want to read the original. Can you provide more details about where this was published so I can find it? Also, is it possible you’re confusing this with a different paper?”

Sometimes asking AI to double-check itself catches errors. But always verify independently–don’t just accept AI’s “yes, I’m sure.”

Where to verify citations:

• Google Scholar (scholar.google.com)
• Library databases (if you have academic access)
• The journal’s website directly
• DOI lookup (doi.org)

Technique 3: The Reverse Claim

Ask AI to argue against its own claims.

“You just said [AI’s claim]. Now, play devil’s advocate. What’s the strongest argument against this claim? What evidence contradicts it?”

If AI can convincingly argue against its own claim, there’s real debate in the field that the original answer oversimplified.

Technique 4: The Specificity Probe

Vague claims hide behind generality. Force specificity.

“You said ‘research shows [claim].’ Be more specific: Which researchers? What study? What methodology? What year? What were the exact findings?”

If AI can’t provide specifics, the claim likely isn’t based on identifiable research.

Quick Check

Pick a factual claim you’ve recently read (from AI or any source). Apply the Triangle Method: can you find three independent sources that confirm it?

Building a Verification Habit

The goal isn’t to verify every single thing AI says–that would take forever. It’s to develop an efficient triage system:

Verify Always

• Specific statistics and numbers
• Named sources and citations
• Claims about current events or recent developments
• Information that will be in a published work (paper, report, article)
• Medical, legal, or financial claims

Verify When It Matters

• Historical facts you’re not sure about
• Technical explanations outside your expertise
• Claims that seem surprising or counterintuitive

Light Verification Okay

• General explanations of well-established concepts
• Definitions of standard terms
• Widely-known frameworks and models

The “Sounds Too Good” Test

If an AI output perfectly supports your argument with no caveats, no nuance, and no disagreement… be extra skeptical. Real research is messy. If the answer is too clean, something’s probably been simplified or fabricated.

Key Takeaways

• AI can be confidently wrong–plausible-sounding doesn’t mean accurate
• The four main failure modes: hallucination, outdated info, oversimplification, bias
• Use the CRAAP test (Currency, Relevance, Authority, Accuracy, Purpose) for any source
• Triangle Method: verify important claims with 3 independent sources
• Always verify citations–AI frequently fabricates realistic-looking references
• Build a triage system for verification: some things need checking, some don’t
• If it sounds too perfect, it probably needs extra scrutiny

Up Next

In Lesson 4, you’ll learn the art of summarizing and synthesizing information from multiple sources. This is where research becomes understanding–transforming scattered findings into coherent knowledge you can actually use.