Human Judgment and AI Limits

AI Has Limits

In the previous lesson, we explored transparency and disclosure. Now let’s build on that foundation. AI is powerful. But it’s not suited for everything.

Understanding AI’s limits isn’t pessimism—it’s practical wisdom. Knowing when AI shouldn’t be the decision-maker helps you use it well where it should.

What AI Can’t Do

AI can’t understand context like you do. AI doesn’t know your specific situation, relationships, or stakes. You do.

AI can’t take responsibility. When things go wrong, “the AI said so” isn’t an acceptable defense. Humans bear accountability.

AI can’t weigh values. When there are genuine trade-offs between competing goods (efficiency vs. fairness, speed vs. thoroughness), AI doesn’t have values to weigh. You do.

AI can’t know what matters to you. AI doesn’t understand your priorities, what you care about, or what you’re trying to achieve beyond the literal prompt.

AI can’t catch all its own errors. AI confidently produces wrong answers. Only human verification catches this.

High-Stakes Decision Categories

Decisions that should always have human judgment:

Affecting people’s lives:

• Hiring and firing
• Medical diagnoses and treatment
• Criminal justice decisions
• Immigration and asylum
• Child welfare determinations

Affecting rights and freedom:

• Bail and sentencing
• Loan and housing approvals
• Access to services
• Fraud determinations
• Account suspensions

Irreversible consequences:

• Anything that can’t easily be undone
• One-way door decisions
• Actions that affect long-term wellbeing

The higher the stakes, the more human judgment matters.

The “Human in the Loop” Problem

The theory: Human oversight prevents AI errors.

The problem: Automation bias.

Automation bias: The tendency to defer to automated systems, even when they’re wrong.

Why it happens:

• AI seems authoritative (it’s a computer!)
• Questioning AI takes effort
• Humans are busy; rubber-stamping is easier
• We trust consistency (AI is always the same)

Result: “Human in the loop” often means “human rubber-stamps AI decision.”

Real oversight requires:

• Time to actually review
• Training to know what to look for
• Authority to override
• Culture that supports questioning AI
• No penalty for catching errors

AI as Advisor, Not Decider

Useful framing: AI is an advisor, not a decision-maker.

Good advisor use:

• “What factors should I consider?”
• “What are the options here?”
• “What am I missing?”
• “How might others see this?”

Then you decide. You weigh the factors. You consider what matters. You take responsibility.

Bad use:

• “What should I do?” + immediate compliance
• Deferring to AI because it sounds confident
• Letting AI make calls you should be making

Quick check: Before moving on, can you recall the key concept we just covered? Try to explain it in your own words before continuing.

Context Matters

What AI doesn’t know:

• Your specific relationship with this person
• The unstated politics of your organization
• What’s actually at stake emotionally
• The history that led to this moment
• What you really want deep down

You provide context. AI provides options and information. You decide.

Appropriate Automation Levels

When to Override AI

Override AI when:

• Your gut says something is wrong
• The stakes are high enough that errors matter
• You have context AI doesn’t
• The AI output contradicts other reliable information
• You’d be uncomfortable defending this decision

Don’t defer to AI just because it’s consistent or confident.

The Accountability Question

Who’s responsible when AI-assisted decisions go wrong?

Not the AI. AI doesn’t bear responsibility.

The human who used AI does.

Implications:

• Don’t use AI to make decisions you’re not comfortable owning
• Understand what AI is recommending and why
• Be able to explain your decision without just pointing to AI
• If you can’t verify it, don’t bet on it

Exercise: Draw Your Lines

Think about your work and life. For which decisions:

1. AI can make the call (low stakes, reversible)
2. AI can advise, but you decide (moderate stakes)
3. Human judgment is essential (high stakes, affects others)
4. AI probably shouldn’t be involved at all

Draw your own lines. Know where AI helps and where it should step back.

Key Takeaways

• AI has real limits: context, responsibility, values, catching its own errors
• High-stakes decisions affecting lives and rights need human judgment
• Automation bias: humans tend to defer to AI even when it’s wrong
• “Human in the loop” only works if humans actually exercise judgment
• Frame AI as advisor, not decision-maker; you take responsibility
• Don’t use AI to make decisions you’re not comfortable owning
• Know your lines: where AI helps vs. where human judgment is essential

Next: How to critically evaluate AI outputs instead of trusting blindly.

Up next: In the next lesson, we’ll dive into Critical Evaluation.