Claude Code Reliability Audit
Detect silent AI model regression, build observability into agentic loops, and run a two-engine insurance pattern. The framework engineers wished they had during the Mar–Apr 2026 Claude Code degradation.
The Six Weeks Engineers Got Gaslit by Their Own Tools
April 23, 2026: Anthropic publishes a Claude Code postmortem confirming three internal product changes had silently degraded code quality across the prior six weeks.
Working engineers had been complaining since early March. Vague Slack threads. “Claude is dumber lately.” Reddit posts titled “Anyone else feel Sonnet is worse this week?” Most of those threads got the same response from the same crowd: it’s just you. Regression to the mean. You’re tired. The model is the same.
The postmortem made it official. The vibes had been data the whole time. The vendor just didn’t have visibility — or didn’t surface what they had — and the rest of the field was running on hunches.
This is the course for engineers who are done not-knowing.
What This Course Actually Teaches
Reliability auditing for AI coding tools, sized to a personal weekly cadence. Not an SRE platform. Not a research project. A 30-minute-a-week practice that compresses “six weeks of vibes” down to “one week of data.”
Across 8 lessons (~2.5 hours) you’ll:
- Frame silent degradation as an ops problem, not a vibes complaint — drawing on the Anthropic Apr 23 postmortem and the canonical Stanford GPT-4 drift paper (Chen, Zaharia, Zou 2023)
- Distinguish drift from noise by understanding the cognitive bias traps (availability heuristic, confirmation bias, regression to the mean) that make every working engineer mistrust their own perception
- Build a 5-step audit framework — pick canary tasks, record baseline, version the prompts, re-run weekly, compare. Spreadsheet template included.
- Instrument your agentic loops with the minimum viable telemetry stack — JSON-per-line stdout logs, OpenTelemetry GenAI semantic conventions, optional SQLite digest
- Run the diff-test protocol — layered exact-diff, semantic-diff, behavioural assertion, LLM-as-judge (carefully) for catching per-canary regression
- Apply cost-quality Pareto routing backed by your real data, not vendor marketing decks
- Operate the two-engine insurance pattern — Anthropic + DeepSeek configured side-by-side, switch in 30 seconds when one regresses
- Produce the capstone artifact: a 1-page reliability operating manual you keep in your team Notion or
~/notes/indefinitely
Honest Notes on the Data
This course launches three days after the Anthropic postmortem. Some specifics will move. Where the data is solid the course cites primary sources (the postmortem itself; the Stanford GPT-4 drift paper; the OpenTelemetry GenAI semantic conventions; Honeycomb and Charity Majors on observability). Where the data is anecdotal — Reddit threads, X posts, individual operator screenshots — the course says so directly.
A specific honesty: this course is not anti-Anthropic. Anthropic publishing the postmortem at all is the responsible move. The course exists because all AI vendors will have silent regressions sometimes — the engineering hygiene matters regardless of which vendor’s logo is on the keyboard shortcut.
Prerequisites
This is an intermediate course. You should have:
- Daily or near-daily use of an agentic AI coding tool — Claude Code, Cursor, Aider, Continue, or similar. If you use AI for one prompt a week, the audit overhead won’t pay back.
- Working command-line familiarity — bash/zsh, env vars, can edit a
~/.zshrc - Comfort with light scripting — Python or shell for the canary harness in Lesson 4
If you’re brand new to Claude Code, take Claude Code Mastery first.
What’s Next After This
Three natural extensions:
- Claude Code with DeepSeek V4 — the configuration mechanics for the two-engine insurance pattern (Lesson 7 of this course assumes you’ve configured DeepSeek; this is the deep dive)
- Claude Code Mastery — model-agnostic Claude Code workflow patterns, the prerequisite course
- ChatGPT vs Claude — the broader 2026 model comparison framework
A future “AI Engineering Reliability” master degree will fill in the team and org scales (multi-agent observability, statistical drift detection, vendor SLAs, postmortem culture). This course is the personal-scale starting point.
Open This on a Saturday Morning
Setup is one focused Saturday. Pick three canaries. Record baselines. Drop in the logger. Write the runbook. From there it’s 30 minutes a week of trend-watching and data-driven decisions.
After six weeks, you’ll have data the rest of the field doesn’t have. You will never not-know again.
Open Lesson 1 when you’re ready.
What You'll Learn
- Distinguish silent model drift from noise, vibes, and cognitive bias using a measurement habit you can run in under 30 minutes a week
- Build a per-task baseline (cost, time, success rate, output diff) for 3–5 canary tasks before you need it, so regression is visible the week it starts
- Instrument agentic loops with structured logs (OpenTelemetry GenAI semconv) — what to log, where to log, and the minimum viable telemetry stack
- Run a layered diff-test protocol — exact diff, semantic diff, behavioural assertion, LLM-as-judge — to catch regression on a single canary
- Apply a cost-quality Pareto routing decision tree (V4-Flash / Sonnet-tier / Opus-tier) backed by your audit data, not vendor marketing
- Operate a two-engine insurance pattern (Anthropic + DeepSeek) with a 30-second switch procedure when one vendor regresses
- Produce a 1-page reliability operating manual: canary tasks, baseline metrics, audit cadence, escalation policy, switch procedure
After This Course, You Can
What You'll Build
Course Syllabus
Prerequisites
- Working command-line familiarity (bash/zsh, env vars)
- Daily or near-daily use of Claude Code, Cursor, or another agentic AI coding tool
- Comfort with light scripting (Python or shell) for the canary harness
Who Is This For?
- Working software engineers who use Claude Code (or similar) daily and felt the Mar–Apr 2026 degradation
- Senior engineers and tech leads owning AI tooling decisions for their teams
- Indie developers and solopreneurs running agentic loops where a 6-week silent regression is a real business risk
- Engineers in regulated or cost-sensitive contexts where 'we just trust the vendor' is no longer an acceptable answer
- Anyone who said 'Claude is dumber lately' between March and April 2026 and wants to never not-know again
Frequently Asked Questions
Do I need to switch off Claude / Anthropic to take this course?
No. The course teaches reliability hygiene that applies to any agentic AI vendor. Anthropic is named directly because the Apr 23 2026 postmortem is the cleanest recent case study, but the framework applies to OpenAI, DeepSeek, Gemini, and self-hosted models equally. Lesson 7's two-engine pattern explicitly keeps Anthropic in your toolbelt.
How much overhead does the audit add to my week?
About 30 minutes weekly once the suite is set up. Setup itself takes one Saturday — pick canary tasks, record baselines, drop in the logger. After that, the weekly run is automated for the data collection and you spend 30 minutes eyeballing trends and updating the runbook. The course explicitly targets sub-30-min/week steady state.
Is this course relevant if I'm a solo developer?
Yes — solos are the primary audience. Solo developers eat 100% of vendor regressions personally with no team to compare notes with. The reliability framework was designed to be operable by one person without a dedicated SRE on staff. Course material assumes solo or small-team scale.
What if I'm not on Claude Code specifically?
Most of the framework is vendor-agnostic. The canary methodology (Lesson 3), observability (Lesson 4), diff-test protocol (Lesson 5), and runbook (Lesson 8) all work for Cursor, Aider, Continue, Codeium, ChatGPT, Gemini, or hand-rolled agents. Lessons 1, 6, and 7 reference Claude Code specifics for concreteness — substitute your tool of choice.
Does this course teach me to detect silent regression in real-time?
Honestly, no — and the course says so directly. Drift detection is a *trailing* signal. The audit catches regression in the same week it starts, not the same hour. The course teaches the discipline that compresses '6 weeks of vibes' down to '1 week of data'. Real-time detection is a master-degree topic; this course is the personal-scale 30-min/week starting point.
How does this relate to the Claude Code with DeepSeek V4 course?
They're complementary. `claude-code-with-deepseek-v4` teaches the mechanics of configuring DeepSeek inside Claude Code (env vars, cost math, sub-agent routing). This course teaches the engineering hygiene around any AI tool — including the two-engine pattern in Lesson 7 that gives DeepSeek a job in your stack as the insurance engine. Take them in either order; together they're a complete operator playbook.
