Understanding AI's Defects
Last revised 10/9/2026

Understanding AI's Defects

Why AI gets things wrong, organized by root cause — a diagnostic field guide for engineers, PMs, and serious users who want a causal map instead of an anecdote.

Most writing about AI defects circulates anecdotally — one-off complaints about hallucinations, sycophancy, or confidently wrong answers. Readers experience these failures constantly but lack a systematic frame for understanding why they happen. Without that frame, every defect feels like a surprise, a one-off bug, or a personal mistake. This collection organises defects by root cause rather than by symptom — data, architecture, training, reasoning, performed judgement, conversational drift, and the user's own contribution as amplifier — and ends with the judgement layer for working with the tool well. The pair to Understand Harness Engineering: that one explained how the agent works, this one explains how it fails. Honest, clinical, calm. Not contrarian, not doomer — calibrated for the working user who wants the vocabulary to use AI competently because they understand its limits.

Field GuideDiagnostic
Earn5CreditsinArtificial IntelligenceDigital Tech
6Modules27Sessions182Cards54Quizzes

Modules in this Collection’s System

The Framing — Why Defects Exist

*Defects are not bugs but structural consequences — and the four sources (data, architecture, training, interaction) that organise the rest of the collection.*

4Sessions

Defects of the Substrate — Data and Architecture

*What the training corpus bakes in, and what token-by-token generation cannot do no matter how the prompt is shaped.*

5Sessions

Defects of Process — Training and Reasoning

*The side effects of RLHF and reward modelling, and the failure modes that pass for reasoning but aren't.*

5Sessions

Defects of Surface — Judgement and Conversation

*Performed judgement that looks like real judgement, and the drift modes that emerge over a long thread.*

5Sessions

The User as Amplifier

*The prompt habits and reading habits that turn the model's defects into the user's defects.*

4Sessions

Working With an Imperfect Tool

*The judgement layer — how a defect-aware user prompts, verifies, and routes work differently.*

4Sessions

What You'll Walk Away With

  • 4structural sources of defect (data, architecture, training, interaction) that organise every failure you see
  • 7defects of data the training corpus bakes in before a single token is generated
  • 7defects of architecture that come from token-by-token generation and the context window
  • 7defects of training that RLHF and reward-modelling produce as side effects
  • 6defects of performed judgement that look like answers but aren't
  • 6conversational drift modes that emerge over a long thread
  • 6user-as-amplifier patterns that turn the model's defects into your defects
  • 5judgement principles for working with an imperfect tool well

You'll Have Answers To

  • ?Why does AI hallucinate — and is the next version actually going to fix it?
  • ?Why does AI agree with you even when you're wrong, and how does the training pipeline produce that as a side effect?
  • ?What's the difference between an answer that's correct and an answer that performs judgement — and why does experienced AI use feel so often like the second?
  • ?How does a long conversation drift away from your original instructions without you noticing?
  • ?Which defects come from the user, not the model — and which prompt habits manufacture bad output?
  • ?Where is AI structurally good, where is it structurally bad, and how should you route work between the two?
  • ?Why does understanding AI's defects make you better at using it than people who don't?

Critical Concepts Explored

Structural defects vs. bugsThe four sources of defectHallucination as a featureToken-by-token generationSycophancy and reward hackingPerformed judgementProbabilistic coverageThe opposition reflexConversational driftThe verification gap
Editor's Note
“A diagnostic map of AI's defects, organised by root cause rather than by symptom.”

What sets this collection apart is its refusal to treat AI's failures as anecdote. It opens the model, names where each defect comes from, and does the harder editorial work of separating defects of correctness from defects of performed judgement. Honest about the user's contribution as well as the model's. The pair to the harness-engineering collection — that one explained how the agent works, this one explains how it fails.

Editor's Brief
Who it's for
Engineers, PMs, operators, and serious AI users who keep noticing the same failure modes and want a causal frame instead of another think-piece on what AI gets wrong.
What stands out
The collection organises defects by root cause rather than by symptom, and is honest about both the model's contribution and the user's. It covers not only correctness defects (hallucination, error) but performed-judgement defects — the response shapes AI is trained to produce that look like judgement but aren't. The territory experienced users complain about most and have the least vocabulary for.
Read if
Read if you keep noticing the same kinds of AI failures and want a causal map of why they happen, not another anecdote about a chatbot that got something wrong.
Gold Quotes
Defects are not bugs — they are direct consequences of how these systems are built.

Hallucination, sycophancy, confident wrongness, performed judgement — none of these are anomalies waiting for the next release. They are predictable outputs of a token-by-token generator trained on an internet-shaped corpus and tuned by human feedback. Naming the structure is how the surprise stops.

About the Curator
WWorking with AI

Working with AI is a LearningFirst editorial line publishing field-level skills for working professionals adapting to AI in their daily craft. The aim is durable practice, not feature coverage — the patterns that transfer across tools and outlast any one model release.