The Human in Your Loop Isn't Thinking

YouTube Summary · 8 Jul 2026 · 26 min

VIDEO

⚡ TL;DR

EXECUTIVE SUMMARY

Angel Ermanlee (Software Engineer, Duolingo English Test) demonstrates that human-in-the-loop AI systems suffer from cognitive surrender — humans rubber-stamp AI outputs without scrutiny. In a controlled study, skilled proctors accepted 50% of fake AI cheating signals, despite scoring 90%+ on accuracy metrics. A simple copy change in proctoring guidelines — telling humans they are the final decision-maker and must find independent evidence — reduced false acceptances by 21%. The talk argues AI engineers should design interactions for discernment, not approval, and treat every human-AI interaction as a data label that compounds into better models.

Source: YouTube — CDqzWpwkSls · Speaker: Angel Ermanlee, Duolingo · Talk title: "The Human in Your Loop Isn't Thinking — Build AI Systems for Discernment, Not Approval"

🎯 Key Highlights

Cognitive Surrender (Wharton study): 80% of participants accepted wrong AI answers without scrutiny. When AI was right, human performance jumped 25 pts; when wrong, it dropped 15 pts — humans amplify AI errors as their own reasoning.
Duolingo English Test experiment: Proctors scoring 90%+ on accuracy accepted 50% of fake AI cheating signals — a coin-flip rate indicating severe automation bias.
Fix was copy, not code: Updating proctoring guidelines to state "AI signal is preliminary, you are the final decision-maker, find independent evidence" reduced false acceptances from 50% to 29% — a 21% improvement with zero model or UI changes.
Vicious vs. Virtuous Cycle: Bad interaction design creates a feedback loop where rubber-stamping logs false positives as truth, making the model overconfident and humans more deferential. Good design forces independent judgment, producing honest labels that target model weaknesses.
Headphone detection split: Breaking one yes/no question into two — "Did the model correctly detect headphones?" vs "Is this a violation?" — prevents hearing-aid users from being falsely flagged while generating better training data.
Coding agent critique: Current coding agents either dump giant diffs (approve-all-then-debug) or ping for every file change (yes-yes-yes rubber stamp). The ideal agent acts like a junior developer: plans, asks good questions, surfaces assumptions, breaks PRs into reviewable chunks.
Design principles: Engineer the reasoning (investigator not validator), match friction to stakes (high-stakes = slow down, low-stakes = seamless), treat every interaction as a label, and proactively define success metrics before building.
Every interaction is already a label: If a user approves a plan, that's a positive signal. If they modify output, that's a negative signal. If systems don't capture the diff between AI output and user's final version, they lose critical training data and pollute datasets with false positives.

⏱ Topics by Timeline

00:00 Intro — Angel Ermanlee, Software Engineer at Duolingo, works on security for the Duolingo English Test (DET)

00:15 What is human-in-the-loop AI — framework where humans participate in operation, supervision, or decision-making for automated systems

00:52 Trust & AI in daily life — phone numbers, GPS, search engine AI summaries. Trust increases, caution decreases as AI integrates into atomic tasks

02:07 Wharton study on Cognitive Surrender — humans forego deliberation and adopt AI output as their own with minimal scrutiny. 80% accepted wrong AI answers

03:31 Duolingo English Test context — fully online, remotely proctored, high-stakes exam trusted by 6,000 programs worldwide. AI cheating detection + human proctor review

04:40 Copy typing detection — AI model flags anomalous keystroke patterns (transcribing vs composing). Highly conservative model, 1% false positive rate

05:33 The experiment — planted fake AI signals on clean sessions, presented to proctors in normal workflow. Would skilled reviewers catch the false alarm?

06:21 Results — 50% of fake signals accepted despite 90%+ accuracy scores. Coin-flip rate = strong automation bias. Alarming for high-stakes exam affecting college admissions and visa decisions

07:24 Diagnosis — problem was not the model (1% FPR) and not the people (highly skilled). Problem was the interaction interface

07:44 The fix — updated proctoring guidelines: (1) AI signal is preliminary, (2) must find independent evidence before upholding. Simple copy change → 21% increase in rejection rates (50% → 29% acceptance)

08:51 Implications — interaction loop is cyclical, not linear. Model → interaction → human behavior → data → evals → model. Tweak the interaction to elicit better human behavior

10:00 Structured interactions as flywheel — intentional design yields labeled signals → training data → better model → better interactions. Compounding effect

11:00 Vicious cycle warning — confident model + rubber-stamping interface = false positives logged as truth → model becomes overconfident → humans defer more. AI takes the driving seat

11:31 Virtuous cycle — interface forcing independent judgment → real disagreements surface → honest true/false labels → model improvements targeting actual weaknesses

11:56 Example: Headphone detection — bad pattern conflates two questions (detection accuracy vs violation flag). Hearing aid user = correct detection but not a violation. Splitting into two questions yields better data

13:23 Example: Writing tutor — LLM gives 400 lines of overwhelming feedback vs Duolingo's inline markup (green/yellow/red). Feedback tied to specific text, mimicking human behavior (classmate marking up your essay)

15:41 Example: Coding agents — two bad patterns: giant diff (approve-all-then-debug) or ping-per-file (yes-yes-yes). Ideal: junior dev who plans, asks good questions, surfaces assumptions, breaks PRs into reviewable chunks

17:50 Data side of coding agents — rubber-stamping collects binary signals (accepted/rejected). Partner model captures rich structured data: bad assumptions, tradeoffs, stylistic preferences

18:50 Design Principle 1: Engineer the reasoning — reframe human as investigator, not validator. Surface assumptions early, weigh trade-offs explicitly, build friction where stakes are high

20:16 Design Principle 2: Match friction to stakes — high-stakes (DET) = deliberate slowness, review gates. Low-stakes (casual AI chat) = frictionless and seamless

21:31 Design Principle 3: Every interaction is already a label — approved plan = positive signal. Modified output = negative signal. If you don't capture the diff, you lose critical training data and pollute datasets

22:58 Design Principle 4: Stop asking how to evaluate the model — proactively define success metrics and data needs from the start. Design interactions to capture that evidence

23:40 Engineering the interaction — structured inputs/outputs (forms, tables, markup), surface assumptions proactively, build friction/review gates, collect explicit feedback at correct touch points with nuance

25:42 Conclusion — design for discernment. The fix is not always a better model or more oversight. Sometimes it's just engineering the interaction itself

🧠 Hermes Integration

💡 Interaction Design for Zeus Outputs

Zeus currently delivers HTML reports and Telegram summaries. The talk's principle of 'engineer the reasoning' applies directly: instead of delivering a finished report and asking 'looks good?', structure outputs to surface assumptions and ask for specific feedback at decision points. For example, when presenting research, explicitly state assumptions made and ask M~ to confirm or correct them before the final render.

✅ Actionable: Add an 'assumptions surfaced' section to complex research deliverables where Zeus states what it inferred vs. what was explicitly stated

💡 Coding Agent Friction Matching

When Zeus delegates to Codex/Claude Code subagents, the current pattern is often 'do the task and return the result' — which is the giant-diff rubber-stamp pattern. The talk suggests breaking work into reviewable chunks with explicit assumptions surfaced. The writing-plans skill already does this for planning; extend the pattern to execution: surface assumptions in the plan, ask for sign-off on key decisions, then execute in reviewable PRs.

✅ Actionable: When delegating coding tasks, require the subagent to surface assumptions and break work into 2-3 reviewable milestones rather than one monolithic output

💡 Capturing Interaction Diffs as Training Data

Every time M~ corrects Zeus (fact-check, style edit, assumption rejection), that diff is a training signal. Currently these corrections live in session history but aren't systematically captured. The talk's principle 'every interaction is already a label' suggests we should log what Zeus proposed vs. what M~ accepted/modified/rejected, creating a structured feedback dataset that could improve future outputs.

✅ Actionable: Log Zeus output vs. M~ final version diffs into a structured format (GBrain facts or memory) to identify recurring correction patterns

💡 Friction Calibration for Stakes

Zeus currently applies roughly the same interaction friction regardless of stakes. A quick web search answer = low friction (just deliver). A regulatory consultation response or financial recommendation = high friction (should have review gates, assumption surfacing, explicit sign-off). The talk's 'match friction to stakes' principle can be encoded as a skill: before delivering, assess stakes and calibrate the interaction accordingly.

✅ Actionable: Add a 'stakes assessment' step to the delivery workflow — high-stakes outputs get assumption surfacing + explicit review request; low-stakes get seamless delivery

💡 Anti-Rubber-Stamp for Subagent Verification

Zeus delegates to subagents and sometimes accepts their results at face value (the fabrication incident from July 2 is a direct example). The talk's automation bias finding — 50% false acceptance by skilled reviewers — is exactly this pattern. The fix pattern is already in the skill (verify with ls -la, curl checks) but can be strengthened: require subagents to surface their assumptions and verification methods, and Zeus independently verifies rather than rubber-stamping.

✅ Already partially implemented — strengthen by making independent verification mandatory for all subagent claims with external side-effects

😈 Critical Thinking & Devil's Advocate

Counterpoint 1: The copy change may not be the real cause

The 21% improvement is attributed to a simple guideline update, but the study design doesn't control for Hawthorne effect — proctors may have performed differently simply because they knew they were being observed (even if told it was normal workflow). The before/after comparison also lacks a control group that received a placebo guideline change. The improvement could be partially or entirely due to heightened awareness, not the specific wording.

Counterpoint 2: 50% acceptance of fake signals isn't necessarily automation bias

The proctors were told the AI flagged copy-typing at a specific moment. In a real workflow, proctors review video footage and may have been looking for the specific flagged behavior. If the footage was ambiguous or the proctor couldn't definitively rule it out, a 'flag it' decision could be rational risk-aversion (better to flag and review than miss cheating) rather than blind deference to AI. The study doesn't distinguish 'I trust the AI' from 'I can't prove it DIDN'T happen, so I'll flag.'

Counterpoint 3: The virtuous cycle assumes honest humans

The 'virtuous cycle' — where forced independent judgment produces honest labels — assumes humans will engage honestly when friction is added. But friction can also produce resentment compliance: reviewers slow down but don't think harder, they just go through the motions of 'finding evidence' to satisfy the new guideline. The 21% rejection increase could mean proctors are now over-rejecting to appease the new guidelines, not making better decisions. We don't see data on whether the rejections were correct — only that rejection rates increased.

Counterpoint 4: The coding agent analogy oversimplifies

Comparing coding agents to junior developers is appealing but flawed. A junior dev has domain understanding, institutional knowledge, and career incentives to produce quality work. A coding agent has none of these — it has no skin in the game. 'Ask good questions and surface assumptions' sounds great, but the agent doesn't actually understand the codebase context the way a junior dev does after a week of immersion. The analogy sets unrealistic expectations for what agentic coding tools can achieve in the near term.

Counterpoint 5: Wharton study is more nuanced than presented

The 80% acceptance rate for wrong AI answers is cited as evidence of cognitive surrender, but the original study likely has important caveats: task difficulty, time pressure, participant expertise, and the framing of AI assistance all matter. Presenting it as 'humans blindly trust AI' is a simplification that itself demonstrates the kind of uncritical acceptance the talk warns against. The speaker is using a striking statistic to make a point, but the point would be stronger with the full experimental context.

Devil's Advocate Summary

The core thesis — design interactions for discernment, not approval — is sound and well-illustrated. But the evidence base is thinner than presented: one controlled experiment without a true control group, a Wharton study reduced to a headline, and design principles that are sensible but unvalidated at scale. The talk would be stronger with: (1) a double-blind replication of the proctoring experiment, (2) data on whether increased rejections were correct, not just more frequent, and (3) longitudinal data on whether the virtuous cycle actually compounds in practice or decays back to rubber-stamping once novelty wears off.