An AI That Gets Better at Cracking CAPTCHAs the More It Tries

Visual CAPTCHAs rest on a simple assumption: a bot that fails a challenge will keep failing it. New research flagged by Bruce Schneier shows how fragile that assumption is.

Vision-language models (VLMs) can solve visual puzzles with no task-specific training, but inefficiently: the hundredth instance of a familiar challenge costs an agent as much time and compute as the first. Narrow specialized detectors do the opposite — they answer in milliseconds, but only for the categories they were trained on. Neither improves with exposure.

A detector that learns from the model

The authors paired a fine-tuned YOLOv8 detector with an open-weight VLM, placing a confidence-based router between them. The system runs entirely from screenshots and operating-system input events — no browser automation, no DOM access. Across 16 classes it reaches 85.4% overall and 84.2% macro accuracy, beating either component alone.

The crux is that every answer the VLM produces doubles as a training label. That lets the detector absorb categories it was never trained for, typically after one or two encounters and without any human annotation.

Self-repair in an arms race

The same loop also repairs the system. A CAPTCHA operator can perturb images against the publicly released detector and drive its accuracy to zero. But the perturbations leave the VLM untouched, and its labels let the detector recover.

In a simulated year-long arms race, where the operator re-crafts its perturbations every month, the solver recovers in every round. A cheap open-weight "teacher" of only ~70% accuracy hardens the system as effectively as a perfect oracle would.

The researchers' conclusion is grim for defenders: visual CAPTCHA defenses that assume a failing bot stays failing badly understate how quickly an adaptive solver comes back.

Source: Schneier on Security