AI Tips Made People Three Times Less Accurate but Twice as Confident

A team of scholars from three universities in France and Italy tested how access to artificial-intelligence advice affects a person's ability to reason. The result proved disheartening: the willingness to answer honestly “I don't know” dropped from 44% to 3%, the accuracy of answers fell from 27% to 9%, while confidence in one's own correctness rose from 30% to 76%.

“People became far worse, with accuracy only a third of what it had been, yet they were twice as confident,” sums up Valerio Capraro, an associate professor at the University of Milano-Bicocca. He prepared the work together with Chiara Marcocci of the Scuola Normale Superiore and Walter Quattrociocchi of Rome's Sapienza University.

The trap was deliberately rigged

The experiment was set up to rule out the reasonable delegation of a task to a reliable tool. Participants were asked questions on which the models usually err — for example, about visual details in films: what color the team's uniform was in the movie “Bend It Like Beckham.” They used the Step 3.5 Flash model, which as a rule answered such questions incorrectly. In other words, the AI systematically prompted wrong answers. And some people capable of answering correctly on their own asked the AI for advice — and got it wrong.

A cash reward for correct answers helped, but only weakly. The willingness to admit ignorance rose from 3% to 8%, accuracy from 9% to 16%. Both figures remained far from the baseline values without AI: 44% and 27%, respectively.

“Cognitive capitulation”

The new study echoes work by scholars from the Wharton School, who earlier this year coined the term “cognitive capitulation”: people accepted the AI's incorrect answers in 80% of cases and were more confident than those who worked without an assistant. The fresh data add a sharper detail: it is not just about trusting wrong answers. The mere presence of AI at hand suppresses the habit of recognizing the limits of one's own knowledge.

“For a person, the ability to say ‘I don't know’ is very important, because it reflects an acknowledgment of the limits of our knowledge,” Capraro notes. He is especially troubled by children who grow up with these systems before critical thinking has even formed in them. Here the researcher recalls the overhaul of Google search, where links were replaced by confident AI summaries: this week the organization Common Sense Media called such an approach an “unacceptable risk” for schoolchildren.

The pattern, in the authors' view, is robust: AI-based products are designed to always give an answer and never say “I don't know.” And people, in using them, gradually learn to do the same.

Source: Hacker News