Anthropic shows off AI that improves its own “alignment”

Training some AI models with the help of others is gradually becoming a shared goal for new labs. On Friday, Anthropic published a paper titled “Automated Researchers Can Reliably Mitigate Alignment Failures,” which shows what this might look like in practice.

The work was led by Anthropic Fellows program participant Chen Yueh-Han. The system described reproduces the familiar cycle of scientific research, only without a human: each automated agent studies the available literature, proposes a method, trains a model on it for half an hour, and then, over several iterations, gradually pulls up the result on a chosen benchmark. Successful methods are kept, unsuccessful ones are discarded — this makes it possible to carry out the work quickly and at large scale.

Results across all ten tests

Given ten benchmarks targeting specific manifestations of “misaligned” behavior, the automated systems managed to improve the scores on each of them without degrading the model's overall performance. “Overall, these results provide the first evidence that automated post-training for alignment may become practical in the near term,” the paper states.

The authors draw a direct comparison between their “automated alignment researcher” (AAR) and a living colleague — and the comparison does not favor the human. “The best AAR method beats what skilled humans come up with, on average, in six hours,” TechCrunch quotes the paper as saying. “Human-provided research directions do not lead to stronger results.” A cost estimate is also given: the AAR's work runs at roughly $4 an hour thanks to the API, whereas a human researcher at a lab is paid $150 an hour.

A step toward recursive self-improvement

The outlet views the work as a step toward recursive self-improvement — a scenario many consider the next major milestone in AI development. If a model is capable of improving its own alignment training, it is reasonable to assume that in time it will take on training in general — and then living AI researchers risk being left out in the cold.

That said, the paper does not gloss over the limitations. The method works exactly to the extent that the benchmarks themselves reflect real alignment goals. And that means a great deal of separate work: the tests have to be written and maintained, just as the literature from which the automated researchers draw their ideas has to be expanded and updated.

Source: TechCrunch