AI Learns to Break Ciphers: The CryptanalysisBench Benchmark Finds New Attacks

Cryptanalysis — the search for attacks against ciphers — sits at the intersection of two fields where language models are advancing fastest: mathematical reasoning and cybersecurity. What makes the task so convenient is that the result can be verified automatically: an attack either works or it doesn't. And the stakes could hardly be higher — we're talking about the primitives that digital security rests on. That's exactly why a new benchmark, CryptanalysisBench, has appeared, whose authors pose the question directly: can LLMs do cryptanalysis? Their conclusion — increasingly, yes.

How the benchmark works

The benchmark comprises 191 tasks across six families of cryptographic primitives — block ciphers, hash functions and others — drawn mostly from four NIST standardization competitions. The tasks are split into three difficulty levels: primitives with already known practical breaks; primitives with no known breaks (tested both at full strength and in weakened variants); and, finally, a set of live, current primitives at the cutting edge of cryptanalysis.

Five models — Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5 and the open GLM-5.2 — broke between 65 and 86 percent of the first-level schemes. At the second level, 6 to 12 schemes fell at full strength, and among the weakened variants, 24 to 61.

More than replaying the known

The most intriguing part is that the models didn't stop at reproducing already-published results. According to the authors, they also produced original cryptanalysis: a key-recovery attack exploiting a flaw in the construction of the SpoC AEAD scheme, as well as an error in a published proof of the CCA security of the KINDI cipher. Both results, as far as the authors are aware, had not been published before.

Separately, Anthropic ran a preliminary version of the Mythos model through the benchmark and discovered new vulnerabilities in the Hawk scheme and in AES with a reduced number of rounds.

The authors propose CryptanalysisBench as a tool for tracking whether (and when) AI cryptanalysis becomes a serious factor, and as a means of stress-testing candidate standards before they are deployed. These are early results for now, but the attacks found are a snapshot of a fast-moving frontier that may soon catch up with the published state of the art, and in places surpass it. This is definitely worth watching.

Source: Schneier on Security