A Prompt Injection Hidden in a Court Filing

On his blog, Bruce Schneier noted a new example of how the prompt injection technique is creeping into the most unexpected areas. This time hidden instructions for an artificial intelligence turned up in a court document: someone had embedded directions in the text that were aimed not at a human reader but at a language model that might process the document.

The logic of the trick is simple, and all the more troubling for it. As courts, lawyers and support services begin using AI to read, summarize and analyze paperwork, the text itself stops being a neutral carrier of information. Commands can be written into it that the model will treat as a guide to action — for instance, to distort a summary, nudge it toward a particular conclusion, or ignore inconvenient passages. A human reader might simply fail to notice such insertions.

The post is short on specifics, and this is precisely the case where the fact itself matters more than the details. Prompt injection has long been discussed in connection with chatbots and web pages, but legal documents are an area where the cost of a mistake is especially high. Court materials are by definition adversarial: the parties have a direct interest in influencing the outcome. If one of them knows that an opponent or the court itself might run the document through a language model, the temptation to slip an invisible hint into the text becomes an entirely rational move.

The case is a good illustration of a general problem that security specialists have been warning about for years: modern AI systems draw no clear line between data that needs to be analyzed and instructions that need to be followed. As long as that line remains blurred, any document that reaches a model is potentially also a command to it.

The practical takeaway is obvious. Organizations bringing AI into their work with legal and official paperwork should assume that incoming text may be hostile, and should not trust the model's output without human review. Until developers learn to reliably separate content from commands, hidden instructions in documents will remain a convenient and cheap tool for manipulation.

Source: Schneier on Security