In Linux 7.3, binfmt_misc will learn to pick the interpreter through BPF

The Linux kernel subsystem binfmt_misc is responsible for launching “non-native” and non-standard files: it is what hands Windows executables off to Wine and JAR archives to the Java runtime. By the time the Linux 7.3 merge window opens, this mechanism is in for a notable expansion of its capabilities.

The main batch of VFS changes has already been sent in by Christian Brauner, well ahead of time — before the start of the 7.3 development cycle, which is due to open right after the 7.2 release scheduled for August 17 (assuming it ships on time). Besides fixing long-standing bugs and reworking the locking, the patches change the very logic of the format registry.

The interpreter is computed on the fly

Previously, when a format was registered, the interpreter was set as a fixed string. Now the type of a binary file can be matched programmatically, and the interpreter itself can be computed at each launch. The key innovation: binfmt_misc entries are allowed to have BPF programs attached to them, which dynamically choose the execution environment — a loader or an interpreter — on a per-file basis.

According to Brauner, this opens the door to relocatable executables, where the dynamic loader is determined, for example, relative to the location of the binary itself. Such flexibility is especially interesting for NixOS and other systems with a non-standard path layout; the mechanism is general-purpose and allows for other policies as well.

Two dispatch modes

Alongside the rework, two launch modes have been added. Until now, the chosen interpreter effectively hijacked the entire “identity” of the process: argv[0], /proc/pid/cmdline and /proc/self/exe pointed to it rather than to the original file. Because of this, gdb in particular stumbled — the AT_ENTRY and AT_PHDR values did not match the actual executable, and PIE symbols remained unrelocated.

The first mode is transparent dispatch: the interpreter loads the binary through AT_EXECFD, the argument vector remains exactly as the caller formed it, and mm->exe_file and comm are marked with the name of the file itself. In this case the AT_FLAGS_TRANSPARENT_INTERP bit is set, and control over mapping the binary stays with the interpreter. The second mode is loader substitution: the file is executed natively, and only the interpreter changes.

If Linus Torvalds raises no objections during the merge window, the updated binfmt_misc will land in 7.3. Exactly how distributions and runtimes will take advantage of the new capabilities remains to be seen.

Source: Phoronix