EROFS Disables LZ4 Rolling Decompression Over Data Corruption Risk

The read-only EROFS filesystem, widely used in embedded systems, containers and other scenarios, was forced mid-development-cycle to temporarily disable its rolling decompression mechanism for the LZ4 algorithm. The corresponding change made it into the preparations for the Linux 7.3-rc3 kernel release.

What the problem is

The trigger was a report from AWS engineers: on their systems, LZ4 turned out to corrupt data with rare, specific datasets. A deeper analysis pinpointed the root of the problem — in the current LZ4 implementation, uncontrolled backward copies in memory are possible, and these violate an assumption underlying the rolling decompression optimization. When copying long literals, the memmove() function on x86 may copy data “backward,” going by an address comparison, even when the source and destination ranges do not overlap. This breaks the sliding-window assumption.

Since the LZ4 codebase in the kernel is outside EROFS's control, the developers decided not to risk data correctness in real-world use and to disable the optimization until a further plan is worked out.

How the optimization worked

Rolling decompression was introduced to cut down on temporary memory usage. LZ4 is based on LZ77 and only references the last 64 KiB of already-decompressed data, so in theory a limited sliding window of temporary pages is enough. The savings are noticeable: for example, 601,960 bytes can be compressed into a 256 KiB LZ4 extent, and without rolling decompression the worst case would require 146 additional pages per request.

Consequences

The main effect of disabling it is increased RAM usage during operation. This is partly mitigated by a recently added reserved buffer pool for LZ4 decompression, but even that solution is not ideal.

The developers stress that EROFS is used everywhere today, so data correctness is the priority. The sliding-window mechanism can be brought back later: either if the official LZ4 code is guaranteed to only copy non-overlapping ranges “forward,” or if EROFS gains its own LZ4 implementation. For now, starting with Linux 7.3, LZ4 rolling decompression in EROFS is disabled.

Source: Phoronix