Why you can't glue .tar.gz files together with cat

Alex Chan ran into an everyday task: his project creates several .tar.gz archives, and they need to be rolled into a single file. You'd think it would be enough to glue the bytes together with cat — but the result refuses to read. Digging into this error made the author reconsider his own understanding of how tar and gzip work, and led him into the thickets of tape drives and patent law.

tar remembers tape

The name tar stands for tape archive, and the structure of the format directly reflects the constraints of magnetic tape: sequential reads, appending only at the end, a fixed block size. Inside, a tar is a run of files, each with its own header (name, size, time) and data blocks. It all ends with two or more blocks filled with zeros: this is the end-of-file (EOF) marker, which tells the reader to ignore everything that comes after.

This same logic explains several non-obvious quirks of the format. The file size has to be given in the header ahead of time — forget about tarinfo.size in Python, and you'll get an empty archive. There can be files with identical names inside: since you can't erase a block on tape, an update is appended at the end, and on extraction the later version overrides the earlier one. And finally, everything after the EOF marker is simply discarded.

This is exactly why the naive glue-together with cat fails: the reader program reaches the first EOF marker of the first archive and stops.

gzip remembers patents

gzip is a streaming compressor that compresses a single data stream without loss. Its shape was defined not by tape, but by the circumstances of the era: it was conceived as a free replacement for the compress utility, whose LZW algorithm was encumbered by patents. Hence the requirements — to work without patents, to compress and decompress within a small fixed amount of memory (RAM was scarce in the early 1990s), and to be portable across systems.

Inside, a gzip file is a sequence of “members,” each with its own header, compressed data, and a trailer with a checksum. gzip has no end marker. It's tempting to take members for an analog of files, but that isn't so: tools treat them as parts of a single stream, they can't be extracted individually, and decompression yields one continuous stream. Since there's no EOF marker, gzip files glue together beautifully with a simple cat.

What happens with .tar.gz

And here's where the catch lies. When you glue .tar.gz files together, gzip dutifully merges its members into a single stream — but the decompressed stream is read by tar again, hits the EOF marker of the first archive, and stops. There's no magic shortcut.

The right way is to decompress the members and write them into a new archive with a single EOF marker. The author gives a short Python function using the tarfile module, where the read and write modes are changed to r:gz/w:gz. It comes out to more code than gluing bytes, but the resulting archive reads with standard tools, without flags like --ignore-zeros.

A story that began with an annoying error turned into a pleasant side quest: now the author knows for certain that he hasn't missed any clever optimization.

Source: Lobsters