New zlib decompressor may read more data than necessary

[ianprime0509]

Zig Version

0.12.0-dev.2790+fc7dd3e28

Steps to Reproduce and Observed Behavior

Clone https://github.com/ianprime0509/zig-zlib-issue at commit 076f340acd113b73162800b43cc0add3a0141bd0 and run zig test test-new.zig. The test fails:

Test [1/1] test.decompress... FAIL (TestUnexpectedResult)
/var/home/ian/src/zig/lib/std/testing.zig:546:14: 0x103c6cf in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/var/home/ian/src/zig/lib/test-new.zig:25:9: 0x103c961 in test.decompress (test)
0 passed; 0 skipped; 1 failed.

If the expect call on line 25 is commented out, the test will fail with the error BadZlibHeader.

Expected Behavior

The test should pass. The input data to the test is the concatenation of two zlib data streams: when using std.compress.zlib.decompressor at the beginning of the concatenated stream and reading the entire decompressed content, the underlying reader should not be advanced past the end of the first zlib data stream, so that the second data stream can be decompressed starting at the end of the first one.

A successful outcome can be reproduced by checking out 7204ecc (the commit just before the new gzip changes were merged) and running zig test test-old.zig, which is identical to test-new.zig except it uses the old std.compress.zlib API.

---

For context on where this is necessary, an entry in a Git packfile consists of a header followed by zlib-compressed data, and the header only contains the uncompressed length of the data, so it is impossible to know where the second entry in a packfile begins without reading the first entry's compressed data precisely to the end and no further. Unfortunately, this means the Git package fetching support is currently broken.

I haven't delved too deeply into the new zlib code to find out how large of a change this would be, but I think this is a reasonable constraint to make on the decompress reader API, given formats such as Git packfiles which rely on knowing exactly where the decompressed data ends.

[squeek502]

cc @ianic

[ianic]

Thanks @ianprime0509 for bringing this up.

Can we solve problem by implementing reset method on decompressor. Reset will instruct decompressor to reset its internal state to initial so it could parse another zlib data stream if available in input reader.

Here is example:

test "flate bug 18967" {
    const first = @embedFile("testdata/fuzz/first.input");
    const second = @embedFile("testdata/fuzz/second.input");

    var in = std.io.fixedBufferStream(first ++ second);
    var out = std.ArrayList(u8).init(testing.allocator);
    defer out.deinit();

    var dcp = decompressor(.zlib, in.reader());
    try dcp.decompress(out.writer());
    std.debug.print("{s}", .{out.items});

    try dcp.reset();
    try dcp.decompress(out.writer());
    std.debug.print("{s}", .{out.items});
}

outputs:

tree 254b029bac4da1ef2c779044adc3c0fad85f8068
parent 28ddaf81d9cd0a71b139d09b88ae5957c5a91f1d
author Ian Johnson <ian@ianjohnson.dev> 1695580224 -0400
committer Ian Johnson <ian@ianjohnson.dev> 1695580224 -0400

commit 20

tree 254b029bac4da1ef2c779044adc3c0fad85f8068
parent 28ddaf81d9cd0a71b139d09b88ae5957c5a91f1d
author Ian Johnson <ian@ianjohnson.dev> 1695580224 -0400
committer Ian Johnson <ian@ianjohnson.dev> 1695580224 -0400

commit 20
tree d71cfb798acac6507d07c311bc6dca2e1c1f3fca
parent 1da9fdcf31b0342d1792cbfd8d6a8300eb21a1dc
author Ian Johnson <ian@ianjohnson.dev> 1695580186 -0400
committer Ian Johnson <ian@ianjohnson.dev> 1695580186 -0400

commit 19

[ianprime0509]

Thanks @ianic for your response! That would indeed solve this particular example I created to demonstrate the issue, but it wouldn't solve all potential use-cases, including the Git packfile use-case, since there can be other data between the zlib data streams that needs to be read using the underlying reader. The structure of a packfile looks like this:

+------------------+
|  Packfile header |
+------------------+
|   Entry header   |
+------------------+
| Zlib object data |
+------------------+
|   Entry header   |
+------------------+
| Zlib object data |
+------------------+
        ...
+------------------+
|     Checksum     |
+------------------+

So if the zlib decompressor reads too much data, some or all of the entry header or the next entry might be consumed by the underlying reader and available only in the decompressor's internal buffer.

If there were a way to tell how many bytes were actually used by the decompressor, and have a known upper bound on the amount of lookahead that could possibly be used, then it would be possible to work around this by replacing the buffered reader currently used in the packfile reader with an implementation that is guaranteed to always retain that known amount of lookahead in the buffer, so that after reading the zlib object data for an entry, we could see how much data was actually used and rewind the buffered reader to the end of the data.

For example, if we knew that the decompressor can only read ahead up to 128 bytes, then we could do something like

var pack_buffered_reader = std.io.rewindableBufferedReader(pack.reader(), 128); // New API to guarantee rewinding in a buffered reader
// ... loop and read entry header ...
var entry_decompress_stream = std.compress.zlib.decompressor(entry_crc32_reader.reader());
// ... read entry data ...
pack_buffered_reader.rewind(entry_decompress_stream.unusedBytes()); // New API to tell how many extra bytes were read by the decompressor for its buffer but not used

I think the new API you added in #18979 will be helpful for other use-cases, though, and it seems like a good addition to make.

Edit: actually, if I understand the code correctly, this may already be possible by reading decompressor.bits.nbits, right? I'll give that a try and see if I'm able to achieve something like what I described above with the information in that field.

Edit 2: hmm, yes, something like that does indeed seem to work: https://github.com/ianprime0509/zig-zlib-issue/blob/e843d77608810afbd0273cfb9fc933dfc509fcf6/test-new-workaround.zig I'll have to think a little more about the API of the rewindable buffered reader, though; this one was just thrown together for this experiment using std.io.BufferedReader as a base. It might be helpful to have the unused_bytes calculation available as a method on the decompressor to avoid reaching into its internals to obtain it.

[ianprime0509]

I tried proceeding with a workaround to fix the Git package fetching logic based on my comment above. However, I ran into another issue which I'm not sure how to solve with the "rewinding buffer" approach: the buffered reader is wrapped in two other readers, a CountingReader (so that we know the current byte position in the packfile) and a HashedReader (to confirm the integrity of the packfile using the checksum at the end of the file). If I rewind the buffered reader, these wrapper readers will end up with incorrect results.

I will need to think about this more and see if there is any other way this can be fixed aside from imposing the constraint described in this issue (that the decompressor should never read any more bytes than necessary from the underlying reader).

[daurnimator]

I'll have to think a little more about the API of the rewindable buffered reader,

I hope you'd be able to use std.LinearFifo and only call discard when the data is consumed?