compress.zlib: don't overshoot underlying reader #19163
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Inflate is internally using 8 byte bit buffer. We don't know in advance how many
bytes is compressed data block long. Once we reach end of block there may be
some bytes left in that internal buffer. That result that we move underlying
reader few bytes ahead of end of compressed data block. That is problem when
compressed block is followed by another data which need to be consumed.
We run into this problem in zig package manager when unpacking git
packfile and than again in zigimg project.
This PR ensures that zlib decompressor never overshoots. Zlib is now using 4
byte internal buffer. At the end of deflate stream we can overshoot for at most
4 bytes. After deflate stream, in zlib, there is 4 byte checksum. After reading
that checksum we are now sure that there is no overshoot.
This PR also ensures that there is no overshoot on gzip decompression. Gzip uses
8 byte bit buffer and has 8 byte footer, but previous implementation was not
careful about overshooting.
Using smaller buffer means more fillings of that buffer, means lower performance.
I compared this and previous implementation: on x86 Linux there is no
significant performance difference, on arm Linux there is ~20% slowdown.
We can provide configuration option to the decompressor so it can use 8 or 4
byte bit buffer and left the user decide what to use. But for now I decide not
to introduce that interface change.
Benchmarks of the two implementations on few sample data.
Linux x86:
Linux arm: