Reimplement PR 112032, preserve trailing extra data when updating ZIP files

[edwardneal]

Contributes to #112017. Reimplements #112032. From the PR description:

aapt2 creates ZIP entries with a "version needed to extract" of 0.0, but sometimes writes these entries with deflate compression (which requires a "version needed to extract" of 2.0.)

When a ZipArchive is opened in Update mode and a new item is added, the central directory is written with the correct value for this field. However, the local file header for existing entries isn't rewritten (because it hasn't changed.) This leads to a mismatch between the local file header values and the central directory header values, which causes Android app builds to fail.

When an existing entry's "version needed to extract" is changed, we force its local file header to be rewritten.

There's a test to cover this. I've also added an entry to the packaged_resources.zip file from the issue and confirmed that the field matches between the local file header and the CD header for all compressed entries in the resultant file.

The second commit enables ZipArchive to perform in-place updates of ZIP entries with slightly malformed "extra data" fields. This field must consist of [header, data] pairs, where the header is four bytes long. This isn't always adhered to though - zipalign will pad the extra data field with zeroes to make sure that the compressed data which follows is aligned to an X byte boundary.

To tolerate this, we introduce the concept of "trailing extra data" - the part of the extra data field which can't be successfully parsed as a [header, data] pair. This is detected when parsing the central directory header (and the local file header), used when calculating the "extra data length" part of the relevant header, then written out after the parsed extra data.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

[carlossanlop]

Feel free to mark it as ready to review when you're ready and ping me to give it a full review.

[edwardneal]

Thanks @carlossanlop - I can see enough tests passing on enough platforms for review.

Besides the tests, I've rolled the latest build of System.IO.Compression into a local .NET 10 preview installation and published a simple .NET for Android project. I was able to reproduce the problem resolved by PR 112032 with this, but not the issue with ZIP alignment. To reproduce that, I ran zipalign against a ZIP file and tried to modify it.

[ericstj]

@carlossanlop -- seems like 44c08b8 created some conflicts here that were a bit too involved for me to easily resolve -- can you work with @edwardneal to get those resolved and review this change, please?

[edwardneal]

Thanks both. I've merged and verified that all's well in CI, and have managed to successfully build an Android .apk and .aab file using the new version. This is ready to review.

[carlossanlop]

Thanks for the PR @edwardneal . Left some comments/questions for you to consider.

[carlossanlop]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[carlossanlop]

Just in case, I ran the outerloop tests. If nothing concerning shows up, I'll approve and merge this.

[carlossanlop]

cc @jonathanpeppers this is the fix for the zipalign issue.

[jonathanpeppers]

I will test this here:

• [XABT] remove $(_AndroidUseLibZipSharp)=true default value android#9840

The most recent Maestro PR didn't have this one yet:

• [main] Update dependencies from dotnet/sdk android#10008

I can try it on the next Maestro PR.

[edwardneal]

Thanks @jonathanpeppers. I tested this PR earlier, and re-tested it with the most recent preview, using a new Android project with the settings:

<AndroidUseAapt2>True</AndroidUseAapt2>
<AndroidPackageFormat>apk</AndroidPackageFormat>
<AndroidSigningKeyStore>[keystore]</AndroidSigningKeyStore>
<AndroidSigningStorePass>[password]</AndroidSigningStorePass>
<AndroidSigningKeyPass>[password]</AndroidSigningKeyPass>
<AndroidKeyStore>true</AndroidKeyStore>
<AndroidSigningKeyAlias>apk</AndroidSigningKeyAlias>
<_AndroidUseLibZipSharp>false</_AndroidUseLibZipSharp>

I cleaned the project, then ran dotnet publish -f net10.0-android -bl -v:diag. The build failed and returned the output:

error ANDZA0000: 04-15 20:03:18.161  4356 15832 W zip     : WARNING: header mismatch
error ANDZA0000:                                                                    
error ANDZA0000: 04-15 20:03:18.164  4356 15832 W zip     : WARNING: header mismatch
error ANDZA0000:                                                                    
error ANDZA0000: 04-15 20:03:18.165  4356 15832 W zip     : WARNING: header mismatch
error ANDZA0000:                                                                    
error ANDZA0000: 04-15 20:03:18.165  4356 15832 W zip     : WARNING: header mismatch
error ANDZA0000:                                                                    

I've replaced .NET 10 Preview 3's System.IO.Compression.dll with the output from building dotnet/runtime, then cleaned the Android project again and re-ran dotnet publish ... with varying combinations of AndroidUseAapt2 = True/False and AndroidPackageFormat = aab/apk. The build completes without error, and when I use 7-Zip to test the archive, that test passes.

I couldn't force the zipalign step to fail, so had to test that independently - I just ran zipalign against an appropriate ZIP file, verified that it had the unusual padding in the extra data field, then used that as the initial baseline data for the new unit tests.

Are there any other tests I can run locally prior to the next Maestro PR?

[jonathanpeppers]

I would have thought a Release build (or dotnet publish) with default settings would run the zipalign step. It runs any time AndroidPackageFormat=aab, which should be the default for Release mode.

Your description sounds like you tested it, but I can check a .binlog (https://aka.ms/binlog) if you have one?

[jonathanpeppers]

This change hasn't shown up yet this morning:

• [main] Update dependencies from dotnet/sdk android#10041

I'll try again later, this is currently dotnet/runtime/main from ~April 11.

[jonathanpeppers]

I just rebased this one, to test out this change:

• [XABT] remove $(_AndroidUseLibZipSharp)=true default value android#9840