Roundtrip test for encoding and decoding images

[NicolasHug]

This goal of this issue is to add tests for both jpeg and png implementation on all platforms, to test that:

decode(encode(image)) ~= image.

This test will require an image-comparison util that is robust to minor changes. One possible approach is to compare histograms as done in PIL: https://github.com/python-pillow/Pillow/blob/affa059e959280bf7826ec1a023a64cb8f111b6d/Tests/helper.py#L110-L134

This test is not as robust as the ones we currently have where we individually test encode and decode w.r.t. to a reference implementation (PIL), but it's still good to have as a functional / integration test and it will hopefully help #3913 move forward.

CC @fmassa @datumbox @pmeier

[pmeier]

What I asked myself a few times for these kinds of issues: does the jpeg / png standard define a test for encoding / decoding?

[NicolasHug]

does the jpeg / png standard define a test for encoding / decoding?

Yes, according to https://en.wikipedia.org/wiki/JPEG#Required_precision the tests requirements seem to be in the DCT domain though, not pixel domain. I don't think we'll want to go as far as testing the DCT domain (I'm not even sure we can anyway, since we just use libjpeg)

[NicolasHug]

Actually looking at an older version of this: https://en.wikipedia.org/w/index.php?title=JPEG&oldid=814219419#Required_precision there seem to be pixel-level checks (for the decoding phase though, not the encoding obviously):

a maximum of one bit of difference for each pixel component
low mean square error over each 8×8-pixel block
...

[farleylai]

If the critical options are exposed when the decoder is allowed to make a decision, the decoded output can be made identical but some image processing toolkit such as PIL, unlike Tensorflow, could be opaque and simply depend on libjpeg versions.

• https://towardsdatascience.com/image-read-and-resize-with-opencv-tensorflow-and-pil-3e0f29b992be

In this regard, blaming the installation versions may not always make sense since it is not the very root cause but the responsibility of the toolkit to provide APIs exposing those critical options that may result in different output. Then having PIL as the reference implementation does not sound like a good idea.

• Loads different pixel values on Windows vs Linux python-pillow/Pillow#3833 (comment)

[NicolasHug]

@farleylai I'm not sure I understand your comment in the context of this issue.

This issue is about having round-trip test for the torchvision implementation. These tests are worth having, regardless of anything else.

Regarding expected results consistency: as you noted PIL will yield different results depending on the libjpeg backend that is used. Whether this is right or wrong, it's something we have to deal with and there isn't much we can do about this. Typically on Windows they link against libjpeg-turbo, but not necessarily on linux or Mac.

[farleylai]

This is to concern the decoding consistency especially when the results apparently affect reproducibility if care is not taken.

• the decoding options should capture what the real backend offers (that is, libjpeg(-turbo) mostly). tensorfolow's jpeg decoding API essentially provides whatever options supported by libjpeg(-turbo) such that users are allowed to approximate particular decoder's output such as opencv if necessary. I suppose torchvision is doing something similar and should expose the options as well.
• As for PIL, unless it intentionally decodes a jpeg image in a custom workflow, the output should correspond to some given or default options passed to libjpeg(-turbo)
• Regarding the package versions inconsistency, why does the package manager (say, conda) not ensure the the same version to be used to build torchvision and PIL if a particular jpeg version is specified on all supported platforms?

[NicolasHug]

One thing we could use here is the pytest-mpl plugin (also available on fbcode) for image comparisons: https://pypi.org/project/pytest-mpl/