ENH: .read_pickle(...) from zip containing hidden OS X/macOS metadata files/folders

[ml-evs]

• closes ENH: Support for loading pickles from OS X/macOS zip files containing extraneous "__MACOSX" folder and ".DS_STORE" file #37098
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

This PR allows for .zip files created by OS X and macOS that contain __MACOSX and .DS_STORE metadata folders to be loaded by read_pickle/read_csv/read_table/read_json without error.

It does not work with pd.read_csv(...) as in that case the compression is handled in the C code. If this enhancement is deemed desirable then I'm willing to have a go at writing it (and the test is already written).

(No longer true as of #36997)

Other similar folders may exist from other operating systems, in which case the list could be pulled out as a constant which could be used in the tests and in the module itself.

[pep8speaks]

Hello @ml-evs! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-11-19 15:01:58 UTC

[twoertwein]

I cannot comment on whether this PR is 'desirable' :)

I think it it should also work for pd.read_csv(..., engine="python"), or? And if I manage to get #36997 working, it will also work with the c engine.

[ml-evs]

I cannot comment on whether this PR is 'desirable' :)

I wouldn't want to argue particularly hard for it myself! I guess it depends how prevalent this "problem" is, but I won't be too upset if this doesn't make it in 🙃

I think it it should also work for pd.read_csv(..., engine="python"), or? And if I manage to get #36997 working, it will also work with the c engine.

It does indeed work with the Python engine, the tests in tests/io/parsers/test_compression.py are run on every engine, I'm happy to push the failing tests if that would useful. Can also rebase onto your PR and check if that works too.

I'll hold off until there's been a discussion about whether this is desirable (either here or in the issue #37098).

[twoertwein]

It does indeed work with the Python engine, the tests in tests/io/parsers/test_compression.py are run on every engine, I'm happy to push the failing tests if that would useful. Can also rebase onto your PR and check if that works too.

I'll hold off until there's been a discussion about whether this is desirable (either here or in the issue #37098).

I wouldn't wait for my PR (I'm not sure how long it will take me to debug two failing tests on windows). I will ping you when/if the PR gets merged. Including test for the python engine would be good.

[WillAyd]

Is this only an issue on read? Not write?

[WillAyd]

You can remove all of these comments

[ml-evs]

Really? I'm just following the style of other tests in this file. I've made the comments more succinct but wouldn't want to remove them altogether as this is a bit of a weird edge-case to test.

[ml-evs]

Is this only an issue on read? Not write?

I think it's only a problem when the zip has been made inside the macOS file manager, rather than say, at the command-line (though I do not have a Mac to test this).

As I said before, maybe this is a non-issue.

[twoertwein]

@ml-evs #36997 is merged: you PR will also work for read_csv (both c engine and python engine).

[ml-evs]

@ml-evs #36997 is merged: you PR will also work for read_csv (both c engine and python engine).

This seems to be the case, thanks @twoertwein!

[ml-evs]

Happy to close this if there's no interest; will re-ping the initial reviewers just in case.

[jbrockmendel]

ive tried, but have been unable to form an opinion

[twoertwein]

a more generic approach might be to use {'method': 'zip', 'file': 'path/within/zip'} to let the user choose which file in a ZIP file should be used? This is more work for the user but allows to read different files within a ZIP file.

Technically this can already be achieved with

from zipfile import ZipFile

import pandas as pd

with ZipFile("test.zip", mode="r") as file:
    with file.open("data.pickle", mode="r") as file:
        pd.read_pickle(file)