ENH: use native filesystem (if available) for read_parquet with pyarrow engine

[jorisvandenbossche]

Some timings in https://issues.apache.org/jira/browse/ARROW-12428 suggest that it can be beneficial for reading parquet from S3 to use the pyarrow filesystem implementation, instead of pandas always using fsspec if installed (especially when selecting columns).

[jreback]

is there a reason we are not actually exposing filesystem as a named keyword?

[jorisvandenbossche]

is there a reason we are not actually exposing filesystem as a named keyword?

That could also be an option. I think the problem is that right now, this is a keyword that is just passed through to the underlying engine (as we do for other engine-specific keywords as well), and filesystem is not actually a keyword supported by fastparquet.

[jorisvandenbossche]

Actually, to be more correct, fastparquet supports it but names it fs. So we could indeed add a keyword filesystem, and map it to fs for fastparquet?

[jreback]

yes this sounds reasonable

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[alimcmaster1]

@jorisvandenbossche - was there much else that needed doing on this? Happy to help as this is functionality I use a lot :)

[TomAugspurger]

Might want to document this behavior. If so, here's my docs from the geopandas PR:

        When no storage options are provided and a filesystem is implemented by
        both ``pyarrow.fs`` and ``fsspec`` (e.g. "s3://") then the ``pyarrow.fs``
        filesystem is preferred. Provide the instantiated fsspec filesystem using
        the ``filesystem`` keyword if you wish to use its implementation.

[jreback]

this is quite old, happen to reopen if actively worked on.

[jreback]

@jorisvandenbossche i fyou can rebase and move the note to 1.5

[jreback]

yes this sounds reasonable

[simonjayhawkins]

closing as stale