Skip to content

ENH: use native filesystem (if available) for read_parquet with pyarrow engine #41194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

ENH: use native filesystem (if available) for read_parquet with pyarrow engine #41194

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
13dff55
ENH: use native filesystem (if available) for read_parquet with pyarr…
jorisvandenbossche Apr 28, 2021
2686641
update comment
jorisvandenbossche Apr 28, 2021
c3230c1
Merge remote-tracking branch 'upstream/master' into parquet-pyarrow-f…
jorisvandenbossche Sep 13, 2021
d5eab50
fixup version comparison
jorisvandenbossche Sep 14, 2021
ac79a63
Merge remote-tracking branch 'upstream/master' into parquet-pyarrow-f…
jorisvandenbossche Dec 7, 2021
6791f9e
add docstring
jorisvandenbossche Dec 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion pandas/io/parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,9 +173,25 @@ def write(

table = self.api.Table.from_pandas(df, **from_pandas_kwargs)

filesystem = kwargs.pop("filesystem", None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason we are not actually exposing filesystem as a named keyword?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason we are not actually exposing filesystem as a named keyword?

That could also be an option. I think the problem is that right now, this is a keyword that is just passed through to the underlying engine (as we do for other engine-specific keywords as well), and filesystem is not actually a keyword supported by fastparquet.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, to be more correct, fastparquet supports it but names it fs. So we could indeed add a keyword filesystem, and map it to fs for fastparquet?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this sounds reasonable

if (
isinstance(path, str)
and storage_options is None
and filesystem is None
and Version(self.api.__version__) >= Version("5.0.0")
):
try:
from pyarrow.fs import FileSystem

filesystem, path = FileSystem.from_uri(path)
except Exception:
# fallback to use get_handle / fsspec for filesystems
# that pyarrow doesn't support
pass

path_or_handle, handles, kwargs["filesystem"] = _get_path_or_handle(
path,
kwargs.pop("filesystem", None),
filesystem,
storage_options=storage_options,
mode="wb",
is_dir=partition_cols is not None,
Expand Down Expand Up @@ -470,6 +486,12 @@ def read_parquet(

.. versionadded:: 1.3.0

When using the 'pyarrow' engine, no storage options are provided
and a filesystem is implemented by both ``pyarrow.fs`` and ``fsspec``
(e.g. "s3://"), then the ``pyarrow.fs`` filesystem is preferred.
Provide the instantiated fsspec filesystem using the ``filesystem``
keyword if you wish to use its implementation.

use_nullable_dtypes : bool, default False
If True, use dtypes that use ``pd.NA`` as missing value indicator
for the resulting DataFrame. (only applicable for the ``pyarrow``
Expand Down