Skip to content

DOC: Add pyarrow type equivalency table #50474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 4, 2023
Merged

DOC: Add pyarrow type equivalency table #50474

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d8ee275
DOC: Add pyarrow type equivalency table
mroeschke Dec 28, 2022
a228f38
Merge remote-tracking branch 'upstream/main' into doc/pyarrow/types
mroeschke Dec 29, 2022
ee45cd9
Merge remote-tracking branch 'upstream/main' into doc/pyarrow/types
mroeschke Dec 29, 2022
267c15d
Merge remote-tracking branch 'upstream/main' into doc/pyarrow/types
mroeschke Jan 3, 2023
001b18b
Address review
mroeschke Jan 3, 2023
7596805
Merge remote-tracking branch 'upstream/main' into doc/pyarrow/types
mroeschke Jan 4, 2023
cfc98ce
Fix another typo
mroeschke Jan 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions doc/source/reference/arrays.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,36 @@ is an :class:`ArrowDtype`.
`Pyarrow <https://arrow.apache.org/docs/python/index.html>`__ provides similar array and `data type <https://arrow.apache.org/docs/python/api/datatypes.html>`__
support as NumPy including first-class nullability support for all data types, immutability and more.

The table below shows the equivalent pyarrow-backed (``pa``), pandas extension, and numpy (``np``) types that are recognized by pandas.

===================================== ========================== ===================
Pyarrow type pandas extension type Numpy type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Pyarrow type pandas extension type Numpy type
PyArrow type pandas extension type NumPy type

I think this is the preferred capitalization.

===================================== ========================== ===================
``pd.ArroeDtype(pa.bool_())`` :class:`BooleanDtype` ``np.bool_``
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
``pd.ArroeDtype(pa.bool_())`` :class:`BooleanDtype` ``np.bool_``
``pd.ArrowDtype(pa.bool_())`` :class:`BooleanDtype` ``np.bool_``

typo

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think repeating pd.ArrowDtype to wrap every arrow type in this column adds value? Since the column title is Pyarrow type, feels like just using pa.bool_()... is good enough. And maybe a note to comment on creating a pandas arrow type before or after the table could be useful.

Copy link
Member Author

@mroeschke mroeschke Jan 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I'll link to the pyarrow factory type functions and explain they need to be passed into pd.ArrowDtype explicitly

``pd.ArrowDtype(pa.int8())`` :class:`Int8Dtype` ``np.int8``
``pd.ArrowDtype(pa.int16())`` :class:`Int16Dtype` ``np.int16``
``pd.ArrowDtype(pa.int32())`` :class:`Int32Dtype` ``np.int32``
``pd.ArrowDtype(pa.int64())`` :class:`Int64Dtype` ``np.int64``
``pd.ArrowDtype(pa.uint8())`` :class:`UInt8Dtype` ``np.uint8``
``pd.ArrowDtype(pa.uint16())`` :class:`UInt16Dtype` ``np.uint16``
``pd.ArrowDtype(pa.uint32())`` :class:`UInt32Dtype` ``np.uint32``
``pd.ArrowDtype(pa.uint64())`` :class:`UInt64Dtype` ``np.uint64``
``pd.ArrowDtype(pa.float32())`` :class:`Float32Dtype` ``np.float32``
``pd.ArrowDtype(pa.float64())`` :class:`Float64Dtype` ``np.float64``
``pd.ArrowDtype(pa.time32(...))`` (none) (none)
``pd.ArrowDtype(pa.time64(...))`` (none) (none)
``pd.ArrowDtype(pa.timestamp(...))`` :class:`DatetimeTZDtype` ``np.datetime64``
``pd.ArrowDtype(pa.date32())`` (none) (none)
``pd.ArrowDtype(pa.date64())`` (none) (none)
``pd.ArrowDtype(pa.duration(...))`` (none) ``np.timedelta64``
``pd.ArrowDtype(pa.binary(...))`` (none) (none)
``pd.ArrowDtype(pa.string())`` :class:`StringDtype` ``np.str_``
``pd.ArrowDtype(pa.decimal128(...))`` (none) (none)
``pd.ArrowDtype(pa.list_(...))`` (none) (none)
``pd.ArrowDtype(pa.map_(...))`` (none) (none)
``pd.ArrowDtype(pa.dictionary(...))`` :class:`CategoricalDtype` (none)
===================================== ========================== ===================

.. note::

For string types (``pyarrow.string()``, ``string[pyarrow]``), PyArrow support is still facilitated
Expand Down