Skip to content

PERF: Bypass chunking/validation logic in StringDtype__from_arrow__ #47781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Feb 24, 2023
Merged

PERF: Bypass chunking/validation logic in StringDtype__from_arrow__ #47781

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d63037d
Bypass chunking/validation logic in __from_arrow__
timlod Jul 18, 2022
9bb0312
Handle zero-chunks correctly & convert None to NA
timlod Jul 18, 2022
b538260
Add change to whatsnew
timlod Jul 18, 2022
9f102af
Merge branch 'main' into opt-stringarray-from-arrow
timlod Oct 22, 2022
7176086
Add to v2 whatsnew
timlod Oct 23, 2022
3d1162e
Add GH issue to comment about validation bypass
timlod Nov 30, 2022
c2d131a
Add # to GH issue
timlod Nov 30, 2022
e5028ff
Merge branch 'main' into opt-stringarray-from-arrow
timlod Jan 28, 2023
af5fbef
Merge branch 'main' into opt-stringarray-from-arrow
timlod Feb 24, 2023
563e247
Move release note to v2.1.0
timlod Feb 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ Performance improvements
- Performance improvements to :func:`read_sas` (:issue:`47403`, :issue:`47405`, :issue:`47656`, :issue:`48502`)
- Memory improvement in :meth:`RangeIndex.sort_values` (:issue:`48801`)
- Performance improvement in :class:`DataFrameGroupBy` and :class:`SeriesGroupBy` when ``by`` is a categorical type and ``sort=False`` (:issue:`48976`)
- Performance improvement in :func:`read_parquet` on string columns when using ``use_nullable_dtypes=True`` (:issue:`47345`)

.. ---------------------------------------------------------------------------
.. _whatsnew_200.bug_fixes:
Expand Down
21 changes: 12 additions & 9 deletions pandas/core/arrays/string_.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,16 +198,19 @@ def __from_arrow__(
# pyarrow.ChunkedArray
chunks = array.chunks

results = []
for arr in chunks:
# using _from_sequence to ensure None is converted to NA
str_arr = StringArray._from_sequence(np.array(arr))
results.append(str_arr)

if results:
return StringArray._concat_same_type(results)
if len(chunks) == 0:
arr = np.array([], dtype=object)
else:
return StringArray(np.array([], dtype="object"))
arr = pyarrow.concat_arrays(chunks).to_numpy(zero_copy_only=False)
arr = lib.convert_nans_to_NA(arr)
# Bypass validation inside StringArray constructor
new_string_array = StringArray.__new__(StringArray)
NDArrayBacked.__init__(
new_string_array,
arr,
StringDtype(storage="python"),
)
return new_string_array


class BaseStringArray(ExtensionArray):
Expand Down