Skip to content

ARROW-4629: [Python] Pandas arrow conversion slowed down by imports #3706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

ARROW-4629: [Python] Pandas arrow conversion slowed down by imports #3706

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f1c8b40
Don't use local imports
fjetter Feb 19, 2019
b4604be
Only import pandas_compat if pandas is available
fjetter Feb 19, 2019
eb5c8ba
Apply suggestions from code review
xhochy Feb 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions python/pyarrow/array.pxi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@
# specific language governing permissions and limitations
# under the License.

from pyarrow.compat import HAVE_PANDAS

if HAVE_PANDAS:
import pyarrow.pandas_compat as pdcompat


cdef _sequence_to_array(object sequence, object mask, object size,
DataType type, CMemoryPool* pool, c_bool from_pandas):
Expand Down Expand Up @@ -165,9 +170,9 @@ def array(object obj, type=None, mask=None, size=None, bint from_pandas=False,
from_pandas=True, safe=safe,
memory_pool=memory_pool)
else:
import pyarrow.pandas_compat as pdcompat
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesm I'm just realising that here is one of the potential problems users may have reported about pdcompat import problems. This path should be supported without pandas but currently isn't.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I would agree with you. can you create a follow up issue about this? We should set up a docker-compose "no pandas" build to make sure that the project is usable without pandas

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and opened https://issues.apache.org/jira/browse/ARROW-4640

values, type = pdcompat.get_datetimetz_type(values, obj.dtype,
type)
if HAVE_PANDAS:
values, type = pdcompat.get_datetimetz_type(
values, obj.dtype, type)
return _ndarray_to_array(values, mask, type, from_pandas, safe,
pool)
else:
Expand Down
6 changes: 1 addition & 5 deletions python/pyarrow/pandas_compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
import six

import pyarrow as pa
from pyarrow.compat import builtin_pickle, PY2, zip_longest # noqa
from pyarrow.compat import builtin_pickle, DatetimeTZDtype, PY2, zip_longest # noqa


def infer_dtype(column):
Expand Down Expand Up @@ -447,8 +447,6 @@ def convert_column(col, ty):


def get_datetimetz_type(values, dtype, type_):
from pyarrow.compat import DatetimeTZDtype

if values.dtype.type != np.datetime64:
return values, type_

Expand Down Expand Up @@ -543,7 +541,6 @@ def _reconstruct_block(item):


def _make_datetimetz(tz):
from pyarrow.compat import DatetimeTZDtype
tz = pa.lib.string_to_tzinfo(tz)
return DatetimeTZDtype('ns', tz=tz)

Expand All @@ -554,7 +551,6 @@ def _make_datetimetz(tz):

def table_to_blockmanager(options, table, categories=None,
ignore_metadata=False):
from pyarrow.compat import DatetimeTZDtype

index_columns = []
columns = []
Expand Down