Skip to content

DataFrame.quantile fails with timedelta data #18608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mroeschke opened this issue Dec 3, 2017 · 3 comments · Fixed by #21214
Closed

DataFrame.quantile fails with timedelta data #18608

mroeschke opened this issue Dec 3, 2017 · 3 comments · Fixed by #21214
Labels
Bug Internals Related to non-user accessible pandas implementation Timedelta Timedelta data type
Milestone

Comments

@mroeschke
Copy link
Member

>>> t1 = pd.timedelta_range('1 days', freq='D', periods=5)
>>> t2 = pd.timedelta_range('1 hours', freq='H', periods=5)
>>> df = pd.DataFrame({'t1': t1, 't2': t2})
>>> df.quantile(np.array([0.25, 0.5, 0.75]))
ValueError: need at least one array to concatenate

Problem description

This works for Series:

>>> df['t1'].quantile(np.array([0.25, 0.5, 0.75])) 
0.25   2 days
0.50   3 days
0.75   4 days
Name: t1, dtype: timedelta64[ns]

The larger issue is that the get_numeric_data in BlockManager doesn't pick up timedelta data.

The quantile issue arrises in the BlockManager here, https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals.py#L3596, because I TimeDeltaBlocks have is_numeric=False. Not entire sure if is_numeric=True would be the right fix as it might have impacts downstream (one test in pandas/tests/frame/test_analytics.py failed after changing it.)

Expected Output

pd.concat([df['t1'].quantile(np.array([0.25, 0.5, 0.75])), df['t2'].quantile(np.array([0.25, 0.5, 0.75]))], axis=1) 
         t1       t2
0.25 2 days 02:00:00
0.50 3 days 03:00:00
0.75 4 days 04:00:00

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.26
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.8
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.8
lxml: 3.8.0
bs4: 4.3.2
html5lib: 0.9999999
sqlalchemy: 1.1.13
pymysql: 0.7.9.None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

@mroeschke mroeschke changed the title DataFrame.quantile fails with timedeltas DataFrame.quantile fails with timedelta data Dec 3, 2017
@gfyoung gfyoung added Timedelta Timedelta data type Internals Related to non-user accessible pandas implementation Bug labels Dec 3, 2017
@gfyoung
Copy link
Member

gfyoung commented Dec 3, 2017

@mroeschke : This indeed looks strange. Poke around to see what fix works best. If you get stuck, feel free to submit what you have as a PR, and we can examine further.

@jreback
Copy link
Contributor

jreback commented Dec 3, 2017

so we treat Timedelta as non-numeric (like datetime). But some operatators can deal with this type of data (e.g. min, max, median, quantile), so this should work (I am pretty sure we test on datetime for this, but obviously not fully on quantile).

Timedeltas are a special beast because they are numeric in numpy.

@jreback jreback added this to the Next Major Release milestone Dec 3, 2017
@mroeschke
Copy link
Member Author

Actually, I just realized this works if numeric_only=False is specified:

In [9]: df.quantile(np.array([0.25, 0.5, 0.75]), numeric_only=False)
Out[9]:
         t1       t2
0.25 2 days 02:00:00
0.50 3 days 03:00:00
0.75 4 days 04:00:00

(As a note, numeric_only isn't fully specified in the docstring https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.quantile.html)

I guess the question still stands whether timedelta data should be included if numeric_only=True is specified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Internals Related to non-user accessible pandas implementation Timedelta Timedelta data type
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants