pd.timedelta has smaller than expected range

[has2k1]

The pandas timedelta class has less than expected minimum and maximum timedeltas. The range is [-99999, 99999] days instead of [-999999999, 999999999] days

Code Sample

expected_max_days = 999999999  # 9 nines
max_days = 99999               # 5 nines

print("1. Expected minimum and maximum")
print(pd.Timedelta.min, '\t', pd.Timedelta.max)
print(datetime.timedelta.min, '\t', datetime.timedelta.max)

print("\n2. Expected min and max as pd.Timedelta objects")
print(pd.Timedelta(pd.Timedelta.min), '\t', pd.Timedelta(pd.Timedelta.max))  # overflow, no error

print("\n3. True min and max (in days) and comparing with numpy")
print(np.timedelta64(-max_days, 'D'), '\t\t', np.timedelta64(max_days, 'D'))
print(pd.Timedelta(-max_days, 'D'), '\t', pd.Timedelta(max_days, 'D'))

print("\n4. Expected min and max (in days) and comparing with numpy")
print(np.timedelta64(-expected_max_days, 'D'), '\t', np.timedelta64(expected_max_days, 'D'))
print(pd.Timedelta(-expected_max_days, 'D'), '\t', pd.Timedelta(expected_max_days, 'D')) # overflowError

Expected Output

1. Expected minimum and maximum
-999999999 days, 0:00:00     999999999 days, 23:59:59.999999
-999999999 days, 0:00:00     999999999 days, 23:59:59.999999

2. Expected min and max as pd.Timedelta objects
52654 days 06:07:35.539769   -52654 days +17:52:24.460229

3. True min and max (in days) and comparing with numpy
-99999 days          99999 days
-99999 days +00:00:00    99999 days 00:00:00

4. Expected min and max (in days) and comparing with numpy
-999999999 days      999999999 days
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-104-9f01918ed468> in <module>()
     15 print("\n4. Expected min and max (in days) and comparing with numpy")
     16 print(np.timedelta64(-expected_max_days, 'D'), '\t', np.timedelta64(expected_max_days, 'D'))
---> 17 print(pd.Timedelta(-expected_max_days, 'D'), '\t', pd.Timedelta(expected_max_days, 'D')) # overflowError

pandas/tslib.pyx in pandas.tslib.Timedelta.__new__ (pandas/tslib.c:43558)()

pandas/tslib.pyx in pandas.tslib.convert_to_timedelta64 (pandas/tslib.c:53419)()

pandas/tslib.pyx in pandas.tslib.cast_from_unit (pandas/tslib.c:58740)()

OverflowError: Python int too large to convert to C long

The problem is illustrated at points 2 (silent overflow) and 4(overflow error)

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.1-gentoo-r1
machine: x86_64
processor: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.18.0
nose: None
pip: 8.1.1
setuptools: 20.3
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: 4.1.2
sphinx: 1.3.6
patsy: 0.4.1
dateutil: 2.5.0
pytz: 2016.1
blosc: None
bottleneck: None
tables: None
numexpr: 2.5
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None

I get similar results on python 3.5.1.

[shoyer]

Unlike in numpy, pandas.Timdelta always uses nanosecond precision, so anything more than about 100000 days cannot be represented. Something does seem to be going wrong with the numeric overflow, though.

[jreback]

@has2k1 I guess. This is by definition as to the implementation as @shoyer comments, and in-line with the dateranges represented by Timestamps.

[jreback]

I suppose a doc-section along the lines of: http://pandas.pydata.org/pandas-docs/stable/gotchas.html#gotchas-timestamp-limits would be fine. Maybe we should move these to the Timeseries/Timedelta section though. @jorisvandenbossche ?

In [8]: pd.Timedelta(np.iinfo(np.int64).max)
Out[8]: Timedelta('106751 days 23:47:16.854775')

In [9]: pd.Timedelta(np.iinfo(np.int64).min)
Out[9]: NaT

In [10]: pd.Timedelta(np.iinfo(np.int64).min+1)
Out[10]: Timedelta('-106752 days +00:12:43.145224')

[has2k1]

@jreback, yes that would be helpful. In PR #12728 I'll include a section to that end in the
timedelta documentation.