Skip to content

df.applymap() produces unwanted type conversion of datetime64[ns] to timedelta #19256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Apeder opened this issue Jan 15, 2018 · 1 comment
Closed

Comments

@Apeder
Copy link

Apeder commented Jan 15, 2018

Code Sample, a copy-pastable example if possible

time_data.head()

print time_data.dtypes

time_data[nt] = time_data[nt].applymap(lambda x: x.strftime('%I:%M:%S %p') if pd.notnull(x) else pd.NaT)

print time_data.dtypes

Problem description

When executing applymap on a masked subset of df datetime columns, two of the four columns are converted to timedelta. Can't figure out what might be happening, perhaps an error similar to #18493? But why only two of the four?!

Here's what time_data.head() looks like:

         Date           Name               In AM              Out AM  \
0  2017-11-06   AUSTIN LEWIS 1900-01-01 06:10:24 1900-01-01 12:03:23   
1  2017-11-06     FRED MOORE 1900-01-01 06:58:37 1900-01-01 12:12:11   
2  2017-11-06  KERRIE PAUSSA 1900-01-01 11:58:48 1900-01-01 19:39:49   
3  2017-11-06   OMAR CUELLAR                 NaT                 NaT   
4  2017-11-07   AUSTIN LEWIS 1900-01-01 07:07:27 1900-01-01 12:06:43   

                In PM              Out PM Sick Time  
0 1900-01-01 12:32:03 1900-01-01 17:31:50       NaT  
1 1900-01-01 12:42:53 1900-01-01 17:31:50       NaT  
2                 NaT                 NaT       NaT  
3 1900-01-01 20:00:19 1900-01-01 23:59:41       NaT  
4 1900-01-01 12:35:26 1900-01-01 17:33:20       NaT              

Expected Output

Date                        object
Name                        object
In AM               datetime64[ns]
Out AM              datetime64[ns]
In PM               datetime64[ns]
Out PM              datetime64[ns]
Sick Time           datetime64[ns]
Total Clock Time            object
dtype: object

Date                        object
Name                        object
In AM               datetime64[ns]
Out AM              datetime64[ns]
In PM               datetime64[ns]
Out PM              datetime64[ns]
Sick Time           datetime64[ns]
Total Clock Time            object
dtype: object

Output of pd.show_versions()

Name                        object
In AM               datetime64[ns]
Out AM              datetime64[ns]
In PM               datetime64[ns]
Out PM              datetime64[ns]
Sick Time           datetime64[ns]
Total Clock Time            object
dtype: object

Date                         object
Name                         object
In AM                        object
Out AM                       object
In PM               timedelta64[ns]
Out PM              timedelta64[ns]
Sick Time            datetime64[ns]
Total Clock Time            float64
dtype: object 
INSTALLED VERSIONS ------------------ commit: None python: 2.7.10.final.0 python-bits: 64 OS: Darwin OS-release: 17.3.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.2.4
Cython: None
numpy: 1.14.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.1.15
pymysql: 0.8.0
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@Apeder
Copy link
Author

Apeder commented Jan 15, 2018

Ah, turns out the issue was using pd.NaT rather than np.NaN. Le sigh.

@Apeder Apeder closed this as completed Jan 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant