Skip to content

PERF: do NPY_NAT check inside get_datetime64_nanos #24031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 2, 2018
Merged

PERF: do NPY_NAT check inside get_datetime64_nanos #24031

merged 1 commit into from
Dec 2, 2018

Conversation

jbrockmendel
Copy link
Member

Bonus of a little simplification in array_to_datetime

The perf pickup comes from changing a python call ts.view('i8') to a cython call get_datetime64_value(ts).

In [3]: val = np.datetime64('NaT', 'D')
In [4]: %timeit pd.Timestamp(val)

master (leaving out "The slowest run [...]"

100000 loops, best of 3: 2.33 µs per loop
1000000 loops, best of 3: 2.02 µs per loop
100000 loops, best of 3: 2.44 µs per loop
100000 loops, best of 3: 1.4 µs per loop
1000000 loops, best of 3: 2.32 µs per loop

PR

1000000 loops, best of 3: 533 ns per loop
1000000 loops, best of 3: 758 ns per loop
1000000 loops, best of 3: 734 ns per loop
1000000 loops, best of 3: 570 ns per loop
1000000 loops, best of 3: 779 ns per loop

In [9]: val2 = np.datetime64(55, 'us')
In [10]: %timeit pd.Timestamp(val2)

master

100000 loops, best of 3: 3.58 µs per loop
100000 loops, best of 3: 3.49 µs per loop
100000 loops, best of 3: 3.65 µs per loop
100000 loops, best of 3: 2.99 µs per loop

PR

1000000 loops, best of 3: 1.61 µs per loop
1000000 loops, best of 3: 1.76 µs per loop
1000000 loops, best of 3: 1.66 µs per loop
1000000 loops, best of 3: 1.49 µs per loop

@codecov
Copy link

codecov bot commented Dec 1, 2018

Codecov Report

Merging #24031 into master will increase coverage by 49.88%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #24031       +/-   ##
===========================================
+ Coverage   42.46%   92.35%   +49.88%     
===========================================
  Files         161      161               
  Lines       51557    51557               
===========================================
+ Hits        21892    47613    +25721     
+ Misses      29665     3944    -25721
Flag Coverage Δ
#multiple 90.75% <ø> (?)
#single 42.46% <ø> (ø) ⬆️
Impacted Files Coverage Δ
pandas/core/computation/pytables.py 92.37% <0%> (+0.3%) ⬆️
pandas/io/pytables.py 92.3% <0%> (+0.92%) ⬆️
pandas/util/_test_decorators.py 93.24% <0%> (+4.05%) ⬆️
pandas/compat/__init__.py 58.36% <0%> (+8.17%) ⬆️
pandas/core/config_init.py 99.24% <0%> (+9.84%) ⬆️
pandas/core/reshape/util.py 100% <0%> (+11.53%) ⬆️
pandas/compat/numpy/__init__.py 92.85% <0%> (+14.28%) ⬆️
pandas/core/computation/common.py 85.71% <0%> (+14.28%) ⬆️
pandas/core/api.py 100% <0%> (+14.81%) ⬆️
pandas/core/indexes/api.py 99% <0%> (+14.85%) ⬆️
... and 119 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa547d8...ce040ce. Read the comment docs.

1 similar comment
@codecov
Copy link

codecov bot commented Dec 1, 2018 •
edited
Loading

Codecov Report

Merging #24031 into master will increase coverage by 49.88%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #24031       +/-   ##
===========================================
+ Coverage   42.46%   92.35%   +49.88%     
===========================================
  Files         161      161               
  Lines       51557    51557               
===========================================
+ Hits        21892    47613    +25721     
+ Misses      29665     3944    -25721
Flag Coverage Δ
#multiple 90.75% <ø> (?)
#single 42.46% <ø> (ø) ⬆️
Impacted Files Coverage Δ
pandas/core/computation/pytables.py 92.37% <0%> (+0.3%) ⬆️
pandas/io/pytables.py 92.3% <0%> (+0.92%) ⬆️
pandas/util/_test_decorators.py 93.24% <0%> (+4.05%) ⬆️
pandas/compat/__init__.py 58.36% <0%> (+8.17%) ⬆️
pandas/core/config_init.py 99.24% <0%> (+9.84%) ⬆️
pandas/core/reshape/util.py 100% <0%> (+11.53%) ⬆️
pandas/compat/numpy/__init__.py 92.85% <0%> (+14.28%) ⬆️
pandas/core/computation/common.py 85.71% <0%> (+14.28%) ⬆️
pandas/core/api.py 100% <0%> (+14.81%) ⬆️
pandas/core/indexes/api.py 99% <0%> (+14.85%) ⬆️
... and 119 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa547d8...ce040ce. Read the comment docs.

@TomAugspurger TomAugspurger added this to the 0.24.0 milestone Dec 1, 2018
@TomAugspurger TomAugspurger added the Performance Memory or execution speed performance label Dec 1, 2018
@jbrockmendel jbrockmendel mentioned this pull request Dec 2, 2018
Merged
jreback
jreback requested changes Dec 2, 2018
View reviewed changes
pandas/_libs/tslibs/conversion.pyx
ival = get_datetime64_value(val)
if ival == NPY_NAT:
return NPY_NAT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this hit a currently tested case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Within tests/indexes/datetimes there are 14

@jreback jreback added the Datetime Datetime data dtype label Dec 2, 2018
@jreback jreback merged commit 26d7c52 into pandas-dev:master Dec 2, 2018
@jreback
Copy link
Contributor

jreback commented Dec 2, 2018

thanks @jbrockmendel

@jbrockmendel jbrockmendel deleted the a2d-pre3 branch December 2, 2018 16:41
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
do NPY_NAT check inside get_datetime64_nanos (pandas-dev#24031)
8dde0ab
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
do NPY_NAT check inside get_datetime64_nanos (pandas-dev#24031)
3877974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Datetime Datetime data dtype Performance Memory or execution speed performance
Projects
None yet
Milestone
0.24.0
Development

Successfully merging this pull request may close these issues.
3 participants
@jbrockmendel @jreback @TomAugspurger