PERF: improve DTI string parse #13692
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sinhrks
added
Datetime
| @@ -1046,7 +1046,12 @@ def _get_binner_for_grouping(self, obj): | |||
| l = [] | |||
| for key, group in grouper.get_iterator(self.ax): | |||
| l.extend([key] * len(group)) | |||
| grouper = binner.__class__(l, freq=binner.freq, name=binner.name) | |||
|
|
|||
There was a problem hiding this comment.
maybe isolate this a bit with some functions like _get_binner_for_resample does (maybe we should abstract this out even a bit more and have some PeriodIndexGrouper, DatetimeinexGrouper, which subclass TimeGrouper, but this might require some effort).
Current coverage is 84.54%@@ master #13692 diff @@
==========================================
Files 141 141
Lines 51185 51159 -26
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
- Hits 43275 43250 -25
+ Misses 7910 7909 -1
Partials 0 0
|
| if not (is_datetime64_dtype(data) or is_datetimetz(data) or | ||
| is_integer_dtype(data)): | ||
| data = tools.to_datetime(data, dayfirst=dayfirst, | ||
| yearfirst=yearfirst) | ||
|
|
||
| if issubclass(data.dtype.type, np.datetime64) or is_datetimetz(data): |
There was a problem hiding this comment.
any reason not to use is_datetime64_dtype here?
There was a problem hiding this comment.
In addition to normal datetime64, datetime64tz (DatetimeIndex) and int can be directly converted to DatetimeIndex.
|
@sinhrks this cleans up lots of junk! just a couple of clarifications. |
|
lgtm. ready to go? |
|
yes, it's ready:) |
This was referenced Jul 21, 2016
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
git diff upstream/master | flake8 --diffcleaned up
DatetimeIndexconstructor removing slower string-parsing path.Performance Improvement
related to #7599, internally use
to_datetimealways as it tries some fastpath.Bug Fixes
The cleanup fixed these 2 kind of issues:
1. #11169 and #11287 Invalid string parsing may raise
TypeErrorI met the same issue on travis and fixed with try-except clause (I can't reproduce it on my local Mac).
2. Index may incorrectly coerces mismatched tz
on current master,
DatetimeIndexand normalIndexbehaves differently.after the PR both behave the same, showing understandable error.