Skip to content

!I fix for BUG: resample with tz-aware: Values falls after last bin #15549 #18337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Nov 21, 2017
Merged

!I fix for BUG: resample with tz-aware: Values falls after last bin #15549 #18337

Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
68a02aa
!B [pandas-dev/pandas#15549] resample with tz-aware: Values falls aft…
ahcub Mar 2, 2017
47b99ac
restore mistakenly removed line
ahcub Mar 2, 2017
04ce929
remove redundant whitespaces
ahcub Mar 2, 2017
a43bb19
remove redundant whitespaces
ahcub Mar 2, 2017
44fc3f2
!U add tz to DatetimeIndex initialization
ahcub Mar 3, 2017
2a1d24c
!U add test for a bug: resample with tz-aware: Values falls after las…
ahcub Mar 3, 2017
d28ff98
!U do not convert TZ to UTC if it not set
ahcub Mar 3, 2017
7c770c0
!U revert changes with timezone
ahcub Nov 17, 2017
effa650
!U revert spaces
ahcub Nov 17, 2017
aa8c0af
!U change way the end timestamp is defined
ahcub Nov 17, 2017
2db6576
Merge branch 'master' of https://github.com/pandas-dev/pandas into pa…
ahcub Nov 17, 2017
2744ad3
Merge branch 'pandas-dev-master'
ahcub Nov 17, 2017
fb4d405
!U add conditional change in the formula for defining the end timestamp
ahcub Nov 17, 2017
93842d0
!U make the values_present argument optional
ahcub Nov 18, 2017
fc795c6
!U add line feed
ahcub Nov 19, 2017
514fe24
!U fix formatting
ahcub Nov 19, 2017
777a12a
!U add a line feed
ahcub Nov 19, 2017
037c9dc
!U fix formatting
ahcub Nov 19, 2017
d5ac67e
!U change the fix to only handle the problem on core/resample.py level
ahcub Nov 19, 2017
61e84d6
!U fix formatting
ahcub Nov 19, 2017
e049112
!U change the index from which we get the dates to appent
ahcub Nov 19, 2017
e252b20
!U remove redundant check and fix index
ahcub Nov 19, 2017
1888355
!U add check for binner len
ahcub Nov 19, 2017
f4ed7c0
!U add missing arguments to the extra dates generation
ahcub Nov 20, 2017
9b18f4b
!U add comments
ahcub Nov 20, 2017
5b035f0
!U change test name and add expected df to compare result to
ahcub Nov 20, 2017
0ec3bd0
!R reformat the code
ahcub Nov 20, 2017
b523915
!U add whatsnew bug fix entry
ahcub Nov 20, 2017
a1b59c4
!U reformat the code
ahcub Nov 20, 2017
8cd36c9
Merge branch 'master' into master
ahcub Nov 20, 2017
aec765f
!U reformat the code according to PR comments
ahcub Nov 21, 2017
2e77f72
!U fix formatting
ahcub Nov 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.21.1.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Bug Fixes
- Bug in ``pd.Series.rolling.skew()`` and ``rolling.kurt()`` with all equal values has floating issue (:issue:`18044`)
- Bug in ``pd.DataFrameGroupBy.count()`` when counting over a datetimelike column (:issue:`13393`)
- Bug in ``pd.concat`` when empty and non-empty DataFrames or Series are concatenated (:issue:`18178` :issue:`18187`)
- Bug in ``DataFrame.resample(...)`` when there is a time change, resampling frequecy is big enough (:issue:`15549`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

say DST here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sp on frequency

- Bug in :class:`IntervalIndex` constructor when a list of intervals is passed with non-default ``closed`` (:issue:`18334`)
- Bug in :meth:`IntervalIndex.copy` when copying and ``IntervalIndex`` with non-default ``closed`` (:issue:`18339`)

Expand Down
10 changes: 10 additions & 0 deletions pandas/core/resample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1141,6 +1141,16 @@ def _get_time_bins(self, ax):
tz=tz,
name=ax.name)

# GH 15549
# In edge case of tz-aware resapmling binner last index can be
# less than the last variable in data object.
# This leads to `Values falls after last bin` error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this last sentence is not relevant; you are fixing this!
add why the bonner is too short (eg because of DST)

if len(binner) > 1 and binner[-1] < last:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment here on what is going on.

extra_date_range = pd.date_range(binner[-1], last + self.freq,
freq=self.freq, tz=tz,
name=ax.name)
binner = labels = binner.append(extra_date_range[1:])

# a little hack
trimmed = False
if (len(binner) > 2 and binner[-2] == last and
Expand Down
28 changes: 28 additions & 0 deletions pandas/tests/test_resample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2719,6 +2719,34 @@ def test_resample_weekly_bug_1726(self):
# it works!
df.resample('W-MON', closed='left', label='left').first()

def test_resample_tz_aware(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be more explicit in the test name
this is particularl to dst

# GH 15549
index = pd.DatetimeIndex([1457537600000000000, 1458059600000000000],
tz='UTC').tz_convert('America/Chicago')
df = pd.DataFrame([1, 2], index=index)
res_df = df.resample('12h', closed='right',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

call this result

label='right').last().ffill()

expected_index_values = ['2016-03-09 12:00:00-06:00',
'2016-03-10 00:00:00-06:00',
'2016-03-10 12:00:00-06:00',
'2016-03-11 00:00:00-06:00',
'2016-03-11 12:00:00-06:00',
'2016-03-12 00:00:00-06:00',
'2016-03-12 12:00:00-06:00',
'2016-03-13 00:00:00-06:00',
'2016-03-13 13:00:00-05:00',
'2016-03-14 01:00:00-05:00',
'2016-03-14 13:00:00-05:00',
'2016-03-15 01:00:00-05:00',
'2016-03-15 13:00:00-05:00']
index = pd.DatetimeIndex(expected_index_values,
tz='UTC').tz_convert('America/Chicago')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

call this expected

expected_df = pd.DataFrame([1.0, 1.0, 1.0, 1.0, 1.0,
1.0, 1.0, 1.0, 1.0, 1.0,
1.0, 1.0, 2.0], index=index)
assert_frame_equal(res_df, expected_df)

def test_resample_bms_2752(self):
# GH2753
foo = Series(index=pd.bdate_range('20000101', '20000201'))
Expand Down