Skip to content

Fix hour transform #1146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 31, 2025
Merged

Fix hour transform #1146

merged 4 commits into from
Mar 31, 2025

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented Mar 28, 2025

Which issue does this PR close?

Seems like we're rounding towards zero by default by Rust. While we want to floor the value.

Checked against the reference implementation:

image

  • Closes #.

What changes are included in this PR?

Are these changes tested?

@Fokko Fokko force-pushed the fd-fix-hour-truncation branch from e01d6e1 to c3b3ba9 Compare March 28, 2025 08:18
Seems like we're rounding towards zero by default by Rust. While
we want to `floor` the value.
@Fokko Fokko force-pushed the fd-fix-hour-truncation branch from c3b3ba9 to f606fd4 Compare March 28, 2025 08:19
@Fokko Fokko added this to the 0.5.0 Release milestone Mar 28, 2025
@Fokko Fokko force-pushed the fd-fix-hour-truncation branch from c978057 to 8d6e797 Compare March 28, 2025 08:38
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Fokko for this fix! Rust std has provided functionality for us so that we don't need to do the hack by ourself.

@@ -337,12 +337,12 @@ pub struct Hour;
impl Hour {
#[inline]
fn hour_timestamp_micro(v: i64) -> i32 {
(v / MICROSECONDS_PER_HOUR) as i32
(v as f64 / MICROSECONDS_PER_HOUR).floor() as i32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(v as f64 / MICROSECONDS_PER_HOUR).floor() as i32
(v.div_euclid(MICROSECONDS_PER_HOUR)) as i32

There is a div_euclid which could help use to do this. The div_floor method seems better, but it's unstable. Give MICROSECONDS_PER_HOUR is always positive, div_euclid is good enough here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it, thanks for the suggestion! 🙌

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what's the risk of using the unstable API? We have limited test cases in place to verify the behavior. Shouldn't you use unstable API's at all?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using unstable api means we need to use unstable rust sdk, what's worse it requires all of our downstream users to use unstable rust sdk, which is anti pattern for a rust library.
Unstable rust sdk contains experimental apis, but there is no guarantee that those apis will be still there when a stable version is released.
See doc about rust channels

kevinjqliu
kevinjqliu previously approved these changes Mar 28, 2025
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. +1 to renjie's suggestion

test_timestamp_and_tz_transform("1970-01-01T22:01:01.000000", &hour, Datum::int(22));
test_timestamp_and_tz_transform("1969-12-31T23:00:00.000000", &hour, Datum::int(-1));
test_timestamp_and_tz_transform("1969-12-31T22:01:01.000000", &hour, Datum::int(-2));
test_timestamp_and_tz_transform("0022-05-01T22:01:01.000000", &hour, Datum::int(-17072906));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-17_072_905.98 but rounded up 👍

@liurenjie1024 liurenjie1024 merged commit 7eae524 into apache:main Mar 31, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants