All result timestamps off by 1ms (without alignment turned on)

[juliusv]

If you run a query like this against the latest Cortex release, started with the single-process config at https://github.com/raw/cortexproject/cortex/master/docs/configuration/single-process-config.yaml:

http://localhost:9009/api/prom/api/v1/query_range?query=time()&start=1595846748.806&end=1595846848.806&step=5

...then the first result sample should be at timestamp 1595846748.806, but Cortex reports it at 1595846748.805, and any subsequent samples are off by 1ms as well. Compare to the result by Prometheus at https://demo.promlabs.com/api/v1/query_range?query=time()&start=1595846748.806&end=1595846848.806&step=5.

I didn't turn on the step alignment middleware, so it shouldn't be related to that.

[juliusv]

There was prometheus/prometheus#4941 2 years ago in upstream Prometheus to also fix an off-by-1ms rounding bug in the incoming timestamps. Is this maybe not picked up by Cortex yet? In Cortex's go.mod I see github.com/prometheus/prometheus v1.8.2-0.20200722151933-4a8531a64b32. If this is based off 4a8531a64b32 in Prometheus from July 22, 2020, then I don't see how Cortex and Prometheus would behave different in this regard.

[bwplotka]

Thanks!

Yea, we just need this: prometheus/prometheus#4941 then. The thing is that API package is almost impossible to import (not reusible much unfortunately), so both Thanos & Cortex has to manually apply this fix.

I wish we could somehow move Prometheus api.go into components for easier reuse 🤔

[juliusv]

@bwplotka Totally agreed, the API package is really hard to reuse externally, without injecting a boatload of not-always-necessary stuff. Would be great to change that at some point as well :)

[pracucci]

Thanks for reporting it! I'm fixing it...

[lilic]

Since this was fixed any chance to update the results @juliusv to reflect that? https://promlabs.com/promql-compliance-tests/

[pstibrany]

This bugfix is in Cortex 1.4 release.

[juliusv]

@lilic Yep, I'm just waiting for one vendor to see if they can give me remote_write access so I can include them in my tests as well. But hope to publish an update in a couple of weeks :)