Skip to content

stats: select ok latency becomes -nan after 30 seconds #286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
no1seman opened this issue May 16, 2022 · 0 comments · Fixed by #288
Closed

stats: select ok latency becomes -nan after 30 seconds #286

no1seman opened this issue May 16, 2022 · 0 comments · Fixed by #288
Assignees
@DifferentialOrange

Comments

@no1seman
Copy link

Imagine we have a Cartridge cluster with some data ... if we make the following operations in router console:

unix/:./tmp/run/app-cartridge.router.control> do
                                            >     fiber=require('fiber')
                                            >     crud.cfg({stats = true})
                                            >     t={}
                                            > 
                                            >     res=crud.select('space')
                                            >     table.insert(t, crud.stats().spaces.space.select.ok.latency)
                                            >     fiber.sleep(30);
                                            >     table.insert(t, crud.stats().spaces.space.select.ok.latency)
                                            > end
---
...

unix/:./tmp/run/app-cartridge.router.control> t
---
- - 0.0036880000006931
  - -nan
...

Seems quite strange that "select" "ok" "latency" becomes -nan after 30 seconds
DifferentialOrange added a commit that referenced this issue May 17, 2022
@DifferentialOrange
stats: make quantile age params configurable
0230eee 
The main motivation of making these parameters configurable is to be
able to write time-adequate test cases for #286 issue, but they may be
useful in getting rid of #286 effect at all or in quantile time window
calibration.

After this patch, statistics summary quantile aging params
`age_bucket_count` and `max_age_time` [1] could be configured:

  crud.cfg{
      stats_quantile_age_bucket_count = 3,
      stats_quantile_max_age_time = 30,
  }

Only type validation is conducted in crud.cfg, every other validation is
performed by metrics itself.

Part of #286
DifferentialOrange added a commit that referenced this issue May 17, 2022
@DifferentialOrange
stats: display total average if quantile is -nan
9343fb7 
Metrics quantiles display `-nan` if there were no observations for
a whole age. Such behavior is expected and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests. On the other hand, getting
`-nan` as a `latency` in `crud.stats` is disappointing for a user since
it is rather useless without any historical data. This patch replaces
`-nan` latency value with total average.

Closes #286
DifferentialOrange added a commit that referenced this issue May 17, 2022
@DifferentialOrange
stats: make quantile age params configurable
ec0af6d 
The main motivation of making these parameters configurable is to be
able to write time-adequate test cases for #286 issue, but they may be
useful in getting rid of #286 effect at all or in quantile time window
calibration.

After this patch, statistics summary quantile aging params
`age_bucket_count` and `max_age_time` [1] could be configured:

  crud.cfg{
      stats_quantile_age_bucket_count = 3,
      stats_quantile_max_age_time = 30,
  }

Only type validation is conducted in crud.cfg, every other validation is
performed by metrics itself.

1. https://www.tarantool.io/ru/doc/latest/book/monitoring/api_reference/#summary

Part of #286
DifferentialOrange added a commit that referenced this issue May 17, 2022
@DifferentialOrange
stats: display total average if quantile is -nan
aca29f3 
Metrics quantiles display `-nan` if there were no observations for
a whole age. Such behavior is expected and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests. On the other hand, getting
`-nan` as a `latency` in `crud.stats` is disappointing for a user since
it is rather useless without any historical data. This patch replaces
`-nan` latency value with total average.

Closes #286
@DifferentialOrange DifferentialOrange mentioned this issue May 17, 2022
Merged
3 tasks
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: make quantile age params configurable
4c9d95e 
The main motivation of making these parameters configurable is to be
able to write time-adequate test cases for #286 issue, but they may be
useful in getting rid of #286 effect at all or in quantile time window
calibration.

After this patch, statistics summary quantile aging params
`age_bucket_count` and `max_age_time` [1] could be configured:

  crud.cfg{
      stats_quantile_age_bucket_count = 3,
      stats_quantile_max_age_time = 30,
  }

Only type validation is conducted in crud.cfg, every other validation is
performed by metrics itself.

1. https://www.tarantool.io/ru/doc/latest/book/monitoring/api_reference/#summary

Part of #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: separate quantile and average fields
ebc730a 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: separate quantile and average fields
8fe6dbb 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: separate quantile and average fields
4429a00 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: make quantile age params configurable
712c50b 
The main motivation of making these parameters configurable is to be
able to write time-adequate test cases for #286 issue, but they may be
useful in getting rid of #286 effect at all or in quantile time window
calibration.

After this patch, statistics summary quantile aging params
`age_bucket_count` and `max_age_time` [1] could be configured:

  crud.cfg{
      stats_quantile_age_bucket_count = 3,
      stats_quantile_max_age_time = 30,
  }

Only type validation is conducted in crud.cfg, every other validation is
performed by metrics itself.

1. https://www.tarantool.io/ru/doc/latest/book/monitoring/api_reference/#summary

Part of #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: separate quantile and average fields
2e5113d 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 18, 2022
@DifferentialOrange
stats: separate quantile and average fields
85e6ac8 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 19, 2022
@DifferentialOrange
stats: separate quantile and average fields
95e6c30 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 20, 2022
@DifferentialOrange
stats: separate quantile and average fields
35ac849 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 20, 2022
@DifferentialOrange
stats: separate quantile and average fields
4a76d92 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 20, 2022
@DifferentialOrange
stats: separate quantile and average fields
c66d330 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
Totktonada pushed a commit that referenced this issue May 20, 2022
@DifferentialOrange @Totktonada
stats: make quantile age params configurable
5524bf4 
The main motivation of making these parameters configurable is to be
able to write time-adequate test cases for #286 issue, but they may be
useful in getting rid of #286 effect at all or in quantile time window
calibration.

After this patch, statistics summary quantile aging params
`age_bucket_count` and `max_age_time` [1] could be configured:

  crud.cfg{
      stats_quantile_age_bucket_count = 3,
      stats_quantile_max_age_time = 30,
  }

Only type validation is conducted in crud.cfg, every other validation is
performed by metrics itself.

1. https://www.tarantool.io/ru/doc/latest/book/monitoring/api_reference/#summary

Part of #286
Totktonada pushed a commit that referenced this issue May 20, 2022
@DifferentialOrange @Totktonada
stats: separate quantile and average fields
e709a87 
Add separate quantile (`latency_quantile_recent`) and average (
`latency_average`) fields to `crud.stats()` output. `latency` field and
tarantool/metrics output remains unchanged.

Before this patch, `latency` displayed `latency_quantile_recent` or
`latency_average` and there wasn't any was to see pre-computed average
if quantiles are enabled. But it may be useful if quantile is `nan`.

Quantiles may display `-nan` if there were no observations for a several
ages. Such behavior is expected [1] and valid: for example, Grafana
should ignore such values and they will be displayed as `No data` for
a window when there wasn't any requests.

1. tarantool/metrics#303

Closes #286
DifferentialOrange added a commit that referenced this issue May 23, 2022
@DifferentialOrange
Release 0.11.2
a6b0319 
Overview

    This is a bugfix release. Several things related to crud
    statistics (0.11.0 and 0.11.1 release features) was improved or
    fixed.

Breaking changes

    There are no breaking changes in the release.

New features

    Make metrics quantile collector age params configurable (#286).

    Add separate `latency_average` and `latency_quantile_recent`
    fields to `crud.stats()` output (#286).

Bugfixes

    Preset `stats_quantile_tolerated_error` `crud.cfg` parameter is no
    more lost on configuration update (#284).
@DifferentialOrange DifferentialOrange mentioned this issue May 23, 2022
Merged
1 task
DifferentialOrange added a commit that referenced this issue May 23, 2022
@DifferentialOrange
Release 0.11.2
c0200c9 
Overview

    This is a bugfix release. Several things related to crud
    statistics (0.11.0 and 0.11.1 release features) was improved or
    fixed.

Breaking changes

    There are no breaking changes in the release.

New features

    Make metrics quantile collector age params configurable (#286).

    Add separate `latency_average` and `latency_quantile_recent`
    fields to `crud.stats()` output (#286).

Bugfixes

    Preset `stats_quantile_tolerated_error` `crud.cfg` parameter is no
    more lost on configuration update (#284).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DifferentialOrange DifferentialOrange

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix nan crud.stats latency tarantool/crud
3 participants
@Totktonada @no1seman @DifferentialOrange