Summary performance drops

[DifferentialOrange]

While developing metrics integration for PR tarantool/crud#244 , I met unexpected hindrances.

For the following configuration,

local DEFAULT_QUANTILES = {
    [0.5] = 1e-5,
    [0.9] = 1e-5,
    [0.99] = 1e-5,
}

local DEFAULT_SUMMARY_PARAMS = {
    age_buckets_count = 5,
    max_age_time = 60,
}

crud performance test with wrapper that observes execution time of each insert/select failed in crucial part of its requests (near 1/5-1/6 of them), dropping in overall rps efficiency at the same time. (There were no errors at usual and no errors when I've collected stats with wrapper to local tables, not computing sophisticated latency.) When I investigated the issue, I found out that some observes spent near 5 second to process without any yields and it was the main reason for timeouts. After lowing my parameters to

local DEFAULT_QUANTILES = {
    [0.99] = 1e-2,
}

local DEFAULT_SUMMARY_PARAMS = {
    age_buckets_count = 2,
    max_age_time = 60,
}

it began to work well (no errors, but rps still dropped to 2-3 times). Lowering only aging parameter to default (no aging) not helped to fix all the issues. Tuning also quantiles tolerance helped out.

@yngvar-antonsson said in personal chat that it may be related to buffer drops on aging.

I think it is worth investigating, maybe as part of the #313

[yngvar-antonsson]

How does it work:

• on each observe call quantile writes in each age bucket
• there is a buffer of size max_samples inside of each bucket
• every nth call buffer of each bucket sorts and flushes (n = max_samples)
• so we got an age_buckets_count * time(sorting + flushing) each n calls of observe, where time(sorting + flushing) ~ O(max_samples * log (max_samples)) + O (log( (max_value - min_value) / min_value / eps))

I propose to distribute buffer flushing between several calls of observe, so there'll be no more than one buffer flushing on each call of observe().

[yngvar-antonsson]

And such behaviour should be documented.

[DifferentialOrange]

The main reason of drops was enabled coverage. After disabling it, there were no more timeouts.

[filonenko-mikhail]

So, the disabling coverage is solution now. Gently kindly closing. Feel free to reopen when brilliant idea comes.