Gitpod is too noisy for benchmarking

[Kreyren]

Describe the bug

Referencing https://gist.githubusercontent.com/Kreyren/1c12dd867bd5edf96142f5431f818fc3/raw/fb1ffae4408fc038313765aef72677e19858df75/gistfile1.txt -> Gitpod's is too noisy to conduct reliable data for benchmarks of developed software.

Note: Mensioned log is invoked on same target benchmarking same file. The inconsistency is quite significant.

The current inconsistency of performance seems to be around 14% (14x times more then expected)

Steps to reproduce

1. Open https://gitpod.io/#https://github.com/Kreyren/gitpod-tests1/tree/benchmarks
2. Invoke cargo bench multiple times and notice performance calculation being inconsistent

Expected behavior

In ideal situation the benchmark should be consistent within 0,05% (upstream mensiones +- 1%) difference after each run.

Example Repository

https://github.com/Kreyren/gitpod-tests1/tree/benchmarks

[Kreyren]

Referencing https://bheisler.github.io/criterion.rs/book/getting_started.html

This is upstream documentation that has two fibonnaci sequence's as example of library which is used for more consistent benchmarking. This difference is next to impossible to notice on gitpod.

EDIT: My fault the difference is significant in this case (-99,99% performance), but for long-term development where 1% improvement is important it's very difficult to deduce.

[Kreyren]

Using more samples seems to reduce the noise in the system.

     Running target/release/deps/fibonacci-b30878705f3bfa87
Gnuplot not found, using plotters backend
Benchmarking fibonacci 5: Collecting 1000 samples in estimated 5.0008 s                                                                       fibonacci 5             time:   [2.3829 ns 2.4117 ns 2.4425 ns]
Found 147 outliers among 1000 measurements (14.70%)
  29 (2.90%) high mild
  118 (11.80%) high severe

Benchmarking fibonacci 20: Collecting 1000 samples in estimated 5.0007                                                                        fibonacci 20            time:   [2.3311 ns 2.3610 ns 2.3936 ns]
Found 112 outliers among 1000 measurements (11.20%)
  41 (4.10%) high mild
  71 (7.10%) high severe

Benchmarking fibonacci 100: Collecting 1000 samples in estimated 5.0011                                                                       fibonacci 100           time:   [2.2935 ns 2.3104 ns 2.3286 ns]
Found 140 outliers among 1000 measurements (14.00%)
  45 (4.50%) high mild
  95 (9.50%) high severe

Benchmarking fibonacci 1000: Collecting 1000 samples in estimated 5.000                                                                       fibonacci 1000          time:   [2.2912 ns 2.3126 ns 2.3365 ns]
Found 100 outliers among 1000 measurements (10.00%)
  30 (3.00%) high mild
  70 (7.00%) high severe

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.