Skip to content

Support multiple results comparison in benchmark scripts #14196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

Eason0729
Copy link
Contributor

Which issue does this PR close?

Close #13446

Rationale for this change

It would be useful to display the difference among multiple benchmark runs.

See issue.

What changes are included in this PR?

I modify bench.sh and compare.py to accept multiple arguments

Are these changes tested?

Are there any user-facing changes?

No.

@Eason0729
Copy link
Contributor Author

I spent some time trying to make it(compare.py) more readable, and I ended up with original code.
Let me know if there is a better way to do this.

Anyway, I thinks it ready for review.

@Eason0729 Eason0729 marked this pull request as ready for review January 22, 2025 02:53
@alamb
Copy link
Contributor

alamb commented Jan 23, 2025

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

@Eason0729
Copy link
Contributor Author

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

../bench.sh compare main main2 main3
Comparing main2 main3 to main
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query        ┃       main ┃      main2 ┃ Change on main2 ┃      main3 ┃ Change on main3 ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Qsort utf8   │ 25191.05ms │ 25191.05ms │       no change │ 25191.05ms │       no change │
│ Qsort int    │ 30908.16ms │ 30908.16ms │       no change │ 40908.16ms │    1.32x slower │
│ Qsort        │ 25652.09ms │ 25652.09ms │       no change │ 25652.09ms │       no change │
│ decimal      │            │            │                 │            │                 │
│ Qsort        │ 32701.67ms │ 32701.67ms │       no change │ 32701.67ms │       no change │
│ integer      │            │            │                 │            │                 │
│ tuple        │            │            │                 │            │                 │
│ Qsort utf8   │ 25588.63ms │ 25588.63ms │       no change │ 25588.63ms │       no change │
│ tuple        │            │            │                 │            │                 │
│ Qsort mixed  │ 29063.22ms │ 29063.22ms │       no change │ 29063.22ms │       no change │
│ tuple        │            │            │                 │            │                 │
└──────────────┴────────────┴────────────┴─────────────────┴────────────┴─────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary              ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (main)              │ 169104.82ms │
│ Average Time (main)            │  28184.14ms │
│ Total Time (main2)             │ 169104.82ms │
│ Average Time (main2)           │  28184.14ms │
│ Queries Faster (main2)         │           0 │
│ Queries Slower (main2)         │           0 │
│ Queries with No Change (main2) │           6 │
│ Total Time (main3)             │ 179104.82ms │
│ Average Time (main3)           │  29850.80ms │
│ Queries Faster (main3)         │           0 │
│ Queries Slower (main3)         │           1 │
│ Queries with No Change (main3) │           5 │
└────────────────────────────────┴─────────────┘

Copy link

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the Stale PR has not had any activity for some time label Mar 25, 2025
@github-actions github-actions bot closed this Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stale PR has not had any activity for some time
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support multiple (>2) results comparison in benchmark scripts
2 participants