-
Notifications
You must be signed in to change notification settings - Fork 196
CI checks on benchmark histograms and associated benchmark fixups #268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…calefactors. We also make the page title generation deterministic cmu-db#171
…benchmark outputs in the future either
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adds a schema file for the hsqldb for unit tests so that we:
- Get an auto increment column
- Use varchar instead of varbinary for columns to avoid invalid character cast exceptions
Note: this doesn't change the ddl-generic.sql file, in part because auto increment column syntax is not well standardized (or at least adhered to).
Up for discussion on whether these changes should go there instead since basically any new db system will currently fail to operate without those.
@anjagruenheid @jcamachor if either of you have a moment, could you please give this a once over? Thanks! |
As previously noted in #122 the Wikipedia benchmark has a bug that results in most transactions getting aborted.
The fix developed in #173 by @apavlo seems to mostly fix it (though there are still some duplicate key errors generated by the benchmark's AddWatchPage procedure).
This PR pulls in those changes plus some additional ones (below) and additionally adds some CI checks on the json histogram outputs so that any benchmark the generates errors more than 1% of the completed transactions should be considered a failure (previously a CI failure was reported only if java exited non-zero, which doesn't happen when the many transactions fail, only when an unexpected exception occurs).
The 1% value is just an initial threshold and can be adjusted both on a per database and a per benchmark basis.
We do this in several places (e.g. TATP benchmark) in order to keep the changes required for this PR somewhat smaller and leave it for future work to improve those benchmarks.
To address the Wikipedia failures in some databases, we expand on auto-increment column support to SqlServer, add an auto-increment DDL schema for hsqldb for unit test purposes.
For others, we introduce improved schemas and/or dialects to fix query errors.
Note: in the case of sqlite, which currently lacks support for things like native
SLEEP
orTABLOCK
we leave a more simplified version of the query for now and mark it for future work to improve.