Skip to content

Update insertion benchmark #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 28, 2024
Merged

Update insertion benchmark #144

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d425749
Simplify instructions for running benchmark
gmile May 3, 2024
a9902f3
Bump mysql version from 5.7 to 8
gmile May 3, 2024
bceff9a
Handle decoding of %Time{} struct
gmile May 4, 2024
117488c
Refresh all benchmark statistics
gmile May 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 3 additions & 9 deletions bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,8 @@ need PostgreSQL and MySQL up and running.

To run the benchmarks tests just type in the console:

```
# POSIX-compatible shells
$ MIX_ENV=bench mix run bench/bench_helper.exs
```

```
# other shells
$ env MIX_ENV=bench mix run bench/bench_helper.exs
```sh
mix run bench/bench_helper.exs
```

Benchmarks are inside the `scripts/` directory and are divided into two
Expand All @@ -44,6 +38,6 @@ script instead of `bench/bench_helper.exs`.
The easiest way to setup mysql and postgresql for the benchmarks is via Docker. Run the following commands to get an instance of each running.

```
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8
docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:13.2
```
119 changes: 119 additions & 0 deletions bench/results/all.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
Benchmark

Benchmark run from 2024-05-04 10:50:11.956493Z UTC

## System

Benchmark suite executing on the following system:

<table style="width: 1%">
<tr>
<th style="width: 1%; white-space: nowrap">Operating System</th>
<td>macOS</td>
</tr><tr>
<th style="white-space: nowrap">CPU Information</th>
<td style="white-space: nowrap">Apple M3 Max</td>
</tr><tr>
<th style="white-space: nowrap">Number of Available Cores</th>
<td style="white-space: nowrap">16</td>
</tr><tr>
<th style="white-space: nowrap">Available Memory</th>
<td style="white-space: nowrap">128 GB</td>
</tr><tr>
<th style="white-space: nowrap">Elixir Version</th>
<td style="white-space: nowrap">1.16.2</td>
</tr><tr>
<th style="white-space: nowrap">Erlang Version</th>
<td style="white-space: nowrap">26.2.4</td>
</tr>
</table>

## Configuration

Benchmark suite executing with the following configuration:

<table style="width: 1%">
<tr>
<th style="width: 1%">:time</th>
<td style="white-space: nowrap">10 s</td>
</tr><tr>
<th>:parallel</th>
<td style="white-space: nowrap">1</td>
</tr><tr>
<th>:warmup</th>
<td style="white-space: nowrap">2 s</td>
</tr>
</table>

## Statistics



Run Time

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Average</th>
<th style="text-align: right">Devitation</th>
<th style="text-align: right">Median</th>
<th style="text-align: right">99th&nbsp;%</th>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Repo.all/2</td>
<td style="white-space: nowrap; text-align: right">639.68</td>
<td style="white-space: nowrap; text-align: right">1.56 ms</td>
<td style="white-space: nowrap; text-align: right">&plusmn;12.01%</td>
<td style="white-space: nowrap; text-align: right">1.54 ms</td>
<td style="white-space: nowrap; text-align: right">2.20 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">SQLite3 Repo.all/2</td>
<td style="white-space: nowrap; text-align: right">465.99</td>
<td style="white-space: nowrap; text-align: right">2.15 ms</td>
<td style="white-space: nowrap; text-align: right">&plusmn;4.82%</td>
<td style="white-space: nowrap; text-align: right">2.13 ms</td>
<td style="white-space: nowrap; text-align: right">2.43 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Repo.all/2</td>
<td style="white-space: nowrap; text-align: right">214.16</td>
<td style="white-space: nowrap; text-align: right">4.67 ms</td>
<td style="white-space: nowrap; text-align: right">&plusmn;12.05%</td>
<td style="white-space: nowrap; text-align: right">4.66 ms</td>
<td style="white-space: nowrap; text-align: right">5.86 ms</td>
</tr>

</table>


Run Time Comparison

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Slower</th>
<tr>
<td style="white-space: nowrap">MyXQL Repo.all/2</td>
<td style="white-space: nowrap;text-align: right">639.68</td>
<td>&nbsp;</td>
</tr>

<tr>
<td style="white-space: nowrap">SQLite3 Repo.all/2</td>
<td style="white-space: nowrap; text-align: right">465.99</td>
<td style="white-space: nowrap; text-align: right">1.37x</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Repo.all/2</td>
<td style="white-space: nowrap; text-align: right">214.16</td>
<td style="white-space: nowrap; text-align: right">2.99x</td>
</tr>

</table>
109 changes: 50 additions & 59 deletions bench/results/insert.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
Benchmark

# Benchmark

Benchmark run from 2021-03-24 02:05:58.706995Z UTC
Benchmark run from 2024-05-04 10:49:35.563150Z UTC

## System

Expand All @@ -10,22 +9,22 @@ Benchmark suite executing on the following system:
<table style="width: 1%">
<tr>
<th style="width: 1%; white-space: nowrap">Operating System</th>
<td>Linux</td>
<td>macOS</td>
</tr><tr>
<th style="white-space: nowrap">CPU Information</th>
<td style="white-space: nowrap">AMD Ryzen 7 PRO 4750U with Radeon Graphics</td>
<td style="white-space: nowrap">Apple M3 Max</td>
</tr><tr>
<th style="white-space: nowrap">Number of Available Cores</th>
<td style="white-space: nowrap">16</td>
</tr><tr>
<th style="white-space: nowrap">Available Memory</th>
<td style="white-space: nowrap">14.92 GB</td>
<td style="white-space: nowrap">128 GB</td>
</tr><tr>
<th style="white-space: nowrap">Elixir Version</th>
<td style="white-space: nowrap">1.11.3</td>
<td style="white-space: nowrap">1.16.2</td>
</tr><tr>
<th style="white-space: nowrap">Erlang Version</th>
<td style="white-space: nowrap">23.2.6</td>
<td style="white-space: nowrap">26.2.4</td>
</tr>
</table>

Expand All @@ -50,7 +49,6 @@ Benchmark suite executing with the following configuration:




__Input: Changeset__

Run Time
Expand All @@ -67,35 +65,35 @@ Run Time

<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap; text-align: right">7218.07</td>
<td style="white-space: nowrap; text-align: right">0.139 ms</td>
<td style="white-space: nowrap; text-align: right">±43.60%</td>
<td style="white-space: nowrap; text-align: right">0.123 ms</td>
<td style="white-space: nowrap; text-align: right">0.37 ms</td>
<td style="white-space: nowrap; text-align: right">26.72 K</td>
<td style="white-space: nowrap; text-align: right">37.42 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;89.66%</td>
<td style="white-space: nowrap; text-align: right">32.88 &micro;s</td>
<td style="white-space: nowrap; text-align: right">74.21 &micro;s</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">421.57</td>
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
<td style="white-space: nowrap; text-align: right">±12.13%</td>
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
<td style="white-space: nowrap; text-align: right">2.90 ms</td>
<td style="white-space: nowrap; text-align: right">9.65 K</td>
<td style="white-space: nowrap; text-align: right">103.63 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;68.38%</td>
<td style="white-space: nowrap; text-align: right">102.75 &micro;s</td>
<td style="white-space: nowrap; text-align: right">177.39 &micro;s</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">284.25</td>
<td style="white-space: nowrap; text-align: right">3.52 ms</td>
<td style="white-space: nowrap; text-align: right">±13.34%</td>
<td style="white-space: nowrap; text-align: right">3.53 ms</td>
<td style="white-space: nowrap; text-align: right">5.05 ms</td>
<td style="white-space: nowrap; text-align: right">5.49 K</td>
<td style="white-space: nowrap; text-align: right">182.25 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;49.23%</td>
<td style="white-space: nowrap; text-align: right">182.33 &micro;s</td>
<td style="white-space: nowrap; text-align: right">233.08 &micro;s</td>
</tr>

</table>


Comparison
Run Time Comparison

<table style="width: 1%">
<tr>
Expand All @@ -104,28 +102,26 @@ Comparison
<th style="text-align: right">Slower</th>
<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap;text-align: right">7218.07</td>
<td style="white-space: nowrap;text-align: right">26.72 K</td>
<td>&nbsp;</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">421.57</td>
<td style="white-space: nowrap; text-align: right">17.12x</td>
<td style="white-space: nowrap; text-align: right">9.65 K</td>
<td style="white-space: nowrap; text-align: right">2.77x</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">284.25</td>
<td style="white-space: nowrap; text-align: right">25.39x</td>
<td style="white-space: nowrap; text-align: right">5.49 K</td>
<td style="white-space: nowrap; text-align: right">4.87x</td>
</tr>

</table>



<hr/>


__Input: Struct__

Expand All @@ -143,35 +139,35 @@ Run Time

<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap; text-align: right">7765.76</td>
<td style="white-space: nowrap; text-align: right">0.129 ms</td>
<td style="white-space: nowrap; text-align: right">±32.88%</td>
<td style="white-space: nowrap; text-align: right">0.122 ms</td>
<td style="white-space: nowrap; text-align: right">0.28 ms</td>
<td style="white-space: nowrap; text-align: right">26.71 K</td>
<td style="white-space: nowrap; text-align: right">37.44 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;87.15%</td>
<td style="white-space: nowrap; text-align: right">32.92 &micro;s</td>
<td style="white-space: nowrap; text-align: right">70.50 &micro;s</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">422.86</td>
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
<td style="white-space: nowrap; text-align: right">±10.49%</td>
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
<td style="white-space: nowrap; text-align: right">3.02 ms</td>
<td style="white-space: nowrap; text-align: right">9.34 K</td>
<td style="white-space: nowrap; text-align: right">107.08 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;13.80%</td>
<td style="white-space: nowrap; text-align: right">106.87 &micro;s</td>
<td style="white-space: nowrap; text-align: right">132.46 &micro;s</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">274.00</td>
<td style="white-space: nowrap; text-align: right">3.65 ms</td>
<td style="white-space: nowrap; text-align: right">±38.43%</td>
<td style="white-space: nowrap; text-align: right">3.59 ms</td>
<td style="white-space: nowrap; text-align: right">4.75 ms</td>
<td style="white-space: nowrap; text-align: right">5.67 K</td>
<td style="white-space: nowrap; text-align: right">176.45 &micro;s</td>
<td style="white-space: nowrap; text-align: right">&plusmn;70.69%</td>
<td style="white-space: nowrap; text-align: right">176.79 &micro;s</td>
<td style="white-space: nowrap; text-align: right">234.70 &micro;s</td>
</tr>

</table>


Comparison
Run Time Comparison

<table style="width: 1%">
<tr>
Expand All @@ -180,25 +176,20 @@ Comparison
<th style="text-align: right">Slower</th>
<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap;text-align: right">7765.76</td>
<td style="white-space: nowrap;text-align: right">26.71 K</td>
<td>&nbsp;</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">422.86</td>
<td style="white-space: nowrap; text-align: right">18.37x</td>
<td style="white-space: nowrap; text-align: right">9.34 K</td>
<td style="white-space: nowrap; text-align: right">2.86x</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">274.00</td>
<td style="white-space: nowrap; text-align: right">28.34x</td>
<td style="white-space: nowrap; text-align: right">5.67 K</td>
<td style="white-space: nowrap; text-align: right">4.71x</td>
</tr>

</table>



<hr/>

</table>
Loading
Loading