Skip to content

Performance testing script: cleanup, bugfixes, new features #3517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 18, 2018
Merged

Performance testing script: cleanup, bugfixes, new features #3517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4857bf0
perf-test: Enable ccache for CBMC builds
tautschnig Nov 19, 2018
5d9e0c3
perf-test: openjdk is not required as we only build cbmc/goto-cc
tautschnig Nov 20, 2018
810bc88
perf-test: Python2 compatible date formatting
tautschnig Nov 19, 2018
7a2d2f4
perf-test: Maintain shell scripts outside CloudFormation template
tautschnig Nov 19, 2018
72129a4
perf-test: Update the set of SV-COMP categories
tautschnig Nov 24, 2018
3d55685
perf-test: Switch sv-benchmarks branch to svcomp19
tautschnig Nov 19, 2018
03aa2b8
perf-test: Use Ubuntu 18.04 as SV-COMP'19 does
tautschnig Nov 27, 2018
1614a5f
perf-test: New option --compare-to to include past results in tables
tautschnig Dec 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 39 additions & 4 deletions scripts/perf-test/codebuild.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,14 @@ Resources:
Version: 2012-10-17
Statement:
- Action:
- "s3:GetObject"
- "s3:PutObject"
Effect: Allow
Resource: !Join ["/", [!Sub "arn:aws:s3:::${S3Bucket}", "*"]]
- Action:
- "s3:ListBucket"
Effect: Allow
Resource: !Sub "arn:aws:s3:::${S3Bucket}"
- Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
Expand Down Expand Up @@ -65,16 +70,31 @@ Resources:
- echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu trusty main" > /etc/apt/sources.list.d/toolchain.list
- apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA9EF27F
- apt-get update -y
- apt-get install -y libwww-perl g++-5 flex bison git openjdk-7-jdk
- apt-get install -y libwww-perl g++-5 flex bison git ccache
- update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 1
- update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 1
- |
cd /root/
aws s3 cp \
s3://${S3Bucket}/release-build-cache/ccache.zip \
ccache.zip && unzip ccache.zip || true
cd $CODEBUILD_SRC_DIR
build:
commands:
- echo ${Repository} > COMMIT_INFO
- git rev-parse --short HEAD >> COMMIT_INFO
- git log HEAD^..HEAD >> COMMIT_INFO
- make -C src minisat2-download glucose-download cadical-download
- make -C src -j8
- export CCACHE_NOHASHDIR=1 ; make -C src CXX="ccache g++" -j8
- ccache -s
post_build:
commands:
- |
cd /root/
zip -r ccache.zip .ccache
aws s3 cp ccache.zip \
s3://${S3Bucket}/release-build-cache/ccache.zip
cd $CODEBUILD_SRC_DIR
artifacts:
files:
- src/cbmc/cbmc
Expand Down Expand Up @@ -107,16 +127,31 @@ Resources:
- echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu trusty main" > /etc/apt/sources.list.d/toolchain.list
- apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA9EF27F
- apt-get update -y
- apt-get install -y libwww-perl g++-5 flex bison git openjdk-7-jdk
- apt-get install -y libwww-perl g++-5 flex bison git ccache
- update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 1
- update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 1
- |
cd /root/
aws s3 cp \
s3://${S3Bucket}/profiling-build-cache/ccache.zip \
ccache.zip && unzip ccache.zip || true
cd $CODEBUILD_SRC_DIR
build:
commands:
- echo ${Repository} > COMMIT_INFO
- git rev-parse --short HEAD >> COMMIT_INFO
- git log HEAD^..HEAD >> COMMIT_INFO
- make -C src minisat2-download glucose-download cadical-download
- make -C src -j8 CXXFLAGS="-O2 -pg -g -finline-limit=4" LINKFLAGS="-pg"
- export CCACHE_NOHASHDIR=1 ; make -C src CXX="ccache g++" -j8 CXXFLAGS="-O2 -pg -g -finline-limit=4" LINKFLAGS="-pg"
- ccache -s
post_build:
commands:
- |
cd /root/
zip -r ccache.zip .ccache
aws s3 cp ccache.zip \
s3://${S3Bucket}/profiling-build-cache/ccache.zip
cd $CODEBUILD_SRC_DIR
artifacts:
files:
- src/cbmc/cbmc
Expand Down
2 changes: 1 addition & 1 deletion scripts/perf-test/ebs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Resources:
apt-get -y update
apt-get install git
cd /mnt
git clone --depth 1 --branch svcomp18 \
git clone --depth 1 --branch svcomp19 \
https://github.com/sosy-lab/sv-benchmarks.git
git clone --depth 1 \
https://github.com/sosy-lab/benchexec.git
Expand Down
35 changes: 35 additions & 0 deletions scripts/perf-test/ec2-terminate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛏️ Add docs at the top for how to run this (e.g. it looks like it should be called ec2-terminate.sh start|status)

### BEGIN INIT INFO
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ I don't know what the next 8 lines of code do...? That aren't informative to me?

# Provides: ec2-terminate
# Required-Start: $network $syslog
# Required-Stop:
# Default-Start:
# Default-Stop: 0 1 6
# Short-Description: ec2-terminate
# Description: send termination email
### END INIT INFO
#

case "$1" in
start|status)
exit 0
;;
stop)
# run the below
;;
*)
exit 1
;;
esac

# send instance-terminated message
# http://rogueleaderr.com/post/48795010760/how-to-notifyemail-yourself-when-an-ec2-instance/amp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Is this the source of this code? Probably want to make it more obvious that's what this link is for, e.g.

# Code take from

ut=$(cat /proc/uptime | cut -f1 -d" ")
aws --region us-east-1 sns publish \
--topic-arn #SNSTOPIC# \
--message "instance terminating after $ut s at #MAXPRICE# USD/h"
sleep 3 # make sure the message has time to send
aws s3 cp /var/log/cloud-init-output.log \
s3://#S3BUCKET#/#PERFTESTID#/$HOSTNAME.cloud-init-output.log

exit 0
Loading