Skip to content

Mitigate race condition #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 15, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ The IAM User that is used to run the action requires the following IAM permissio
"codedeploy:CreateDeployment",
"codedeploy:RegisterApplicationRevision",
"codedeploy:GetDeploymentConfig",
"codedeploy:GetDeploymentGroup",
"codedeploy:UpdateDeploymentGroup",
"codedeploy:CreateDeploymentGroup"
],
Expand All @@ -148,6 +149,16 @@ The IAM User that is used to run the action requires the following IAM permissio
}
```

## Race Conditions

As of writing, the AWS CodeDeploy API does not accept new deployment requests for an application and deployment group as long as another deployment is still in progress. So, this action will retry a few times and eventually (hopefully) succeed.

There might be situations where several workflow runs are triggered in quick succession - for example, when merging several approved pull requests in a short time. Since your test suites or workflow runs might take a varying amount of time to finish and to reach the deployment phase (_this_ action), you cannot be sure that the triggered deployments will happen in the order you merged the pull requests (to stick with the example). You could not even be sure that the last deployment made was based on the last commit in your repository.

To work around this, this action includes the GitHub Actions "[run id](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context)" in the `description` field for created deployments. Before creating a new deployment, it will fetch the _last attempted deployment_ from the AWS API and compare its run id with the current run. If the current run has a _lower_ id than the last attempted deployment, the deployment will be aborted.

This workaround should catch a good share of possible out-of-order deployments. There is a slight chance for mishaps, however: If a _newer_ deployment happens to start _after_ we checked the run id and finishes _before_ we commence our own deployment (just a few lines of code later), this might go unnoticed. To really prevent this from happening, ordering deployments probably needs to be supported on the AWS API side, see https://github.com/aws/aws-codedeploy-agent/issues/248.

## Action Input and Output Parameters

### Input
Expand Down
2 changes: 1 addition & 1 deletion cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@

const action = require('./create-deployment');
try {
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, null, core);
} catch (e) {
console.log(`👉🏻 ${e.message}`);
process.exit(1);
Expand Down
36 changes: 35 additions & 1 deletion create-deployment.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ function fetchBranchConfig(branchName) {
process.exit();
}

exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
const branchConfig = fetchBranchConfig(branchName);
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
Expand Down Expand Up @@ -63,19 +63,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
}

let tries = 0;
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';

while (true) {

if (++tries > 5) {
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
return;
}

if (runNumber) {
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
}).promise();

var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
deploymentId: lastAttemptedDeploymentId,
}).promise();

var matches, lastAttemptedDeploymentRunNumber;

if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
lastAttemptedDeploymentRunNumber = matches[1];
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
return;
} else {
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
}
}

/*
There's a slight remaining chance that the above check does not suffice: If we just
passed the check, but another (newer) build creates AND finishes a deployment
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
race condition that requires an extension on the AWS API side in order to be resolved,
see https://github.com/aws/aws-codedeploy-agent/issues/248.
*/
}

try {
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
...deploymentConfig,
...{
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
description: description,
revision: {
revisionType: 'GitHub',
gitHubLocation: {
Expand Down
40 changes: 38 additions & 2 deletions dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ function fetchBranchConfig(branchName) {
process.exit();
}

exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
const branchConfig = fetchBranchConfig(branchName);
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
Expand Down Expand Up @@ -79,19 +79,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
}

let tries = 0;
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';

while (true) {

if (++tries > 5) {
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
return;
}

if (runNumber) {
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
}).promise();

var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
deploymentId: lastAttemptedDeploymentId,
}).promise();

var matches, lastAttemptedDeploymentRunNumber;

if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
lastAttemptedDeploymentRunNumber = matches[1];
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
return;
} else {
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
}
}

/*
There's a slight remaining chance that the above check does not suffice: If we just
passed the check, but another (newer) build creates AND finishes a deployment
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
race condition that requires an extension on the AWS API side in order to be resolved,
see https://github.com/aws/aws-codedeploy-agent/issues/248.
*/
}

try {
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
...deploymentConfig,
...{
applicationName: applicationName,
deploymentGroupName: deploymentGroupName,
description: description,
revision: {
revisionType: 'GitHub',
gitHubLocation: {
Expand Down Expand Up @@ -157,8 +191,10 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);

const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];

try {
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
} catch (e) {}
})();

Expand Down
4 changes: 3 additions & 1 deletion index.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);

const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];

try {
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
} catch (e) {}
})();