Skip to content

aws_s3_deployment: BucketDeployment is deleted if custom resource lambda fails during update #22670

Closed
@rittneje

Description

@rittneje

Describe the bug

We have an aws_s3_deployment.BucketDeployment with retain_on_delete=False. Recently, when updating the files in the deployment, the custom resource lambda failed while invaliding the CloudFront distribution due to some transient issue. Due to a bug in the lambda, it incorrectly sent back a bogus physical resource id to CloudFormation, which caused CloudFormation to conclude it had done an update replacement, so then it tried to roll back by deleting the new physical resource id. Since the custom resource lambda does not consider the physical resource id, it just decided to delete the whole deployment, resulting in data loss.

The source of the issue is here:

responseBody['PhysicalResourceId'] = physicalResourceId or context.log_stream_name

If an exception is thrown and caught, no physicalResourceId is passed, so it defaults to the log stream name, which is different from the real uuid-based physical resource id.

Expected Behavior

It should not have sent back a new bogus physical resource id, so CloudFormation would just do a normal reverse update during rollback instead of a delete.

Current Behavior

data loss

Reproduction Steps

Since the original failure was due to some transient CloudFront issue, I don't know how to reliably reproduce this. Our CDK construct looks like this:

        aws_s3_deployment.BucketDeployment(
            self,
            "deployment",
            sources=[aws_s3_deployment.Source.asset(asset_path)],
            destination_bucket=bucket,
            destination_key_prefix=path_prefix,
            distribution=distribution,
            retain_on_delete=False
        )

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.39.1 (build f188fac)

Framework Version

No response

Node.js Version

v16.17.0

OS

Alpine 3.16

Language

Python

Language Version

3.10.6

Other information

No response

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions