Issue with API Gateway returning 500 server error after renaming function.

[imangerah]

Description

After renaming a function in the SAM template, my API Gateway calls return 500 errors.

Steps to reproduce

I have a stack that uses a custom domain with AWS::ApiGateway::BasePathMapping and AWS::ApiGateway::DomainName. The Base Path Mapping is as follows:

  APIDomainName:
    Type: AWS::ApiGateway::DomainName
    Properties:
      CertificateArn: !ImportValue FlythroughApiRootDomain-APIDomainCertificate
      DomainName: !FindInMap [EndpointMap, !Ref StageName, url]

  APIBasePathMapping:
    Type: AWS::ApiGateway::BasePathMapping
    Properties:
      DomainName: !Ref APIDomainName
      RestApiId: !Ref ServerlessRestApi
      Stage: Prod

I am unsure if above plays a role in the bug.

I first deployed the stack with a function named HelloWorld-xxx, thereafter I decided to rename it to HelloWorld_xxx. After the stack successfully deployed, I started getting 500 errors from API Gateway.

Enabling logging on API Gateway I was able to determine the cause was API Gateway did not have permission to execute the lambda.

Deleting the function from my template, deploying the stack, adding it back and redeploying the stack fixed the issue.

Observed result

Working Permissions Prior to function rename:

aws lambda get-policy --function-name PaymentResponse-Api-prod 
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionProd-1LBGN1UHYWFWS\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse-Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/Prod/POST/paymentresponse\"}}},{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionTest-DMNP3TTVDK7D\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse-Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/*/POST/paymentresponse\"}}}]}", 
    "RevisionId": "85968857-f658-4167-xxxx-a70779bd905b"
}

Permissions after renaming the function (Produces a 500 error)

aws lambda get-policy --function-name PaymentResponse_Api-prod
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionProd-11DFFFOOMFVVP\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse_Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/Prod/POST/paymentresponse\"}}},{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionTest-16ULZVEZEWU1W\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse_Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/*/POST/paymentresponse\"}}}]}", 
    "RevisionId": "8786fe68-8650-4df7-xxxx-4ea06f82a439"
}

Permissions after deleting the function and redeploying (Works again - 500 error solved)

aws lambda get-policy --function-name PaymentResponse_Api-prod
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionTest-988DLJH4ECT1\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse_Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/*/POST/paymentresponse\"}}},{\"Sid\":\"Api-prod-PaymentResponseFunctionRegisterPermissionProd-DNS6C540K5SZ\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:invokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:xxxxxxxxxxxx:function:PaymentResponse_Api-prod\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:xxxxxxxxxxxx:5x12ry2pj0/Prod/POST/paymentresponse\"}}}]}", 
    "RevisionId": "05cf3af3-a9d9-48f9-xxxx-c46566fe74b1"
}

Expected result

Renaming a function should not result in API gateway permission errors.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Debian 9
2. sam --version: SAM CLI, version 0.14.2

[imangerah]

Additional Info: Able to reproduce after upgrading to SAM CLI, version 0.16.0

[jfuss]

This could be an issue with SAM or an issue with IAM taking some time to propagate. Moving to the Serverless Application Model repo for investigation.

[jlhood]

@imangerah Thanks for reporting this and I'm sorry you experienced this. It's difficult to diagnose the issue without seeing a sample template. However, we discussed this and have a guess that this could be a flavor of #660, which is a known issue that's also impacting other customers. When an API Gateway API definition is changed, a new API Gateway Deployment must be created. SAM tries to do this for you automatically, but there are edge cases in the way that it determines when an API definition has been changed.

In your case, from the name of your functions, I'm guessing you're explicitly hardcoding the function name using the FunctionName property. If you change the value of FunctionName without changing the logical id (key of the AWS::Serverless::Function resource), SAM will not detect that the API has changed and will not redeploy the API after it's been updated.

In this case, on deployment, CloudFormation creates a new function with the new name, updates the API Gateway REST definition to point to the new function name, does NOT do a deployment of this updated API definition (so the actual API is still referencing the function with the old name), then deletes the Lambda function with the old name (causing the API to now return 5xx errors).

The current workaround for this is to not use the FunctionName property and instead let CloudFormation auto-generate the name of the function for you. Then if you want to rename your function, you will update the logical id of the resource which will result in a new API Gateway deployment.

If you still want to use the FunctionName property, an alternative is when you change the value of FunctionName, you also need to remember to change the value of the logical id at the same time to trigger a new deployment of the API.

[imangerah]

I can confirm that I have used FunctionName and thereafter renamed it without renaming the logical id of the resource. Thank you for explaining the observed behaviour and the workaround.

[jlhood]

@imangerah Thanks for confirming. Closing this issue since it's covered by other existing issues.

[qmg-rwaller]

Hi @jlhood, I'm pretty sure I hit this yesterday. Thanks for the tip to redeploy the API Gateway! To clarify, is this an issue/bug with SAM or vanilla CloudFormation? (And are there any plans to make this workaround unnecessary?