InvalidSignatureException: Signature expired

[dobrynin]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

This issue is just re-opening #5192 which was never solved. If you initialize a secrets-manager client in a lambda runtime which gets frozen and then later (after 5 minutes) the lambda is triggered and tries to fetch a secret you will hit the signature expired error. The suggested workarounds in #5192 (such as using top-level await) are untenable as they would require extensive code rework. It would be great to identify why exactly expired signatures are being sent. @trivikr explained the process for making signed requests in #5192 (comment) although it does appear as if the constructor is somehow setting a signature which later expires, as described by @deanc in #5192 (comment)

SDK version number

"@aws-sdk/client-secrets-manager": "^3.549.0"

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 20.x

Reproduction Steps

export const secretsManager = AWSXRay.captureAWSv3Client(
  new SecretsManagerClient(commonConfiguration),
);

import secretsManager into a lambda runtime, but don't request any secrets. Wait 5 minutes. Invoke the lambda and request a secret.

Observed Behavior

Invalid signature error

Expected Behavior

Properly set signature.

Possible Solution

No response

Additional Information/Context

No response

[trivikr]

As you've read already, we've done extensive analysis in #5192
This happens only on AWS Lambda, and we're working with them to add a documentation about it.

This is the messaging from AWS Lambda team I can provide for future reference

As stated in our documentation, for functions using on-demand concurrency, Lambda may occasionally initialize execution environments ahead of invocation requests. When this happens, you may also observe a time gap between your function's initialization and invocation phases. In this case, we understand your functions are creating and using the AWS SDK client during the function INIT phase. This creates asynchronous tasks on the Node.js event loop to both sign and send a request to SSM. However, these tasks may be executed during either the INIT or the INVOKE phase. Where a request signature is created during the INIT phase and sent during the INVOKE phase, the occasional delays between INIT and INVOKE can cause the signatures to expire. This explains the intermittent errors you have been seeing.

To resolve this issue, we recommend using Node.js top-level await to ensure asynchronous tasks initiated in the INIT phase are also completed in the INIT phase. This will remove the delay between signing and sending requests, preventing these signature expired errors.

We recognize that updating your code to use top-level await requires switching from CommonJS Modules to ES Modules, which may take some time. As a workaround you can create SDK client in INVOKE phase, but make API call inside the handler.

From the GitHub Issue:

• TLA recommendation: SignatureDoesNotMatch when migrating from AWS SDK v2 to v3 #5192 (comment)
• API call inside handler workaround: SignatureDoesNotMatch when migrating from AWS SDK v2 to v3 #5192 (comment)

[dobrynin]

@trivikr thank you for response, it is very helpful. I think I am missing something about how SSM/Secrets Manager works. One of the solutions you linked is

// ...
const client = new SSM();
let secret;

const handler = async (event) => {
  if (secret === undefined) {
    secret = await client.getSecretValue(params);
  }
  // Use the secret.
}

We do more-or-less exactly this, although we are instantiating a secrets manager client instead of SSM. Is there something specific to secrets manager that causes us to run into the async init problem?

[aBurmeseDev]

Checking in here, is it ongoing issue? If so, please elaborate.

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.