Instrument RDK with observability and monitoring

[mbeacom]

This can likely be easily achieved (initially) with AWS Powertools and liberal usage of the metrics / tracer / logger modules.

[austindimmer]

In my use case we want to be able to use AWS Lambda Powertools and this means I need to be able to set TracingConfig and Environment variables. I do not currently see any documented way I can achieve this. I currently copy the deployed CF template and update it to include these properties as follows:

    "Ec2TestRuleLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "FunctionName": "RDK-Rule-Function-Ec2TestRule",
        "Code": {
          "S3Bucket": {
            "Ref": "SourceBucket"
          },
          "S3Key": "Ec2TestRule/Ec2TestRule.zip"
        },
        "Description": "Function for AWS Config Rule Ec2TestRule",
        "Environment": {
          "Variables": {
            "POWERTOOLS_SERVICE_NAME": "config-rdk-rules",
            "LOG_LEVEL": "Debug",
            "POWERTOOLS_LOGGER_SAMPLE_RATE": "1",
            "POWERTOOLS_LOGGER_LOG_EVENT": "true",
            "POWERTOOLS_METRICS_NAMESPACE": "rdk_config"
          }
        },
        "TracingConfig": {
          "Mode": "Active"
        },
        "Handler": "Ec2TestRule.lambda_handler",
        "MemorySize": "256",
        "Role": "arn:aws:iam::123456789012:role/ConfigRulesCentralizedLambdaRole",
        "Runtime": "python3.9",
        "Timeout": "60",
        "Tags": []
      }
    },

And that configuration does get XRay tracing to work which is very useful.

Interestingly the powertools repo has an open PR to add typing support for config rules
feat(event_sources): add AWS Config Rule event data class #2175 that feature combined with the ability to add the required properties to the centralized lambdas would allow XRay tracing to be setup out of the box.