Bug: publishing a metric with 9 dimensions doesn't work

[dnlopes]

Expected Behaviour

Publishing a CW metric with 9 dimensions should work without problems.

Current Behaviour

When trying to publish a CW metric with 9 dimensions of my own, it seems the metrics never reach CloudWatch, although I can't see errors anywhere.

Code snippet

with single_metric(
    name="MyMetricName",
    unit=MetricUnit.Percent,
    value=percentage,
    namespace=os.environ["POWERTOOLS_METRICS_NAMESPACE"],
) as metric:
    metric.add_dimension(name="pool_id", value=metrics.uuid)
    metric.add_dimension(name="pool_name", value=metrics.name)
    metric.add_dimension(name="pool_min_size", value=metrics.min_size)
    metric.add_dimension(name="pool_curr_size", value=metrics.curr_size)
    metric.add_dimension(name="pool_real_size", value=metrics.real_curr_size)
    metric.add_dimension(name="pool_decommissioning_counter", value=metrics.decommissioning_counter)
    metric.add_dimension(name="pool_ready_counter", value=metrics.ready_counter)
    metric.add_dimension(name="pool_creating_counter", value=metrics.creating_counter)

    return metric

Possible Solution

No response

Steps to Reproduce

• Try to publish a CW metric with 9 dimensions
• let powertools add the service dimension on its own
• no metrics are published

AWS Lambda Powertools for Python version

1.20.2

AWS Lambda function runtime

3.8

Packaging format used

PyPi

Debugging logs

####################### LAMBDA LOG ENTRY WITH LESS THAN 9 DIMENSIONS ####################
{
    "_aws": {
        "Timestamp": 1654210920980,
        "CloudWatchMetrics": [
            {
                "Namespace": "phoenix-orchestration",
                "Dimensions": [
                    [
                        "service_name",
                        "pool_id",
                        "pool_name",
                        "pool_curr_size",
                        "pool_min_size",
                        "pool_real_size",
                        "service"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "SlotAvailability",
                        "Unit": "Percent"
                    }
                ]
            }
        ]
    },
    "service_name": "tenant-pool",
    "pool_id": "c481bab7-ac0c-45d2-bbcc-e5cab7f61156",
    "pool_name": "Test Pool",
    "pool_curr_size": "11",
    "pool_min_size": "11",
    "pool_real_size": "11",
    "service": "tenant-pool",
    "SlotAvailability": [
        100
    ]
}

####################### LAMBDA LOG ENTRY WITH 9 DIMENSIONS ####################
{
    "_aws": {
        "Timestamp": 1654523221124,
        "CloudWatchMetrics": [
            {
                "Namespace": "phoenix-orchestration",
                "Dimensions": [
                    [
                        "pool_id",
                        "pool_name",
                        "pool_min_size",
                        "pool_curr_size",
                        "pool_real_size",
                        "pool_decommissioning_counter",
                        "pool_ready_counter",
                        "pool_creating_counter",
                        "service"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "SlotAvailability",
                        "Unit": "Percent"
                    }
                ]
            }
        ]
    },
    "pool_id": "f1316488-6aa6-464f-be78-13902b20ec33",
    "pool_name": "Prospects (Ireland) v2",
    "pool_min_size": "0",
    "pool_curr_size": "0",
    "pool_real_size": "0",
    "pool_decommissioning_counter": "0",
    "pool_ready_counter": "0",
    "pool_creating_counter": "0",
    "service": "tenant-pool",
    "SlotAvailability": [
        100
    ]
}

[heitorlessa]

Hey @dnlopes thanks a lot for raising this with us.

I've got two questions before I try reproducing it tomorrow morning:

• Does this only happen with 9 dimensions?
• Does this happen with the latest version? We're in 1.26 now

At a first glance we're serialising correctly and CloudWatch EMF should pick it up - CloudWatch has a limit of 10 dimensions, we add "service", hence 9+service should be fine.

Thank you!

[dnlopes]

• Does this only happen with 9 dimensions?

Yes. After facing this issue I simply removed one of the dimensions and everything went back to normal.

• Does this happen with the latest version? We're in 1.26 now

I will bump my dependency and try again layer.

Btw, the last log record I pasted is from a wrong execution, but I think you already have everything you need to try to replicate it. I will try again with the latest version of powertools later and report back.

[dnlopes]

Just tested with the latest powertool version and still got the same issue. Publishing metrics with 9 metrics does nothing, and as soon as I comment on them out, they start appearing in CW metrics board.

[heitorlessa]

Great, thank you @dnlopes; this helps a lot. I gave a heads up to the CloudWatch team, and will try to reproduce it tomorrow with and without powertools to rule out an issue with EMF async processor in CloudWatch Logs Engine.

[heitorlessa]

Hey @dnlopes I wasn't able to reproduce it with 9 dimensions: https://github.com/heitorlessa/pt-1239

I initially suspected it was the value length of one of those dimensions but nope. I've used the exact same values as dimensions you shared.

I wonder if it's one of those common issues: console cache, wrong region, viewing wrong namespace (doubt it), etc.

Let me know if what I tried to reproduce is invalid, or if there's any other data you want us to try reproduce. There are only two small differences in the code:

• No explicit namespace as Metrics utility can pick up POWERTOOLS_METRICS_NAMESPACE env variable value if set
• No explicit return within the context manager (there's no need)

[dnlopes]

@heitorlessa your sample code is only using 8 dimensions, so it will send a total of 9 dimensions in the end, right? This including the one added by Powertools itself.

I copied paste your sample code and everything worked. As soon as I added one more line

metric.add_dimension(name="yet_another_dimension", value=metrics.uuid)

the dimensions are not published. I even change the POWERTOOLS_METRICS_NAMESPACE variable to a different value, to ensure I was comparing 2 clean namespaces (check prints)

[dnlopes]

I noticed now that the sample code in the OP also has 8 dimensions, maybe that's the confusion here.

Rephrasing for clarity: publishing a CW metric with a total of 10 dimensions doesn't seem to work (9 custom dimensions + 1 from powertools)

[heitorlessa]

Ah! Sorry I understood that 9 dimensions in total doesn't work. Let me add a dummy one, make some load, and come back with the results

[heitorlessa]

Actually, it's our bug! CloudWatch Metrics support 10 metric dimensions, however CloudWatch EMF adds 1 "hidden" dimension making the limit 9. We must change our code validation to ensure 8 custom dimensions are the max and raise an exception.

My apologies for the misunderstanding - we'll publish a fix and make an emergency release by EOD.

[dnlopes]

Great, everything makes sense now then 😅

Thanks.

[github-actions]

This is now released under 1.26.1 version!

[heitorlessa]

@dnlopes Alex (@am29d) is kicking off the Lambda Layer release pipeline across all AWS Commercial Regions we support - it will be version 20. We'll ping you as soon as that's available.

Gave you credits in the release notes: https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v1.26.1

On the breaking change not breaking change (plus any topic), feel free to email us aws-lambda-powertools-feedback@ with your availability and timezone, and we can schedule a call to hear your feedback. We'll be working on having a regular Office Hours in the future too if you're busy in the coming weeks/months.

Thank you again! <3

[am29d]

New layer for version powertools 1.26.1 released. As Heitor mentioned current layer version is 20:

arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPython:20