Support Lambda Platform Metrics

[jlvoiseux]

Motivation / Summary

This PR is the continuation of #127, which had become obsolete and non-digestible due to the many changes brought to the extension since the initialization of the PR. The current PR implements the same features, but was rebased to enable a commit-by-commit review approach. It resolves #108.

The general flow is as follows:

• The extension gets started (cold start)
• A metadata container is initialized. This container is populated with the metadata extracted from the first APM payload sent by the agent
• The log processing goroutine now handles a new type of SubEventType alongside RuntimeDone: Report. This report contains the metrics that we want to send to the APM Server. These metrics are packaged inside an APM payload (alongside the extracted metadata) in a dedicated function.

Files / Added Elements

• 944e71f : Add a new package model which is a partial reproduction of the data model used by the Go APM Agent
• 1ec30ab : Add a version utility file that is used to populate metadata
• 10e2c77 and 3ec3bf9 : Functions that extract metadata from a given APM payload and populate a dedicated container. Since metadata extraction is tied to APM data, it is part of the extension package. Metadata is only extracted from the first APM payload received during the current lifetime of the extension.
• a515a6d and c2b056a : Functions that turn the platform report provided by the Lambda runtime into a payload to be sent to the APM Server. Since metrics extraction is tied to logs data, it is part of the logsapi package.
• 2b651c8 and 614be84 : main updates to modify the goroutines in order to extract metadata and process metrics.

Limitations / Elements to discuss

• After several tests with two sample extensions with two different logs processing philosophies, we can conclude that the metrics report corresponding to the invocation i will be received during the execution of the invocation i+1.
• The current implementation changes the Agent name to apm-lambda-extension when the metadata is attached to the generated metrics. While this makes sense in the framework of observability, the current APM UI is not equipped accordingly, and the service is associated to an unknown runtime:
  
  My suggestion would be to stop altering the metadata for now to provide a consistent experience to end users.
• The current package architecture can also be discussed. I wished I was able to keep logsapi and extension completely separate, but in the current implementation, logsapi.ProcessLogs() takes an apmServerTransport in argument in order to enqueue APM metrics payloads.
• Several faas fields will be added to the metricset object in the APM data model (PR to be created). The subsequent modifications are beyond the scope of this initial PR, which aims to provide an initial implementation of Lambda metrics generation.

How to test

• Clone the extension repo
• Follow the instructions presented in the following sample repo
• After executing your sample Lambda function a few times, you should see documents corresponding to your Lambda Platform report in your Elastic Cloud instance:

// Lambda Report
REPORT RequestId: 6862dc9b-3266-470a-9979-f39c21164f8c	Duration: 94.65 ms	Billed Duration: 95 ms	Memory Size: 128 MB	Max Memory Used: 77 MB	Init Duration: 479.30 ms	

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-08T08:07:42.956+0000

• Duration: 8 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	216
Skipped	4
Total	220

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

Generally looks good, mostly minor comments.

[axw]

We should use go.elastic.co/apm/v2, the current major version.

[axw]

LGTM, just one thing (we should use Go agent v2).

I assume we'll want the labels to be recorded as faas.*, rather than labels.faas_*, right? Is your plan that once the server is updated, we'll come back and update the Go agent's model and the extension to send them as metricset fields? Similar to what we do for transaction and span properties: https://github.com/elastic/apm-agent-go/blob/94d06659448a555142c087721a79fff3ee732b49/model/model.go#L798-L804

[jlvoiseux]

@axw Thank you for the review. I have updated the model to v2.
You are absolutely right regarding the faas labels. There was nothing preventing us from modifying the APM model in parallel to this PR, but given the number of changes introduced, I thought that the added complexity of having to test the PR alongside a modified APM Model was going to be too much at once. However, that is definitely the next step once we merge this.