v2: Investigate "full queue" behavior

[roncohen]

Reporting timeout is 1s at the moment., which is probably too high. We should investigate the behavior of v2 when the queue is full.

[roncohen]

The reason we can't just block for a second every time data comes in, is that eventually the agents "write" call on the socket will start timing out. This sort of timeout can also happen for other reasons e.g. APM Server isn't working properly, or there's a faulty proxy in between. In other words, we can't just say that whenever a write in the agent times out, it means the queue in APM Server is full.

When APM Server queue is full we could simply write "Write error: timeout" to the agent logs, but it would be a much better user experience if agents are able to write a more meaningful message to their logs.

After discussions with @urso, we can't get rid of the 1s timeout without making significant changes to libbeat. For the first iteration of v2, i suggest we should do the simple thing and keep the 1s timeout in the APM Server, but return a ERR_FULL_QUEUE response to the agent immediately (see #1240) if we've been unable to deliver the events to the internal beats queue within the 1s timeout.

WDYT @elastic/apm-agent-devs ?

[felixbarny]

Sounds good. But it would be necessary that all HTTP clients used by the agents support receiving a response before the request has been fully written. If it doesn't I'd expect to get an IOException Connection reset by peer, which can be handled as well, but without having access to the response. Maybe the TCP connection can also not be reused in that case, needs investigation.

What is an easy way to reproduce that situation so I can test that? Reporting lots of transactions to the APM server and not starting Elasticsearch?

[roncohen]

@felixbarny yes!

I need every agent dev to test that their http client supports this kind of behavior. You should be able to test it, as you suggested, by starting the v2 branch of APM Server without Elasticsearch. You can set the beats queue to something really small to get the behavior immediately. If you could investigate if that is a good way to test it, then we can try it with the other agents afterwards.

The v2 implementation will at the moment close the connection, so there's no reuse possible. Ideally, I'd like to know if you client supports the behavior with and without the server closing the connection.

[simitt]

fyi - @graphaelli already created a PR in the apm-integration-testing so that you can use the testing repo for starting the v2 branch of the server.

[felixbarny]

To keep track of which agents have already been tested:

• go
• java
• nodejs
• python
• ruby
• RUM

[watson]

In the Node.js agent the risk of errors and how we handle them are the same for both api-v1 or v2. So there's no difference between the two versions for us. We're all good 👍

[urso]

Proposed libbeat changes: elastic/beats#8080

[felixbarny]

@watson how did you test the TCP half-close when the APM server queue is full? I tried shutting down ES and sending lots of transactions but then I got read timeouts from the APM server.

[watson]

@felixbarny I just set up a mock HTTP server that would respond the way I wanted and then send data at it. I can share the code with you if you want

[felixbarny]

that would be helpful

[watson]

@felixbarny https://gist.github.com/watson/16938f3b36e2ce1190f9d7e4e100b1c2