Countless retries and log spam for misconfigure port.

[simitt]

I misconfigured the apm server url by accident serverUrl: 'https://localhost:82000'.
This lead to countless retries of sending data to the APM Server, see logs:

logging error 6a9ff729-cec0-45d5-bc5e-a57e27f2a25d with Elastic APM
logging error 54c4eb0f-5a98-4b41-b07e-5b4f60a2ba5f with Elastic APM
logging error 8cbe19af-37fb-468b-b9bf-8070b5531ca4 with Elastic APM
logging error f9d805e8-488f-4a41-b4f6-20154d8101fe with Elastic APM
logging error 53df0b4b-2cd0-419c-8af4-27be7f36d716 with Elastic APM
logging error 80fd91b6-c09f-48cb-8787-b8ac0d1d1775 with Elastic APM
logging error c9858407-17a3-4968-8cce-297e0bd31f3d with Elastic APM
logging error b3a69e7b-8eec-4580-a219-e72eb4e5b637 with Elastic APM
logging error 6935b29d-9629-4dc2-8b62-9954e6559693 with Elastic APM
logging error 6e31e7cb-4a2d-44e1-bb1c-279bef550de5 with Elastic APM
logging error cf647462-65e1-48e3-a3c4-15a3779bce6f with Elastic APM
logging error 79e1590c-bdad-4c41-b97e-ccc43189bdbb with Elastic APM
...

When I just misconfigure to another wrong port, e.g. 8201 I get a proper Connection Refused error.

[watson]

For reference, this is the actual error that occurs. It's thrown by the agent and then caught again by the agent and so on:

net.js:1023
      throw new RangeError('"port" option should be >= 0 and < 65536: ' + port);
      ^

RangeError: "port" option should be >= 0 and < 65536: 82000
    at lookupAndConnect (net.js:1023:13)
    at TLSSocket.Socket.connect (net.js:995:5)
    at TLSSocket.connect (/Users/watson/code/node_modules/elastic-apm/lib/instrumentation/async-hooks.js:125:29)
    at Object.exports.connect (_tls_wrap.js:1070:12)
    at Agent.createConnection (https.js:111:22)
    at Agent.createSocket (_http_agent.js:214:26)
    at Agent.addRequest (_http_agent.js:184:10)
    at Agent.addRequest (/Users/watson/code/node_modules/elastic-apm/lib/instrumentation/async-hooks.js:144:23)
    at new ClientRequest (_http_client.js:272:16)
    at Object.request (http.js:39:10)
    at Object.request (/Users/watson/code/node_modules/elastic-apm/lib/instrumentation/http-shared.js:92:22)
    at Object.request (https.js:239:15)
    at /Users/watson/code/node_modules/elastic-apm/node_modules/elastic-apm-http-client/index.js:56:35
    at Gzip.onEnd (zlib.js:130:5)
    at emitNone (events.js:110:20)
    at Gzip.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1045:12)
    at elasticAPMCallbackWrapper (/Users/watson/code/node_modules/elastic-apm/lib/instrumentation/index.js:109:27)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[watson]

If the agent ever goes into an infinite loop like this a trick is to add this config option: captureExceptions: false. Then it will catch uncaught exceptions.

[watson]

But obviously the agent shouldn't throw because of this. And it would be nice if we in some way to guard against these infinite call loops 🤔

[watson]

After thinking about this I say that it's expected behaviour for the agent to throw in this case. So the actual issue here is the looping uncaught exception. I don't have a good way of detecting this at the moment. Suggestions much appreciated 😃

[simitt]

Should the agent really be allowed to throw exceptions rather than to just log them once per occurence?
This worries me a bit, from the perspective that this could lead to major overhead in the app if there is some misconfiguration.

[watson]

The looping uncaught exception should obviously not happen, but throwing if you provide invalid input is a fairly normal behaviour in Node.js land. You could argue that this module is a special case though, not sure.

[watson]

@Qard would love your input on this 😃

[Qard]

I don't really have an opinion either way. Crashing is fairly standard in Node.js, but also APM vendors generally try to have as little side effects as possible, so they generally downgrade internal errors to warnings and just disable themselves.

If we want to just warn, we can catch the connection error internally and log that. If we want to throw, we could catch the connection error, put a special property on it and rethrow, then have our uncaughtException handler abort properly when it encounters an error with our special no-really-i-want-to-fail property.

[Qard]

Do we have a decision for what to do about this? This issue has been open for awhile...

[watson]

No, but thanks for bumping it. We should find a good solution. After this was opened another related issue has been created as well: #232

[watson]

The two options we have is either:

1. Throw on errors, but take care not to catch is out selfs as an uncaught exception
   1. One way to ensure we don't catch it our self is to add a property to the error that we can listen for
   2. Another way is to analyze the stack trace to see if we're part of it
2. Log errors to stderr instead of throwing

Only 1.ii handles the case where we throw by a mistake (e.g. a coding mistake or if someone passes us an unexpected argument that we try to access in a wrong way), so I'm leaning towards that solution. But I'm not sure how hard it would be to correctly detect these stack traces

[Qard]

How have the changes in API v2 impacted this? Is this still relevant, or can we close this?

[watson]

This is also an issue with v2 as far as I can see. The backoff algorithm should take care of that however when we implement that