Hostname to IP Address Resolution in http-request-to-url can Overwhelm Service Maps

[astorm]

We've received reports that the "Hostname to final IP Address Resolution" for outgoing span URLs behavior of the agent can overwhelm service maps when a service url has a dynamic IP address.

In one specific case, requests made to the Amazon S3 API can result in over 100 different IP addresses being reported, resulting in a service map that looks something like this

However, this is not a problem restricted to the AWS SDK. This will happen for any service URL that doesn't have a fixed IP address.

We should reconsider whether resolving the hostname to its final IP destination is the right default behavior for instrumentation on outgoing HTTP spans.

Technical Details

This behavior stems from how we're setting the URL portion of the span's HTTP Context.

apm-agent-nodejs/lib/instrumentation/http-shared.js

Lines 212 to 216 in 8d4f10d

	span.setHttpContext({
	method: req.method,
	status_code: statusCode,
	url
	})

The url variable is set by the http-request-to-url module

apm-agent-nodejs/lib/instrumentation/http-shared.js

Line 6 in 8d4f10d

var httpRequestToUrl = require('http-request-to-url')

apm-agent-nodejs/lib/instrumentation/http-shared.js

Line 195 in 8d4f10d

httpRequestToUrl(req).then(_url => {

The http-request-to-url module waits for the socket's connect event, and extract the remoteAddress and remotePort from the socket object, and uses these to create a URL.