"no EC2 instance role found" due to incorrect iamSecurityCredsPath

[eriksw]

Version of AWS SDK for Go?

# github.com/aws/aws-sdk-go-v2 v0.10.0

Version of Go (go version)?

go1.11.5

What issue did you see?

Attempting to run something equivalent to the README.md example:

panic: failed to describe table, EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make Client request
caused by:

goroutine 1 [running]:
...

Steps to reproduce

Reproduction requires an EC2 instance where EC2ws returns a 301 for a GET of http://169.254.169.254/latest/meta-data/iam/security-credentials (without trailing slash) with a with-trailing-slash Location in the 301. On most instances I have checked, except this one particular t2.xlarge in us-west-2, that GET returns the listing of profiles instead of the 301 redirect.

There is no proxy in front of EC2ws on this instance. This is in-the-wild EC2ws behavior that aws-sdk-go-v2 needs to handle.

$ curl -vv http://169.254.169.254/latest/meta-data/iam/security-credentials
*   Trying 169.254.169.254...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.47.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 301 Moved Permanently
< Location: http://169.254.169.254/latest/meta-data/iam/security-credentials/
< Content-Length: 0
< Connection: close
< Date: Thu, 08 Aug 2019 20:02:03 GMT
< Server: EC2ws
<
* Closing connection 0

Related papercuts

While debugging this, none of the obvious approaches to get logging from EC2RoleProvider worked:

• Setting the LogLevel on the Config returned by LoadDefaultAWSConfig did not work.

  cfg, err := external.LoadDefaultAWSConfig()
  ...
  cfg.LogLevel = aws.LogDebugWithHTTPBody
  

  This produced no output.

• Passing the log level to LoadDefaultAWSConfig did not work.

  cfg, err := external.LoadDefaultAWSConfig(aws.WithLogLevel(aws.LogDebugWithHTTPBody))
  

  This also produced no output.

At least one of the above approaches should have resulted in EC2RoleProvider logging its requests. I had to resort to editing a vendored aws-sdk-go-v2 (modifying aws.NewClient to always set the LogLevel, ignoring the config) in order to get debug logging from an EC2RoleProvider created by external.LoadDefaultAWSConfig—that's not reasonable.

[eriksw]

This is related to #198 but at some point the redirect stopped getting followed, making this a showstopper issue.

[CannibalVox]

This is massive, it effectively makes aws-sdk-go-v2 unusable for anyone on kubernetes.

[jasdel]

Thanks for reporting this issue. I created #355 to track the log option not being applied to the credential provider derived from LoadDefaultAWSConfig.

Reviewing the commit history, it looks like #315 would of disabled following the redirect behavior. I think a solution to this issue is to allow redirects for GET methods. EC2 Instance Metadata service is a special case where a redirect could be used. I don't think redirects are valid to follow for any other AWS service due to request signing. With that said the SDK's #198 needs to be fixed as well to prevent the redirect in the first case.

[deanshelton913]

I concur with @CannibalVox. This is failing for my app operating from within an EKS node running on fargate.