Skip to content

Not able to get IAM credentials with IMDSv1  #4896

Closed
@friedrichg

Description

@friedrichg

Describe the bug
Cortex is not able get IAM credentials with IMDSv1 auth

To Reproduce
Steps to reproduce the behavior:

  1. Ensure http://169.254.169.254/latest/api/token is not accessible for pods. (For example: using kube2iam)
  2. Start container with IAM role with Cortex e2cfc51
Click to expand!
$ /bin/cortex -blocks-storage.backend=s3 -blocks-storage.s3.bucket-name=bucket --target compactor --log.level=debug -blocks-storage.s3.endpoint=s3.dualstack.eu-west-1.amazonaws.com
level=info ts=2022-10-05T20:31:27.500222976Z caller=main.go:194 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=e2cfc51)"
level=info ts=2022-10-05T20:31:27.500504103Z caller=server.go:306 http=[::]:9009 grpc=[::]:9008 msg="server listening on addresses"
level=debug ts=2022-10-05T20:31:27.500922031Z caller=api.go:141 msg="api: registering route" methods=GET path=/config auth=false
level=debug ts=2022-10-05T20:31:27.501010322Z caller=api.go:141 msg="api: registering route" methods=GET path=/ auth=false
level=debug ts=2022-10-05T20:31:27.501028576Z caller=api.go:141 msg="api: registering route" methods=GET path=/debug/fgprof auth=false
level=debug ts=2022-10-05T20:31:27.501092732Z caller=api.go:141 msg="api: registering route" methods=GET path=/memberlist auth=false
level=debug ts=2022-10-05T20:31:27.501181221Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/compactor/ring auth=false
level=debug ts=2022-10-05T20:31:27.501209338Z caller=api.go:141 msg="api: registering route" methods=GET path=/services auth=false
level=debug ts=2022-10-05T20:31:27.501263744Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=memberlist-kv
level=info ts=2022-10-05T20:31:27.501381946Z caller=module_service.go:64 msg=initialising module=server
level=debug ts=2022-10-05T20:31:27.501487025Z caller=module_service.go:54 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=info ts=2022-10-05T20:31:27.501624576Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=debug ts=2022-10-05T20:31:27.501672565Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=server
level=info ts=2022-10-05T20:31:27.501683884Z caller=module_service.go:64 msg=initialising module=compactor
level=info ts=2022-10-05T20:31:27.501896465Z caller=blocks_cleaner.go:144 component=cleaner msg="started blocks cleanup and maintenance"
level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"
level=info ts=2022-10-05T20:31:27.540381948Z caller=compactor.go:621 component=compactor msg="discovering users from bucket"
level=info ts=2022-10-05T20:31:27.540545681Z caller=cortex.go:422 msg="Cortex started"
  1. Get access denied from s3

level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"

Expected behavior
it should continue without problem

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: jsonnet

Additional Context
It works in the previous commit f550642

Previous

// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
token, _ := fetchIMDSToken(client, endpoint)
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
u, err := getIAMRoleURL(endpoint)
if err != nil {
return ec2RoleCredRespBody{}, err
}

Current

// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
token, err := fetchIMDSToken(client, endpoint)
if err != nil {
return ec2RoleCredRespBody{}, err
}
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
u, err := getIAMRoleURL(endpoint)
if err != nil {
return ec2RoleCredRespBody{}, err
}

This bugfix minio/minio-go#1682 is most likely the culprit

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions