Description
Describe the bug
Cortex is not able get IAM credentials with IMDSv1 auth
To Reproduce
Steps to reproduce the behavior:
- Ensure http://169.254.169.254/latest/api/token is not accessible for pods. (For example: using kube2iam)
- Start container with IAM role with Cortex e2cfc51
Click to expand!
$ /bin/cortex -blocks-storage.backend=s3 -blocks-storage.s3.bucket-name=bucket --target compactor --log.level=debug -blocks-storage.s3.endpoint=s3.dualstack.eu-west-1.amazonaws.com
level=info ts=2022-10-05T20:31:27.500222976Z caller=main.go:194 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=e2cfc51)"
level=info ts=2022-10-05T20:31:27.500504103Z caller=server.go:306 http=[::]:9009 grpc=[::]:9008 msg="server listening on addresses"
level=debug ts=2022-10-05T20:31:27.500922031Z caller=api.go:141 msg="api: registering route" methods=GET path=/config auth=false
level=debug ts=2022-10-05T20:31:27.501010322Z caller=api.go:141 msg="api: registering route" methods=GET path=/ auth=false
level=debug ts=2022-10-05T20:31:27.501028576Z caller=api.go:141 msg="api: registering route" methods=GET path=/debug/fgprof auth=false
level=debug ts=2022-10-05T20:31:27.501092732Z caller=api.go:141 msg="api: registering route" methods=GET path=/memberlist auth=false
level=debug ts=2022-10-05T20:31:27.501181221Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/compactor/ring auth=false
level=debug ts=2022-10-05T20:31:27.501209338Z caller=api.go:141 msg="api: registering route" methods=GET path=/services auth=false
level=debug ts=2022-10-05T20:31:27.501263744Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=memberlist-kv
level=info ts=2022-10-05T20:31:27.501381946Z caller=module_service.go:64 msg=initialising module=server
level=debug ts=2022-10-05T20:31:27.501487025Z caller=module_service.go:54 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=info ts=2022-10-05T20:31:27.501624576Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=debug ts=2022-10-05T20:31:27.501672565Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=server
level=info ts=2022-10-05T20:31:27.501683884Z caller=module_service.go:64 msg=initialising module=compactor
level=info ts=2022-10-05T20:31:27.501896465Z caller=blocks_cleaner.go:144 component=cleaner msg="started blocks cleanup and maintenance"
level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"
level=info ts=2022-10-05T20:31:27.540381948Z caller=compactor.go:621 component=compactor msg="discovering users from bucket"
level=info ts=2022-10-05T20:31:27.540545681Z caller=cortex.go:422 msg="Cortex started"
- Get access denied from s3
level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"
Expected behavior
it should continue without problem
Environment:
- Infrastructure: Kubernetes
- Deployment tool: jsonnet
Additional Context
It works in the previous commit f550642
Previous
cortex/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go
Lines 291 to 298 in f550642
Current
cortex/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go
Lines 291 to 301 in e2cfc51
This bugfix minio/minio-go#1682 is most likely the culprit