EC2 metadata resolution related exception thrown when running application locally

[nanachimi]

Describe the bug
I used spring-cloud-starter-aws (version 2.2.1.RELEASE) to send email using Amazon SES. I'm able to send email but I have the following error when starting the app. I'm not dealing with EC2 but it looks like the underline aws-java-sdk-ses (version 1.11.787) is trying to load EC2 resources. How can I get rid of this misconfiguration?

2020-05-22 17:41:09.647  WARN 47453 --- [           main] i.InstanceMetadataServiceResourceFetcher : Fail to retrieve token 

com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371) ~[aws-java-sdk-core-1.11.787.jar:na]
	at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
	at org.springframework.cloud.aws.context.annotation.OnAwsCloudEnvironmentCondition.matches(OnAwsCloudEnvironmentCondition.java:37) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
	at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:225) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser.processImports(ConfigurationClassParser.java:599) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser.access$800(ConfigurationClassParser.java:110) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.lambda$processGroupImports$1(ConfigurationClassParser.java:811) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) ~[na:na]
	at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:808) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:779) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:192) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:319) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:96) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
	at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:62) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
	at com.packnmoov.webappserver.WebAppServerRunner.main(WebAppServerRunner.java:9) ~[main/:na]
Caused by: java.net.ConnectException: No route to host (connect failed)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na]
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na]
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na]
	at java.base/java.net.Socket.connect(Socket.java:591) ~[na:na]
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1242) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1221) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1075) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1009) ~[na:na]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[aws-java-sdk-core-1.11.787.jar:na]
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[aws-java-sdk-core-1.11.787.jar:na]
	... 30 common frames omitted

Sample

@Configuration
public class AwsMailConfig {

    private final String awsAccessKey;
    private final String awsSecretKey;
...
    @Bean
    public AmazonSimpleEmailService amazonSimpleEmailService() {
        return AmazonSimpleEmailServiceClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider( new BasicAWSCredentials(awsAccessKey, awsSecretKey)))
                .withRegion(Regions.EU_CENTRAL_1)
                .build();
    }
...
}

[giova333]

This is how OnAwsCloudEnvironmentCondition works. In order to understand if the application is executed on AWS environment it tries to get instance metadata https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/instancedata-data-retrieval.html The issue is more related to the exception handling/logging as I see.
@maciejwalkowiak I can check this

[maciejwalkowiak]

Thanks @giova333. Indeed it's just a log message triggered during @ConditionalOnAwsCloudEnvironment evaluation. It happens only with new version of AWS SDK v1 (checked with 1.11.791). It would be nice if we find away to detect AWS env without having this exception logged. A workaround for today is adding to application.properties:

logging.level.com.amazonaws.util.EC2MetadataUtils=error

[giova333]

@maciejwalkowiak I don't think we will be able to manage that on our side.
EC2 metadata service is the recommended way to identify whether an application is running on AWS environment. Of course, we can call this service without using EC2MetadataUtils and this will solve the problem in this particular case but not everywhere.

Aws sdk uses DefaultAwsRegionProviderChain as default class for region discovery and
DefaultAWSCredentialsProviderChain for credentials lookup.

Both classes have a dependency on EC2MetadataUtils

DefaultAwsRegionProviderChain -> InstanceMetadataRegionProvider -> EC2MetadataUtils
DefaultAWSCredentialsProviderChain -> InstanceProfileCredentialsProvider -> EC2MetadataUtils

This means that whenever developers run our code outside ec2 instances and use some clients such as AmazonS3Client, AWSSecretsManagerClient ... without providing the region explicitly, sdk will try to load the region from env variables, system properties, aws config. If the region is not provided in any of the sources that I mentioned above, as the last resort, DefaultAwsRegionProviderChain
will try to load the region from ec2 metadata service using EC2MetadataUtils. And this is where the same issue with the logging will occur and the same flow for credentials lookup.
It is easy to reproduce, just call the perform s3 GetObjectRequest, catch the exception and you will see the same problem (checked with 1.11.791):

 try {
            AmazonS3Client.builder()
                    .build()
                    .getObject("bucketName", "key");
        } catch (Exception e) {  
        }

I would say that the issue has a global level and as for me the most appropriate way to manage it is to report it to aws sdk saying that the developers should be able to manage such cases themselves, handling the thrown exception, without having the original exception logged. That is what I am going to do.