Possible Memory Leak - Connection Reaper

[mtamassia]

We have an application that creates and destroys EC2, S3, and several other SDK clients.

We build clients within a thread and close() the clients within the thread. Example:

Ec2Client ec2Client = Ec2Client.builder().region(Region.of(currentRegion)).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

// DO SOMETHING

ec2Client.close();

Memory consumption grows over time and here is a link to the memory heap dump, which is self explicatory: https://heaphero.io/my-heap-report.jsp?p=YXJjaGl2ZWQvMjAyMC8wMy8yLy0tbWVtb3J5LnppcC0xOS01NC0zMi5qc29uLS0=

Expected Behavior

client close() should not be leaving resources behind

Current Behavior

Threads and their associated data should be removed upon calling a client close() method

Your Environment

• OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
• SDK 2.10.77

[dagnir]

Thanks for the report. Looks like the connection managers are not being removed from the map during client close shutdown during client close.

Looks like I spoke too soon; looking over the code again, everything seems in order. And doing a simple test continuously creating a client, making a request, and closing it keeps the reaper map size at 0. Do you have a repro case that we can run?

[mtamassia]

Thank you for the quick response @dagnir

I have reviewed our code and found a couple instances where close() was not being properly called, however after the changes, we still see the memory buildup as you can see on the new report below. BTW, this time I forced a GC, just for sanity, before doing the heap dump.

https://heaphero.io/my-heap-report.jsp?p=YXJjaGl2ZWQvMjAyMC8wMy81Ly0tbWVtb3J5My5kdW1wLmd6LTE2LTQ0LTMwLmpzb24tLQ==

Not sure if it matters but here are the SDK clients we are using and how we are instantiating each of them.

SecretsManagerClient smClient = SecretsManagerClient.builder().credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

Ec2Client ec2Client = Ec2Client.builder().region(Region.of(this.getRegion())).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

PricingClient pClient = PricingClient.builder().region(Region.US_EAST_1).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

RdsClient rdsClient = RdsClient.builder().region(Region.of(this.getRegion())).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

CloudTrailClient client = CloudTrailClient.builder().credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).overrideConfiguration(ClientOverrideConfiguration.builder().retryPolicy(RetryPolicy.builder().throttlingBackoffStrategy(BackoffStrategy.defaultStrategy()).build()).build()).build();

ElasticLoadBalancingV2Client client = ElasticLoadBalancingV2Client.builder().region(Region.of(this.getRegion())).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

DirectConnectClient client = DirectConnectClient.builder().region(Region.of(currentRegion)).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

S3Client client = S3Client.builder().region(Region.of(this.getRegion())).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

EfsClient client = EfsClient.builder().region(Region.of(currentRegion)).credentialsProvider(InstanceProfileCredentialsProvider.builder().build()).build();

[zoewangg]

I tried creating multiple sdk clients, making api calls, and then closing the clients but was not able to reproduce the issue. (verified that IdleConnectionReaper#connectionManagers was empty in the end)

Could you provide a complete repro case?

[debora-ito]

We were not able to reproduce it, closing this due to no response. Please feel free to reopen if you are still experiencing this issue and can provide us a complete repro case.

[gdhananjay]

@debora-ito , @dagnir , @zoewangg, @mtamassia

I am facing same issue , I also suspect that there is possible memory leak in Connection Repeater.
My application used to run pretty well for 1 week and then it was crashed because of Out Of memory.
I attached Jprofiler and it seems connectionrepeater was not freeing up.
Please look at below screenshot:

further drilling this heavy object i saw thousands of below instances,

And here is path to GC root to those thousands of objects,

My application is simple, it creates thread => create credential provide as shown in below code => Collect some data from cloudwatch using this credential provider => and then thread dies.

Code to create credential provider :

public static AwsCredentialsProvider getRoleCredentialsProvider(String _roleArn, String _roleSessionName,
			String _externalId, String _region) {

		AssumeRoleRequest assumeRoleRequest = null;

		if (_externalId == null || _externalId.isEmpty()) {
			assumeRoleRequest = AssumeRoleRequest.builder().roleArn(_roleArn).roleSessionName(_roleSessionName)
					.durationSeconds(900).build();
		} else {
			assumeRoleRequest = AssumeRoleRequest.builder().roleArn(_roleArn).externalId(_externalId)
					.roleSessionName(_roleSessionName).durationSeconds(900).build();
		}

		StsClient stsClient = null;

		// AWS recommends using Regional AWS STS endpoints hence region as input.
		if (_region == null || _region.isEmpty()) {
			stsClient = StsClient.builder().region(Region.US_EAST_1).build();
		} else {
			stsClient = StsClient.builder().region(Region.of(_region)).build();
		}

		return StsAssumeRoleCredentialsProvider.builder().stsClient(stsClient).refreshRequest(assumeRoleRequest)
				.asyncCredentialUpdateEnabled(true).build();
	}

Code in finally block of my application to close this provider:

			// Close StsAssumeRoleCredentialsProvider.
			if(credentialProvider != null && credentialProvider instanceof StsAssumeRoleCredentialsProvider) {
				((StsAssumeRoleCredentialsProvider)credentialProvider).close();
			}

I can easily reproduce this issue, As jprofiler is running live it clearly show memory ++ because of these objects.

I am using below SDK version

		<dependency>
  			<groupId>software.amazon.awssdk</groupId>
 			 <artifactId>sts</artifactId>
 			 <version>2.5.40</version>
		</dependency> 

Please let me know if you need any additional info.

[zoewangg]

Hi @Dhananjay-Ghongane closing the credential provider itself does not close the stsClient.

aws-sdk-java-v2/services/sts/src/main/java/software/amazon/awssdk/services/sts/auth/StsCredentialsProvider.java

Line 83 in 3d2e2e5

sessionCache.close();

So you need to close the stsClient too. The reason that the stsClient is not closed automatically is because it's provided by customers.