CRT Client upload causes increase in memory consumption #4094
Description
Describe the bug
When uploading input steams using the CRT s3 client, it seems that Kubernetes memory restrictions are not respected. its increasing more and more
our use case is we running consumer where it will upload the input stream to s3 bucket .
when we try to run execute the parallel requests around 50 request with each steam size in GB's . the Pod memory increasing .
even after job done its not coming down . eventually getting
software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure
is CRT client holding memory even after upload done ?
Expected Behavior
in finally bock iam closing input stream ,executor service , awsclient.
we are creating aws client for every request as authentication details different for every user
Current Behavior
"software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.\n\tat software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)\n\tat software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)\n\tat software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:127)\n\tat software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)\n\tat software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)\nWrapped by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.\n\tat software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)\n\tat java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)\n\tat java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)\n\tat software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)\n\tat software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n\t... 4 frames truncated\n\t... 1 common frames omitted\
Reproduction Steps
S3AsyncClient amazonS3Client = S3AsyncClient.crtBuilder()
.credentialsProvider(StaticCredentialsProvider
.create(AwsSessionCredentials
.create("AccessKey", "SecretKey", "sessionToken"())))
.region(Region.US_EAST_1)
.httpConfiguration(S3CrtHttpConfiguration.builder().proxyConfiguration(S3CrtProxyConfiguration.builder()
.host("").port()).build()).build())
.build();
PutObjectRequest putObjectRequest = PutObjectRequest.builder().bucket().key(filePath).build();
CompletableFuture putObjectResponse =
amazonS3Client.putObject(putObjectRequest,
AsyncRequestBody.fromInputStream(inputStream, fileSize, Executors.newFixedThreadPool( 40)));
putObjectResponse.join();
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.20.61
JDK version used
1.8
Operating System and version
running kubernets