Skip to content

Virtual-host-style S3 requests fail when using a single endpoint for all services (i.e., AWS_ENDPOINT_URL) #7136

@joe4dev

Description

@joe4dev

Checkboxes for prior research

Describe the bug

S3 API calls using virtual-host-style requests (default in most SDKs except for botocore) fail when using a single endpoint URL (e.g., AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566 or AWS_ENDPOINT_URL=http://localhost:4566). This breaks native LocalStack compatibility of many AWS tools such as the AWS Toolkit (aws/aws-toolkit-vscode#2007) and AWS CDK (related to aws/aws-cdk#21014).

AWS endpoint standard: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.830.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.14.0

Reproduction Steps

  1. Install and start LocalStack: https://docs.localstack.cloud/getting-started/installation/

  2. npm install @aws-sdk/client-s3@3.830.0

  3. Run node test.js using the following script:

    import { S3Client, CreateBucketCommand, PutObjectCommand} from "@aws-sdk/client-s3";
    
    const BUCKET_NAME = "my-unique-localstack-bucket-name";
    const OBJECT_KEY = "HelloWorld.txt";
    const CONTENT = "HelloWorld!";
    
    const s3 = new S3Client({
        region: "us-east-1",
        endpoint: "http://localhost:4566",
        credentials: {
          accessKeyId: "test",
          secretAccessKey: "test",
        },
        maxAttempts: 1,
    });
    
    const createBucketResponse = await s3.send(new CreateBucketCommand({ Bucket: BUCKET_NAME }));
    const putObjectResponse = await s3.send(new PutObjectCommand({
        Bucket: BUCKET_NAME,
        Key: OBJECT_KEY,
        Body: CONTENT,
        ContentType: "text/plain",
    }));

Link reproducer test suite (including botocore comparison): https://github.com/joe4dev/s3-endpoint-url-testing

Observed Behavior

The JavaScript SDK raises an S3ServiceException [InternalError]: exception while calling s3 with unknown operation: Unable to find operation for request to service s3: PUT
The root cause of this confusing ProtocolParserError is that emulators such as LocalStack cannot reliably detect virtual host-styled S3 requests without the .s3 prefix when using a single global endpoint. Therefore, the PutBucket (or any virtual-host-style request) fails with an internal server error (i.e., it gets interpreted as an XML-style CreateBucket request).

/Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:388
  const response = new exceptionCtor({
                   ^

S3ServiceException [InternalError]: exception while calling s3 with unknown operation: Unable to find operation for request to service s3: PUT /
    at throwDefaultError (/Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:388:20)
    at /Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:397:5
    at de_CommandError (/Users/joe/Downloads/s3_test/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4953:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/joe/Downloads/s3_test/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
    at async /Users/joe/Downloads/s3_test/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:137:14
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 500,
    requestId: 'a142a92d-7db0-497f-bf8f-11011f69899f',
    extendedRequestId: 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'InternalError',
  RequestId: 'a142a92d-7db0-497f-bf8f-11011f69899f'
}

Node.js v22.14.0

Expected Behavior

S3 requests should work properly when using a single endpoint (e.g., AWS_ENDPOINT_URL=http://localhost:4566), following the AWS standard for endpoint URLs: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

The S3 client in the JavaScript SDK should detect that a single endpoint is incompatible with virtual-host-style requests and switch to path-style.
Testing virtual-host-style should still be possible with emulators when using a compatible service-specific endpoint URL such as AWS_ENDPOINT_URL_S3=http://s3.localhost.localstack.cloud:4566

Possible Solution

Link PR: #7137

The Python library botocore could serve as a reference implementation, as demonstrated in the PR#2785 Endpoint resolution v2.0.

The relevant change happens in compute_endpoint_resolver_builtin_defaults where this conditional adjusts force_path_style if a custom client_endpoint_url is used and the endpoint does not match the S3 Accelerate endpoint scheme (only applicable to amazonaws.com endpoints). It forces path-style S3 requests unless the S3 addressing_style configuration explicitly demands virtual-style:

elif client_endpoint_url is not None and not is_s3_accelerate_url(
            client_endpoint_url
        ):
    force_path_style = s3_config.get('addressing_style') != 'virtual'

This change enables S3 APIs to work with a global endpoint (e.g., AWS_ENDPOINT_URL=http://localhost.localstack.cloud) using path-style S3 requests while still allowing to force virtual-style S3 requests if demanded (e.g., when manually setting AWS_ENDPOINT_URL_S3=http://s3.localhost.localstack.cloud:4566 and s3 > addressing_style=virtual).

The solution from botocore is not directly applicable here because the JavaScript SDK offers a boolean-named configuration forcePathStyle=true|false (auto by default) instead of the triple-choice option addressing_style=auto|virtual|path. We should allow for virtual-style-requests, either by interpreting forcePathStyle=false (if auto reliably detect undefined) as addressing_style_=virtual or by detecting compatible subdomains starting with s3. (e.g., http://s3.localhost.localstack.cloud:4566).

Additional Information/Context

This issue may be related to Smithy code generation and might be solvable using the Smithy rules engine
Related example from botocore for S3 endpoint ruleset: https://github.com/boto/botocore/blob/d1fd992119b5df4f4d2169e2383ab99288466e8b/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json

Disclaimer: I work for LocalStack

Activity

added
bugThis issue is a bug.
needs-triageThis issue or PR still needs to be triaged.
on Jun 18, 2025
self-assigned this
on Jun 18, 2025
kuhe

kuhe commented on Jun 25, 2025

@kuhe
Contributor

Do you currently instruct LocalStack users to set forcePathStyle on the S3Client instance constructor under applicable conditions?

joe4dev

joe4dev commented on Jun 26, 2025

@joe4dev
Author

Do you currently instruct LocalStack users to set forcePathStyle on the S3Client instance constructor under applicable conditions?

Yes, we do for the AWS JavaScript SDK integration: https://docs.localstack.cloud/aws/integrations/aws-sdks/javascript/

  1. We recommend the S3 service-specific endpoint AWS_ENDPOINT_URL_S3=http://s3.localhost.localstack.cloud:4566
  2. We recommend forcePathStyle: true as a fallback (e.g., if DNS rebind protection blocks resolving localhost.localstack.cloud to 127.0.0.1)

Nevertheless, customers expect this to work out of the box using a single AWS_ENDPOINT_URL and for indirect usage through tools such as the AWS Toolkit for VSCode, this is not configurable.

Edit (2025-07-15): Updated SDK documentation link pointing to our new documentation.

added
feature-requestNew feature or enhancement. May require GitHub community feedback.
service-apiThis issue is due to a problem in a service API, not the SDK implementation.
p2This is a standard priority issue
and removed
bugThis issue is a bug.
needs-triageThis issue or PR still needs to be triaged.
on Jul 16, 2025
aBurmeseDev

aBurmeseDev commented on Jul 16, 2025

@aBurmeseDev
Contributor

Checking in here. This's the similar behavior that you requested in another issue and we responded.

After discussing with the team, I confirm that default endpoint resolution is governed by service models and is generally consistent across all SDKs. The SDKs follow the Endpoints 2.0 specification for endpoint resolution, which works from an endpoint ruleset provided in the service models. The hostPrefix/path-style or virtual-host style behavior is defined at the model level, and most SDKs accept this as their default behavior.
Python (boto3) seems to be an exception here, with some customization in its implementation. However, it's important to note that the service model remains the ground truth for endpoint resolution behavior across all SDKs.
The JS SDK, like most other SDKs, adheres to this ruleset, including the prefix resolution as specified in the model. While we acknowledge the difference in Python's behavior, we decided not to add this specific handling to the JS SDK at this time to keep it consistent across SDKs. I've reached out to the service team for further insight, though any changes to the model would likely be breaking. (ref: P265329427)

Here's the same model used across SDKs for reference:
Smithy model: https://github.com/aws/api-models-aws/blob/896693ac2e1c0efd9d03488109601e0c66ba7d35/models/sfn/service/2016-11-23/sfn-2016-11-23.json#L6204
Java v2: https://github.com/aws/aws-sdk-java-v2/blob/master/services/sfn/src/main/resources/codegen-resources/service-2.json#L511
botocore: https://github.com/boto/botocore/blob/develop/botocore/data/stepfunctions/2016-11-23/service-2.json#L511
Go v2: https://github.com/aws/aws-sdk-go-v2/blob/e9a97206aff1d25c17ee29c183c87cdf09c168f0/service/sfn/api_op_StartSyncExecution.go#L264

TLDR; This behavior is specified at the model level (S3's model file here) and the SDKs accept this as the default "out-of-the-box" behavior as dictated by the service. The change in behavior will potentially break current users.

github-actions

github-actions commented on Jul 16, 2025

@github-actions

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

github-actions

github-actions commented on Jul 31, 2025

@github-actions

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

locked as resolved and limited conversation to collaborators on Jul 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

cross-sdkfeature-requestNew feature or enhancement. May require GitHub community feedback.p2This is a standard priority issueservice-apiThis issue is due to a problem in a service API, not the SDK implementation.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @kuhe@joe4dev@aBurmeseDev@siddsriv

      Issue actions

        Virtual-host-style S3 requests fail when using a single endpoint for all services (i.e., AWS_ENDPOINT_URL) · Issue #7136 · aws/aws-sdk-js-v3