Update fsspec.py to respect s3.signer.uri property

[c-thiel]

Fixes #740

[c-thiel]

@amogh-jahagirdar @HonahX @Fokko any thoughts about this PR? Would this be OK to incorporate?
It does not change any existing behavior, just respects an optional config option. spark honors this option - the constant is defined here:
https://iceberg.apache.org/javadoc/1.4.1/constant-values.html

[Fokko]

@c-thiel Since it is in Java, I think it is fair to add it here. Two things:

• Can you mention it in the docs as well?
• Should we introduce a constant?

[c-thiel]

@Fokko thanks a lot for your feedback - I added docs and the constant.
The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it.

I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds.

Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded

iceberg-python/pyiceberg/catalog/rest.py

Line 501 in 42afc43

session.headers["X-Iceberg-Access-Delegation"] = "vended-credentials"

, even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with:

    "rest.sigv4-enabled": "true",
    "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO",

it works like a charm.

[Fokko]

This looks good @c-thiel thanks for working on this 👍

[Fokko]

@c-thiel The idea is to support both. The problem is that with Arrow it is hard to inject in the signing process, since most of the code is pushed down to the C level, and there are no hooks to use a custom signer. Support for remote signing is something that we definitely want to continue to support.

[guitcastro]

@Fokko thanks a lot for your feedback - I added docs and the constant. The constant is a very good idea - I hope we will be able to use remote signing with FileIO as well eventually. Right now only the fsspec impl. respects it.

I saw that tabular.io actually provides explicit S3 credentials (on top of remote signing), presumably via AWS STS, if "vended-credentials" are requested (https://github.com/apache/iceberg/blob/b3c25fb7608934d975a054b353823ca001ca3742/open-api/rest-catalog-open-api.yaml#L1495). This is something that can only ever work for AWS S3 and is noticeably slower than using remote signing. As remote signing works also with on-prem deployments, I really hope this is going to become the default for all clients and not vended-credentials. tabular does this only for pyiceberg. Spark requests remote-signing so there is no need to go the extra mile and generate S3 creds.

Right now unfortunately in pyiceberg, "vended-credentials" is hardcoded

iceberg-python/pyiceberg/catalog/rest.py

Line 501 in 42afc43

session.headers["X-Iceberg-Access-Delegation"] = "vended-credentials"

, even though "remote-signing" is actually supported via fsspec. If the server decides to just ignore what the client requests and push remote signing anyway together with:

    "rest.sigv4-enabled": "true",
    "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO",

it works like a charm.

Unfortunately, for nessie, when using X-Iceberg-Access-Delegation: vended-credentials does not work. The endpoint does not return the s3.signer.uri. When the head is set to remote-signing it does return the correct value.