Support dataset upload to S3

[normanrz]

Detailed Description

Webknossos should be able to deposit datasets, that are uploaded by users, to S3.

Context

We want to switch to S3-compatible object storage for primary storage of wk datasets.

[fm3]

I’m going to have a look at the dataset upload protocol, switching it to dataset id, and new behavior for LayersToLink in #8844 – @frcroth could you have a look at the actual S3 client parts of this issue?

How can wk forward the received upload chunks to S3? Is there a partial write API or does wk have to cache the files locally and then write them? Is there a way to retry this so nothing gets lost?

[normanrz]

I just had a brief conversation about this: https://chatgpt.com/share/6899e1bc-fbd8-800a-ace1-6bcf21742d68
tl;dr

• each part needs to be ≥5MB
• up to 10k parts
• each part can be resumed
• no need to specify number or size of parts upfront
• returned etags per part need to be stored and sent with finalize call

[frcroth]

Can I get credentials or should I try to work with e.g. minio?

[frcroth]

• Currently, the frontend already sends uploads in chunks of size 10MB. Since the multi part upload allows up to 10,000 chunks, we could support files of up to 100GB. We could of course also change the chunk size.
• We need to store the results of each chunk upload (etag), which could be done with the redis, which is currently already used for keeping track of the file size and the files. So much of this (handleUploadChunk) could probably be reused, just with a different upload target
  • Start a multipart upload when we see a file for the first time
  • Upload chunks via UploadPartRequest
  • Track etags in Redis -> CompleteMultipartUpload after finish