Add support for Azure Blob Storage

[huydinhle]

Add support for Azure blob storage along side with GCS and S3

[tamalsaha]

I wonder if it makes sense to use something like https://github.com/google/go-cloud/tree/master/blob to reimplement the ObjectClient ?

[tomwilkie]

@huydinhle let me know if you need any pointers. The ObjectClient interface is the one to implement (as @tamalsaha mentioned), and you'll need to plumb it into the factory in NewObjectClient.

@tamalsaha could do! Its super important we have batching in the interface, which they don't seem to have. And we've need to tweak buffer sizes on the GCS client - not sure how to do that with go-cloud. I think if that library had existed when we started Cortex, we would have used it - but we've got almost all the implementations we'd need, so unless its super simple, I'm not sure we'd be that keen.

[bboreham]

Is anyone seriously looking at this, or it's just an idea?
What are the advantages, compared to other options?
(I now see this request came from Loki; there is a bit more background there but not much)
Should there be a related request to support indexing on Azure?

[henningzahn]

Is anyone working on this topic? Or are there shortcomings on Azure side that cannot be overcome? We would like to adopt Loki but cannot start because of the missing Azure support.

[santhoshparepu]

Is this going to be supported soon? We are Azure shop and without having the support for Azure blob there are challenges in adopting this.

[gouthamve]

You can use this for now: https://az.minio.io/ to provide an S3 interface for Azure Blob storage. Meanwhile you can also watch this issue for a more direct implementation: #1681

[khaines]

https://docs.min.io/docs/deploy-minio-on-kubernetes.html may also be useful if one is running in AKS , as the Azure marketplace setup uses at minimum 2 isolated VMs.

As I mentioned in #1235, I'm creating an AKS installation of cortex using CosmosDB (Cassandra API) for the index. I'll use MinIO for the chunk storage in order to check for any errors or other unexpected behaviour.