Design Proposal: Use multiple buckets for s3 chunk storage for manual cleanup

[thorfour]

Abstract

Add a feature that uses multiple buckets to store spans of data over a given time frame.
Modify table manager to support issuing s3.DeleteBucket requests during cleanup when a given bucket has aged out.

Reason(s)

Presently S3 chunk cleanup is expected to be performed by the S3 storage providers retention policy. Some S3 implementations (ex: DigitalOcean Spaces) do not provide retention policies. This design is a way to keep cortex data more self-contained (not requiring external dependencies to manage the life-cycle of data). It is meant to address issue #1591

Goals

• Self-contained data life-cycle for cortex chunks.
• Efficient delete processing (fewest number of API requests)
• Supporting custom retention periods

Implementation

A new set of flags would be added:
--s3.retain-by-bucket-period that sets the time duration of each bucket
--s3.bucket-prefix that sets the prefix of the buckets that will be created

Ingester will upload chunks to S3 buckets based on the Through timestamp of the chunk, into the given bucket that the chunk would fall into. So it's possible that a chunk that spans bucket boundaries would still be retained by being placed into a newer bucket.

Querier will select the bucket to retrieve chunks from based on the Through timestamp that chunk would place it.

Table-Manager will maintain it's periodic calls to DeleteChunksBefore which for s3 bucket clients will call the s3.DeleteBucket calls to buckets that exceed the retention policy set in table manger.

The complex part of this proposal is the creation life-cycle of the buckets. Since there are multiple ingesters one of them would have to be the "leader" that creates buckets.

Instead I believe table-manger should be responsible for the creation of the buckets as well. Maintaining a rolling window of buckets to ensure there's always an available bucket to write to. Also often with s3 providers bucket names are global. So on creation of a new bucket it would likely be best to be of the format {{bucket_prefix}}-{{uuid}}-{{through_timestamp}}

[gouthamve]

Hrm, this is slightly different from how we handle the periodic tables for chunks.

We have {{table_prefix}}_n where n is similar to the timestamp, but it more the nth period for the timestamp. See: https://github.com/cortexproject/cortex/blob/master/pkg/chunk/schema_config.go#L356-L371

We can do {{bucket_prefix}}-{{uuid}}-n or {{bucket_prefix}}-n-{{uuid}} but now it's hard to determine the buckets statically in the querier (we do that for the periodic tables). This means the schema code should now periodically reload the available buckets through the API.

Further, what if there exist 2 buckets of the form {{bucket_prefix}}-n-{{uuid}} in the account? Do we error out in that case?

Instead can we have a folder per week and parallelise the deletes? I double checked now and object deletion is a free operation in GCS too, I was mistaken before. I think making a folder per week would be operationally simpler.

[thorfour]

@gouthamve Thanks for the reply. I did do some digging on aws/gcs and also noticed that deletes were free. However I'm unable to find any API that supports deletes at the folder level. Part of my goal with this implementation was to keep the number of deletes to a minimum, in an attempt to clean up quickly, and prevent api rate-limiting.

As for bucket naming could we use a single uuid and pass that in as a flag to keep the lookups static? I'd rather not have to change querier to list out buckets. So have a user create a uuid, pass it to cortex as a flag, and cortex uses that when creating buckets? It'll reduce the possibility of collisions, but if a collision should occur I think we'd have to error out. I'm not sure of a way around that.

So if we were to go the folder route we would still have to delete all objects individually as far as I can tell, and in that case there's no real advantage to placing them into a folder and instead should just iterate over them and delete them based on chunk timestamps. I initially implemented this when I was testing some of these ideas and it works fine, but it does have a rate limiting problem. Where you either have to coordinate the rate at which you can delete between ingesters, and queriers, or you have to set the delete rate very low.

So I think we either deal with buckets that have the potential for collisions, or we deal with rate limited parallel deletes running in the background. What do you think?

[thorfour]

I personally lean towards keeping the implementation simple and just having table manager walk the s3 chunks and perform deletes in the background.

[gouthamve]

I personally lean towards keeping the implementation simple and just having table manager walk the s3 chunks and perform deletes in the background.

I share the same inclination.

[thorfour]

The rate at which you'd have to delete objects might be a limiting factor. However using the DeleteObjects API you can delete 1k objects per request.

[thorfour]

I think it's best to close this in favor of using s3's retention policies and not attempt to manage the lifecycle of objects in s3/spaces at this time.