Support caching bucket for cleaner #6778
Conversation
| @@ -245,6 +245,47 @@ func CreateCachingBucket(chunksConfig ChunksCacheConfig, metadataConfig Metadata | |||
| return storecache.NewCachingBucket(bkt, cfg, logger, reg) | |||
| } | |||
|
|
|||
| func CreateCachingBucketForCompactor(metadataConfig MetadataCacheConfig, cleaner bool, bkt objstore.InstrumentedBucket, logger log.Logger, reg prometheus.Registerer) (objstore.InstrumentedBucket, error) { | |||
There was a problem hiding this comment.
Can this and CreateCachingBucket have a inner shared logic? They seems very similar, no?
There was a problem hiding this comment.
Right they are similar. I don't see a good way to share the same function for the two. The main difference is thosecfg.CacheXXX calls can be different for read and compaction path
|
Should we use another config? I dont like creating a new config, but if this causes issues on cleaner we would need to turn off cache in both compactors. It would be nice to have a separate config while we are experiment with it |
Yeah I can create a new flag to enable this for cleaner separately. Good call out. |
|
@alexqyle any concerns with this caches being introduced in cleaner? |
I think it is ok for cleaner to only cache tenant iter and metadata files |
Signed-off-by: yeya24 <[email protected]>
yeya24
force-pushed
the
caching-bucket-cleaner
branch
from
0ababce to
b843a94
Compare
Signed-off-by: yeya24 <[email protected]>
yeya24
force-pushed
the
caching-bucket-cleaner
branch
from
b843a94 to
898582e
Compare
Signed-off-by: Ben Ye <[email protected]>
yeya24
force-pushed
the
caching-bucket-cleaner
branch
from
a8046ff to
01ecacb
Compare
What this PR does:
There was an attempt before to support metadata cache for compactor in #5682.
It only supports compactor and ignores cleaner as it is intended for cleaner to be source of truth of the bucket.
This makes sense but Cleaner also does a lot of calls to object storage so it would be nice to explore if we can use metadata caching bucket for Cleaner.
This PR tries to cache certain objstore operations for cleaner:
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]