Skip to content

Percentage Based Sharding for Compactors #6738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eeldaly opened this issue May 12, 2025 · 1 comment
Open

Percentage Based Sharding for Compactors #6738

eeldaly opened this issue May 12, 2025 · 1 comment

Comments

@eeldaly
Copy link
Contributor

eeldaly commented May 12, 2025

Is your feature request related to a problem? Please describe.
Compactors shard size has to be increased manually for a tenant. This does not work well when scale up of compactors occurs since we are limited by a fixed value.

Describe the solution you'd like
Identical to: #6652 for compactors. Adding support for 0.2 = 20% of compactors for that tenant

@dsabsay
Copy link
Contributor

dsabsay commented May 17, 2025

I understand this makes scaling easier.

For some users, they move tenants between multiple clusters. I could see percentage-based sharding making it slightly more difficult to properly size the clusters. For example, you move a small tenant in a large cluster to a smaller cluster; the tenant was spread on lots of compactors before and now they have less. If you want predictable performance, you'd need to move tenants to only similarly sized clusters. Maybe this is an acceptable challenge; I'm just thinking.

What strategies do you see for properly sizing clusters during tenant moves?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants