Closed
Description
During the period from midnight to midnight + max chunk age, newly matured chunks require 2x the number of index entries to be written to the period tables because their contents span two day buckets. Provisioned capacity must be sized to cope with this nightly spike or the series flush queues start to back up:
However, doing so means that capacity is over-provisioned for the noon to midnight period. The table manager can be enhanced to step the capacity up automatically before midnight, and back down again after noon.
Activity
tomwilkie commentedon Jun 19, 2017
Dupe of #318? @leth added metrics on per-tables used capacity in #460, which is the first step in dynamically adjusting provisioning.
leth commentedon Jun 19, 2017
The scope of this could be a bit smaller; just focused on the midnight rollover, rather than general capacity?
tomwilkie commentedon Jun 19, 2017
Thats up to the person who implements it I guess! I personally don't see how just focusing on the time around midnight make is much easier.
awh commentedon Jun 19, 2017
I suspect the solution for #318 will be far more complex than adding
if time > midnight & time < noon activeCapacity *= 2to the table manager...bboreham commentedon Feb 5, 2018
Do we still need this issue now autoscaling is supported?
Edit: AWS autoscaling does nothing at all when you're not using the table, so we currently still have the problem that the table is over-provisioned from when it is created until when we really need it.
bboreham commentedon Sep 6, 2018
I think the new metrics-based scaling in table-manager is coping ok with this, so closing this issue.