feat: concurrent storage put #1304
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
size-limit report 📦
|
Codecov Report
@@ Coverage Diff @@
## main #1304 +/- ##
=======================================
Coverage 68.51% 68.51%
=======================================
Files 129 129
Lines 4248 4248
Branches 1138 1138
=======================================
Hits 2910 2910
Misses 1333 1333
Partials 5 5 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
kolesnikovae
force-pushed
the
feat/cuncurrent_writes
branch
from
3b4b0c2 to
2bdaba9
Compare
kolesnikovae
force-pushed
the
feat/cuncurrent_writes
branch
from
2bdaba9 to
f80f87d
Compare
|
@petethepig I have updated the PR so that it won't affect existing deployments: storage queue will have one worker by default, thus effectively doing writes synchronously. Please, give it a look |
petethepig
approved these changes
Aug 30, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
storage.Putto work concurrentlyGetOrCreateis thread-safeBuild tags for long ad-hoc tests– It may make sense to introduce a build tag for long tests (ad-hoc). As for now I adjusted the stress test so that it would run <1m.Check if RW mutexes are good in out case– Experiments don't show improvements (See sync: RWMutex scales poorly with CPU count golang/go#17973)The change introduces two configuration options for the storage queue:
storage-queue-workersstorage-queue-sizeOn migration, especially if the deployment is large, storage throughput will potentially increase significantly, as there will be less dropped profiles, which poses some risks. We can set default parameters so that effectively writes happen serially (just one worker) to workaround this.We decided to disable concurrent writes by default, so that the change does not affect existing deployments on update. However, users still may tweak parameters if required.
/cc @petethepig