Use FossilDBPutBuffer for more efficient saving of multiple fossil keys

[fm3]

Measured duplicate on a volume annotation.
Duplicating 923 volume buckets took 827ms on master, 360ms here (factor 2.2).
Similar speedups are to be expected for the other usages.

URL of deployed dev instance (used for testing):

• https://putbuffer.webknossos.xyz

Steps to test:

• Edit a skeleton annotation with multiple trees, reload, should work
• Duplicate annotation, should work (skeleton, volume, editable mapping)
• Merge annotations, should work (skeleton, volume, editable mapping)

Issues:

• contributes to Use fossilDB multi-get to optimize performance, e.g. for editable mappings/proofreading #6962

---

• Removed dev-only changes like prints and application.conf edits
• Considered common edge cases
• Needs datastore update after deployment

[MichaelBuessemeyer]

Looks good to me as well. Testing went also well 👍

Just dumping thoughts here to comment on:
Currently, the buffer blocks while flushing. But having a list of buffers in the buffer would enable us to fill a second buffer while on is being flushed to the fossilddb and then the second buffer could be filled. This might not be necessary, but might save some waiting time while the buffer is being flushed. But this would also increase the complexity of this feature

[fm3]

Thanks for your review! You’re right, this might be a way to speed this up further but as you said, I’d rather avoid the complexity of having to ensure that everything is working correctly in parallel and correctly waiting for everything on the final flush.

Also, since we are serving many users in parallel, maximum parallelization could actually also be a downside. While it would serve to answer individual requests more quickly, it would potentially also use up many threads, CPU, and RPC requests, so that some (possibly smaller) requests by other users might get answered considerably later. This is also why I mostly still use serialCombined, to limit parallelization inside of individual user requests. An alternative would be to use custom thread pools for fine-grained control, but that would add another layer of complexity. I hope this also answers #8469 (comment)