-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Labels
Description
This comment by @flying-sheep made me realise we are missing a potentially large perf optimisation for zarrs-python
. We should reuse partial decoders for batches of chunk_descriptions
that refer to the same chunk in retrieve_chunks_and_apply_index
.
https://github.com/ilan-gold/zarrs-python/blob/53af191c7ca5215c31221520da1ee1c634852b78/src/lib.rs#L312-L329
The ChunkRepresentation
can be reused for each batch too.
Relevant: https://book.zarrs.dev/arrays/array_read.html#reading-multiple-chunk-subsets