This repository was archived by the owner on Sep 11, 2023. It is now read-only.
Experiment with calling dataset.load()
_after_ joining examples into batch
#475
Labels
enhancement
New feature or request
For the Zarr DataSources, it may be faster to load the data into memory after joining (lazily loaded) examples.
i.e. call
.load()
towards the end ofget_batch()
instead of at the end ofget_example()
.This should allow
dask
to do a better job of scheduling what needs to be done. And might result in faster times per batch.The text was updated successfully, but these errors were encountered: