Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Experiment with calling dataset.load() _after_ joining examples into batch #475

Open
Tracked by #341
JackKelly opened this issue Nov 18, 2021 · 0 comments
Open
Tracked by #341
Labels
enhancement New feature or request

Comments

@JackKelly
Copy link
Member

For the Zarr DataSources, it may be faster to load the data into memory after joining (lazily loaded) examples.

i.e. call .load() towards the end of get_batch() instead of at the end of get_example().

This should allow dask to do a better job of scheduling what needs to be done. And might result in faster times per batch.

@JackKelly JackKelly added the enhancement New feature or request label Nov 18, 2021
@JackKelly JackKelly moved this to Todo in Nowcasting Nov 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
No open projects
Status: Todo
Development

No branches or pull requests

1 participant