Add asynchronous load method #10327
Draft
Add asynchronous load method #10327
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
topic-backends
topic-indexing
topic-documentation
topic-zarr
for more information, see https://pre-commit.ci
github-actions
bot
added
CI
for more information, see https://pre-commit.ci
|
These failing tests from the CI do not fail when I run them locally, which is interesting. FAILED xarray/tests/test_backends.py::TestH5NetCDFViaDaskData::test_outer_indexing_reversed - ValueError: dimensions ('t', 'y', 'x') must have the same length as the number of data dimensions, ndim=4
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_outer_indexing_reversed - ValueError: dimensions ('t', 'y', 'x') must have the same length as the number of data dimensions, ndim=4
FAILED xarray/tests/test_backends.py::TestDask::test_outer_indexing_reversed - ValueError: dimensions ('t', 'y', 'x') must have the same length as the number of data dimensions, ndim=4
= 3 failed, 18235 passed, 1269 skipped, 77 xfailed, 15 xpassed, 2555 warnings in 487.15s (0:08:07) =
Error: Process completed with exit code 1. |
TomNicholas
commented
May 19, 2025
TomNicholas
commented
May 19, 2025
| @@ -267,13 +268,23 @@ def robust_getitem(array, key, catch=Exception, max_retries=6, initial_delay=500 | |||
| time.sleep(1e-3 * next_delay) | |||
|
|
|||
|
|
|||
| class BackendArray(NdimSizeLenMixin, indexing.ExplicitlyIndexed): | |||
| class BackendArray(ABC, NdimSizeLenMixin, indexing.ExplicitlyIndexed): | |||
There was a problem hiding this comment.
As __getitem__ is required, I feel like BackendArray should always have been an ABC.
TomNicholas
commented
May 19, 2025
Comment on lines
+277
to
+278
| async def async_getitem(key: indexing.ExplicitIndexer) -> np.typing.ArrayLike: | ||
| raise NotImplementedError("Backend does not not support asynchronous loading") |
There was a problem hiding this comment.
I've implemented this for the ZarrArray class but in theory it could be supported by other backends too.
There was a problem hiding this comment.
This might not be the desired behaviour though - this currently means if you opened a dataset from netCDF and called ds.load_async you would get a NotImplementedError. Would it be better to quietly just block instead?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an
.async_load()method toVariable, which works by plumbing asyncget_duck_arrayall the way down until it finally gets to the async methods zarr v3 exposes.Needs a lot of refactoring before it could be merged, but it works.
whats-new.rstapi.rstAPI:
Variable.load_asyncDataArray.load_asyncDataset.load_asyncDataTree.load_asyncload_dataset?load_dataarray?