Skip to content

tifffile.ZarrTiffStore wrapped in zarr.LRUStoreCache fails to read any chunks #1497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cgohlke opened this issue Aug 12, 2023 · 4 comments · Fixed by #1499
Closed

tifffile.ZarrTiffStore wrapped in zarr.LRUStoreCache fails to read any chunks #1497

cgohlke opened this issue Aug 12, 2023 · 4 comments · Fixed by #1499
Labels
bug Potential issues with the zarr-python library

Comments

@cgohlke
Copy link
Contributor

cgohlke commented Aug 12, 2023

Zarr version

2.16.0

Numcodecs version

0.11.0

Python Version

3.11.4

Operating System

Windows

Installation

py -m pip install -U zarr

Description

Since zarr v2.15.0, the tifffile.ZarrTiffStore fails to read any chunks when wrapping the store in zarr.LRUStoreCache. The chunks returned by LRUStoreCache are always zeroed.

I don't understand yet what change in v2.15 causes the failure. It may be related to ZarrTiffStore not returning all chunk keys from the keys method because there may be many million keys and it would take significant resources to parse the TIFF file(s) for all keys upfront. Instead, the store relies on the __contains__ method.

The following patch fixes the issue for me:

diff --git a/zarr/storage.py b/zarr/storage.py
index 4f7b9905..c3260638 100644
--- a/zarr/storage.py
+++ b/zarr/storage.py
@@ -2436,7 +2436,7 @@ class LRUStoreCache(Store):
         with self._mutex:
             if self._contains_cache is None:
                 self._contains_cache = set(self._keys())
-            return key in self._contains_cache
+            return key in self._contains_cache or key in self._store

     def clear(self):
         self._store.clear()

Steps to reproduce

import numpy
import zarr
import tifffile

data = numpy.random.rand(8, 8)
tifffile.imwrite('test.tif', data)
store = tifffile.imread('test.tif', aszarr=True)

try:
    # without cache
    zarray = zarr.open(store, mode='r')
    numpy.testing.assert_array_equal(zarray, data)
    # with cache
    cache = zarr.LRUStoreCache(store, max_size=2**10)
    zarray = zarr.open(cache, mode='r')
    numpy.testing.assert_array_equal(zarray, data)
finally:
    store.close()

Additional output

Traceback (most recent call last):
  File "D:\Dev\Python\tifffile\test_zarr_lrucache.py", line 16, in <module>
    numpy.testing.assert_array_equal(zarray, data)
  File "X:\Python311\Lib\site-packages\numpy\testing\_private\utils.py", line 920, in assert_array_equal
    assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
  File "X:\Python311\Lib\contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "X:\Python311\Lib\site-packages\numpy\testing\_private\utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

Mismatched elements: 64 / 64 (100%)
Max absolute difference: 0.99263945
Max relative difference: 1.
 x: array([[0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0.],...
 y: array([[0.992639, 0.580277, 0.454157, 0.055261, 0.664422, 0.82762 ,
        0.144393, 0.240842],
       [0.241279, 0.190615, 0.37115 , 0.078737, 0.706201, 0.593853,...
@cgohlke cgohlke added the bug Potential issues with the zarr-python library label Aug 12, 2023
@cgohlke
Copy link
Contributor Author

cgohlke commented Aug 12, 2023

git bisect points to #1131.

@joshmoore
Copy link
Member

Ufff Thanks, @cgohlke. Sounds very much like there was a test missing.

cc: @madsbk

@jeffpeck10x
Copy link
Contributor

To anyone affected by this, note that the results seem to be arrays of zeroes, and so the failure is silent.

Downgrading to zarr 14 will fix this issue until the next release.

@rabernat
Copy link
Contributor

We just released 2.16.1 which contains a fix for this bug. Sorry for the friction! 🙏

clemenshug added a commit to labsyspharm/cellcutter that referenced this issue Feb 21, 2024
…ary to what is claimed in the issue, version 2.16.1 did not seem to fix the bug.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Potential issues with the zarr-python library
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants