Fix a bug when setting complete chunks

[dcherian]

Closes #2849

cc @ilan-gold

[Description of PR]

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[dcherian]

this will fail for scalars, need a test for that too.

[dcherian]

Well it works for scalars, because chunk_value is now NDBuffer which does have shape. I added a test.

[d-v-b]

could you explain what the bug was, and how this PR fixes it?

[dcherian]

g = zarr.open_group("foo.zarr", zarr_format=3, mode="w")
a = g.create_array("bar", shape=(10,), chunks=(3,), dtype=int)
data = np.array([7, 8, 9])
a[slice(7, 10)] = data
np.testing.assert_array_equal(a[slice(7, 10)], data)

I think it's broken for overwriting a complete last chunk that is smaller than the chunk size, AND where your setitem value is a size that is larger than that last-chunk-size. This also suggests that somewhere we can do

chunk[size=2] = array[size=3]

and it will assign array[:2]

[ilan-gold]

Not saying it's related but reminds me of #2469

Thanks for investigating.

[dcherian]

OK I have updated the priorities of our property tests and it caught it immediately. Now we prioritize arrays of dim size >=3, and prioritize indexing the array near its end (so that we test handling the last chunk).

We also don't test complicated attrs, array names, and long array paths in the indexing tests.

I learnt a lesson. :)

[d-v-b]

nice!

[dcherian]

From manual inspection, the unit test in 0e0b34f (#2851) fails on ubuntu, windows with numpy 2.1 only.

It succeeds on my macbook.

Is someone able to debug this locally?

[LDeakin]

@dcherian What do you expect to happen in that repeated index test? Isn't it ambiguous which values should be taken from the input if the output indexes overlap?

It would be great for https://github.com/ilan-gold/zarrs-python if zarr-python could require that output indexes are disjoint.

[dcherian]

I am surprised it works, but zarr-python "sends the indexes to the chunk" so you get the last of the repeated values, numpy style.

[dcherian]

You know, it looks like the failing test is doing the opposite, and we get the first value:

array.oindex[([-1, -1, 0, 0],)] = [0, 1, 2, 3]

  ACTUAL: array([2., 0., 0., 0.])
  DESIRED: array([3, 0, 0, 1])

[dcherian]

I did not succeed in reproducing on a linux machine either so opened #2854 to revert. I've migrated the testing improvements there.

Reproduced and reported in #2855 . I am xfailing here.