Skip to content

Commit eef18f3

Browse files
jhammandstansby
jhamman
and
dstansby
authored
docs: add migration page to user guide (#2596)
* docs: split tutorial into multiple user guide sections * docs: add migration page to user guide * rev migration guide again * fixup * Apply suggestions from code review Co-authored-by: David Stansby <[email protected]> * update migration guide * update migration guide * fixup * removed config section --------- Co-authored-by: David Stansby <[email protected]>
1 parent 38953e3 commit eef18f3

File tree

5 files changed

+217
-8
lines changed

5 files changed

+217
-8
lines changed

‎docs/index.rst

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -52,15 +52,14 @@ Zarr is a file storage format for chunked, compressed, N-dimensional arrays base
5252
.. grid-item-card::
5353
:img-top: _static/index_user_guide.svg
5454

55-
Guide
56-
^^^^^
55+
Guide
56+
^^^^^
5757

58-
A detailed guide for how to use Zarr-Python.
58+
A detailed guide for how to use Zarr-Python.
5959

6060
+++
6161

62-
.. button-ref:: user-guide
63-
:ref-type: ref
62+
.. button-ref:: user-guide/index
6463
:expand:
6564
:color: dark
6665
:click-parent:

‎docs/user-guide/config.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
Runtime configuration
44
=====================
55

6-
The :mod:`zarr.core.config` module is responsible for managing the configuration of zarr
7-
and is based on the `donfig <https://github.com/pytroll/donfig>`_ Python library.
6+
:mod:`zarr.config <zarr.core.config>` is responsible for managing the configuration of zarr and
7+
is based on the `donfig <https://github.com/pytroll/donfig>`_ Python library.
88

99
Configuration values can be set using code like the following::
1010

‎docs/user-guide/index.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@ User guide
1111
attributes
1212
storage
1313
config
14+
v3_migration
1415

1516
.. Coming soon
1617
installation
17-
v3_migration
1818
1919
Advanced Topics
2020
---------------

‎docs/user-guide/storage.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,8 @@ Zarr data (metadata and chunks) to a dictionary.:
9999
>>> zarr.create_array(store=store, shape=(2,), dtype='float64')
100100
<Array memory://... shape=(2,) dtype=float64>
101101

102+
.. _user-guide-custom-stores:
103+
102104
Developing custom stores
103105
------------------------
104106

‎docs/user-guide/v3_migration.rst

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
3.0 Migration Guide
2+
===================
3+
4+
Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of the
5+
goals motivating this refactor included:
6+
7+
* adding support for the Zarr V3 specification (along with the Zarr V2 specification)
8+
* cleaning up internal and user facing APIs
9+
* improving performance (particularly in high latency storage environments like
10+
cloud object stores)
11+
12+
To accommodate this, Zarr-Python 3 introduces a number of changes to the API, including a number
13+
of significant breaking changes and deprecations.
14+
15+
This page provides a guide explaining breaking changes and deprecations to help you
16+
migrate your code from version 2 to version 3. If we have missed anything, please
17+
open a `GitHub issue <https://github.com/zarr-developers/zarr-python/issues/new>`_
18+
so we can improve this guide.
19+
20+
Compatibility target
21+
--------------------
22+
23+
The goals described above necessitated some breaking changes to the API (hence the
24+
major version update), but where possible we have maintained backwards compatibility
25+
in the most widely used parts of the API. This in the :class:`zarr.Array` and
26+
:class:`zarr.Group` classes and the "top-level API" (e.g. :func:`zarr.open_array` and
27+
:func:`zarr.open_group`).
28+
29+
Getting ready for 3.0
30+
---------------------
31+
32+
Before migrating to Zarr-Python 3, we suggest projects that depend on Zarr-Python take
33+
the following actions in order:
34+
35+
1. Pin the supported Zarr-Python version to ``zarr>=2,<3``. This is a best practice
36+
and will protect your users from any incompatibilities that may arise during the
37+
release of Zarr-Python 3. This pin can be removed after migrating to Zarr-Python 3.
38+
2. Limit your imports from the Zarr-Python package. Most of the primary API ``zarr.*``
39+
will be compatible in Zarr-Python 3. However, the following breaking API changes are
40+
planned:
41+
42+
- ``numcodecs.*`` will no longer be available in ``zarr.*``. To migrate, import codecs
43+
directly from ``numcodecs``:
44+
45+
.. code-block:: python
46+
47+
from numcodecs import Blosc
48+
# instead of:
49+
# from zarr import Blosc
50+
51+
- The ``zarr.v3_api_available`` feature flag is being removed. In Zarr-Python 3
52+
the v3 API is always available, so you shouldn't need to use this flag.
53+
- The following internal modules are being removed or significantly changed. If
54+
your application relies on imports from any of the below modules, you will need
55+
to either a) modify your application to no longer rely on these imports or b)
56+
vendor the parts of the specific modules that you need.
57+
58+
* ``zarr.attrs`` has gone, with no replacement
59+
* ``zarr.codecs`` has gone, use ``numcodecs`` instead
60+
* ``zarr.context`` has gone, with no replacement
61+
* ``zarr.core`` remains but should be considered private API
62+
* ``zarr.hierarchy`` has gone, with no replacement (use ``zarr.Group`` inplace of ``zarr.hierarchy.Group``)
63+
* ``zarr.indexing`` has gone, with no replacement
64+
* ``zarr.meta`` has gone, with no replacement
65+
* ``zarr.meta_v1`` has gone, with no replacement
66+
* ``zarr.sync`` has gone, with no replacement
67+
* ``zarr.types`` has gone, with no replacement
68+
* ``zarr.util`` has gone, with no replacement
69+
* ``zarr.n5`` has gone, see below for an alternative N5 options
70+
71+
3. Test that your package works with version 3.
72+
4. Update the pin to include ``zarr>=3,<4``.
73+
74+
Zarr-Python 2 support window
75+
----------------------------
76+
77+
Zarr-Python 2.x is still available, though we recommend migrating to Zarr-Python 3 for
78+
its performance improvements and new features. Security and bug fixes will be made to
79+
the 2.x series for at least six months following the first Zarr-Python 3 release.
80+
If you need to use the latest Zarr-Python 2 release, you can install it with:
81+
82+
.. code-block:: console
83+
84+
$ pip install "zarr==2.*"
85+
86+
.. note::
87+
Development and maintenance of the 2.x release series has moved to the
88+
`support/v2 <https://github.com/zarr-developers/zarr-python/tree/support/v2>`_ branch.
89+
Issues and pull requests related to this branch are tagged with the
90+
`V2 <https://github.com/zarr-developers/zarr-python/labels/V2>`_ label.
91+
92+
Migrating to Zarr-Python 3
93+
--------------------------
94+
95+
The following sections provide details on breaking changes in Zarr-Python 3.
96+
97+
The Array class
98+
~~~~~~~~~~~~~~~
99+
100+
1. Disallow direct construction - the signature for initializing the ``Array`` class has changed
101+
significantly. Please use :func:`zarr.create_array` or :func:`zarr.open_array` instead of
102+
directly constructing the :class:`zarr.Array` class.
103+
104+
2. Defaulting to ``zarr_format=3`` - newly created arrays will use the version 3 of the
105+
Zarr specification. To continue using version 2, set ``zarr_format=2`` when creating arrays
106+
or set ``default_zarr_version=2`` in Zarr's :ref:`runtime configuration <user-guide-config>`.
107+
108+
The Group class
109+
~~~~~~~~~~~~~~~
110+
111+
1. Disallow direct construction - use :func:`zarr.open_group` or :func:`zarr.create_group`
112+
instead of directly constructing the :class:`zarr.Group` class.
113+
2. Most of the h5py compatibility methods are deprecated and will issue warnings if used.
114+
The following functions are drop in replacements that have the same signature and functionality:
115+
116+
- Use :func:`zarr.Group.create_array` in place of :func:`zarr.Group.create_dataset`
117+
- Use :func:`zarr.Group.require_array` in place of :func:`zarr.Group.require_dataset`
118+
119+
The Store class
120+
~~~~~~~~~~~~~~~
121+
122+
The Store API has changed significant in Zarr-Python 3. The most notable changes to the
123+
Store API are:
124+
125+
1. Replaced the ``MutableMapping`` base class in favor of a custom abstract base class
126+
(:class:`zarr.abc.store.Store`).
127+
2. Switched to an asynchronous interface for all store methods that result in IO. This
128+
change ensures that all store methods are non-blocking and are as performant as
129+
possible.
130+
131+
Beyond the changes store interface, a number of deprecated stores were also removed in
132+
Zarr-Python 3. See :issue:`1274` for more details on the removal of these stores.
133+
134+
- ``N5Store`` - see https://github.com/zarr-developers/n5py for an alternative interface to
135+
N5 formatted data.
136+
- ``ABSStore`` - use the :class:`zarr.storage.FsspecStore` instead along with fsspec's
137+
`adlfs backend <https://github.com/fsspec/adlfs>`_.
138+
139+
The following stores have been removed altogether. Users who need these stores will have to
140+
implement their own version in zarr-python v3.
141+
142+
- ``DBMStore``
143+
- ``LMDBStore``
144+
- ``SQLiteStore``
145+
- ``MongoDBStore``
146+
- ``RedisStore``
147+
148+
At present, the latter five stores in this list do not have an equivalent in Zarr-Python 3.
149+
If you are interested in developing a custom store that targets these backends, see
150+
:ref:`developing custom stores <user-guide-custom-stores>` or open an
151+
`issue <https://github.com/zarr-developers/zarr-python/issues>`_ to discuss your use case.
152+
153+
Dependencies
154+
~~~~~~~~~~~~
155+
156+
When installing using ``pip``:
157+
158+
- The new ``remote`` dependency group can be used to install a supported version of
159+
``fsspec``, required for remote data access.
160+
- The new ``gpu`` dependency group can be used to install a supported version of
161+
``cuda``, required for GPU functionality.
162+
- The ``jupyter`` optional dependency group has been removed, since v3 contains no
163+
jupyter specific functionality.
164+
165+
Miscellaneous
166+
~~~~~~~~~~~~~
167+
168+
- The keyword argument ``zarr_version`` available in most creation functions in :mod:`zarr`
169+
(e.g. :func:`zarr.create`, :func:`zarr.open`, :func:`zarr.group`, :func:`zarr.array`) has
170+
been deprecated in favor of ``zarr_format``.
171+
172+
🚧 Work in Progress 🚧
173+
----------------------
174+
175+
Zarr-Python 3 is still under active development, and is not yet fully complete.
176+
The following list summarizes areas of the codebase that we expect to build out
177+
after the 3.0.0 release. If features listed below are important to your use case
178+
of Zarr-Python, please open (or comment on) a
179+
`GitHub issue <https://github.com/zarr-developers/zarr-python/issues/new>`_.
180+
181+
- The following functions / methods have not been ported to Zarr-Python 3 yet:
182+
183+
* :func:`zarr.copy` (:issue:`2407`)
184+
* :func:`zarr.copy_all` (:issue:`2407`)
185+
* :func:`zarr.copy_store` (:issue:`2407`)
186+
* :func:`zarr.Group.move` (:issue:`2108`)
187+
188+
- The following features (corresponding to function arguments to functions in
189+
:mod:`zarr`) have not been ported to Zarr-Python 3 yet. Using these features
190+
will raise a warning or a ``NotImplementedError``:
191+
192+
* ``cache_attrs``
193+
* ``cache_metadata``
194+
* ``chunk_store`` (:issue:`2495`)
195+
* ``meta_array``
196+
* ``object_codec`` (:issue:`2617`)
197+
* ``synchronizer`` (:issue:`1596`)
198+
* ``dimension_separator``
199+
200+
- The following features that were supported by Zarr-Python 2 have not been ported
201+
to Zarr-Python 3 yet:
202+
203+
* Structured arrays / dtypes (:issue:`2134`)
204+
* Fixed-length string dtypes (:issue:`2347`)
205+
* Datetime and timedelta dtypes (:issue:`2616`)
206+
* Object dtypes (:issue:`2617`)
207+
* Ragged arrays (:issue:`2618`)
208+
* Groups and Arrays do not implement ``__enter__`` and ``__exit__`` protocols (:issue:`2619`)

0 commit comments

Comments
 (0)