From 1c7d66312a52157deb82da0d4a2144a0164c57bb Mon Sep 17 00:00:00 2001 From: jmoore Date: Mon, 7 Feb 2022 14:12:54 +0100 Subject: [PATCH 1/8] Add 2.11 release post --- _posts/2022-02-07-zarr-2.11-release.markdown | 48 ++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 _posts/2022-02-07-zarr-2.11-release.markdown diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown new file mode 100644 index 0000000..b91d003 --- /dev/null +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -0,0 +1,48 @@ +--- +layout: post +title: "Zarr Python 2.11 Release" +date: 2022-02-07 +categories: blog +permalink: /release-2-11/ +--- + +Version 2.11 of the [Python Zarr +package](https://zarr.readthedocs.io/en/stable/), has just been released. This +post provides an overview of new features in this release, especially a new +parameter that may impact the performance of writing arrays. + +## Empty chunks will no longer be written by default + +One of the advantages of the Zarr format is that it is sparse, which means that +chunks with no data (more precisely, with data equal to the fill value, which +is usually 0) don't need to be written to disk at all. They will simply be +assumed to be empty at read time. However, until this release, the Zarr library +would write these empty chunks to disk anyway. This changes in this version: a +small performance penalty at write time leads to significant speedups at read +time and in filesystem operations in the case of sparse arrays. To revert to +the old behavior, pass the argument ``write_empty_chunks=True`` to the array +creation function. + +## Fancy numpy-style indexing + + Zarr arrays now support NumPy-style fancy indexing with arrays of integer + coordinates. This is equivalent to using zarr.Array.vindex. Mixing slices and + integer arrays is not supported. + +## New base class + +This release of Zarr Python introduces a new ``BaseStore`` class that all +provided store classes implemented in Zarr Python now inherit from. This is +done as part of refactoring to enable future support of the Zarr version 3 +spec. Existing third-party stores that are a MutableMapping (e.g. dict) can be +converted to a new-style key/value store inheriting from ``BaseStore`` by +passing them as the argument to the new ``zarr.storage.KVStore`` class. For +backwards compatibility, various higher-level array creation and convenience +functions still accept plain Python dicts or other mutable mappings for the +``store`` argument, but will internally convert these to a ``KVStore``. + +## More information + +Details on these features as well as the full list of all changes in 2.11.0 +are available on the release notes: +https://zarr.readthedocs.io/en/stable/release.html#release-2-11-0 From f0d407292c2128f20726464d1fcd22145d1f7a7a Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Tue, 8 Feb 2022 01:51:46 +0530 Subject: [PATCH 2/8] Update Release 2.11.0 Blog --- _posts/2022-02-07-zarr-2.11-release.markdown | 68 +++++++++++++++----- 1 file changed, 53 insertions(+), 15 deletions(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index b91d003..f33affa 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -6,12 +6,12 @@ categories: blog permalink: /release-2-11/ --- -Version 2.11 of the [Python Zarr -package](https://zarr.readthedocs.io/en/stable/), has just been released. This -post provides an overview of new features in this release, especially a new -parameter that may impact the performance of writing arrays. +Version 2.11 of the [Python Zarr package](https://zarr.readthedocs.io/en/stable/) has just been released. 🎉 -## Empty chunks will no longer be written by default +This blog post aims to provide an overview of new features in this release, especially a new parameter that may impact the performance of writing arrays. + + +>### Empty chunks will no longer be written by default One of the advantages of the Zarr format is that it is sparse, which means that chunks with no data (more precisely, with data equal to the fill value, which @@ -20,16 +20,29 @@ assumed to be empty at read time. However, until this release, the Zarr library would write these empty chunks to disk anyway. This changes in this version: a small performance penalty at write time leads to significant speedups at read time and in filesystem operations in the case of sparse arrays. To revert to -the old behavior, pass the argument ``write_empty_chunks=True`` to the array +the old behaviour, pass the argument ``write_empty_chunks=True`` to the array creation function. -## Fancy numpy-style indexing +This feature was added by [Juan Nunez-Iglesias](https://github.com/jni) and [Davis Bennett](https://github.com/jni) with PR [#853](https://github.com/zarr-developers/zarr-python/issues/853) and [#738](https://github.com/zarr-developers/zarr-python/issues/738) +respectively. + +Here are some benchmarks: + +->The benchmark snippet will go here! + +>### Fancy numpy-style indexing + +Zarr arrays now support NumPy-style fancy indexing with arrays of integer +coordinates. This is equivalent to using ``zarr.Array.vindex``. Mixing slices and +integer arrays are not supported. + +This feature was added by [Juan Nunez-Iglesias](https://github.com/jni) with PR [#725](https://github.com/zarr-developers/zarr-python/issues/725). + +Here's a code snippet: - Zarr arrays now support NumPy-style fancy indexing with arrays of integer - coordinates. This is equivalent to using zarr.Array.vindex. Mixing slices and - integer arrays is not supported. +->Code snippet goes here! -## New base class +>### New base class This release of Zarr Python introduces a new ``BaseStore`` class that all provided store classes implemented in Zarr Python now inherit from. This is @@ -41,8 +54,33 @@ backwards compatibility, various higher-level array creation and convenience functions still accept plain Python dicts or other mutable mappings for the ``store`` argument, but will internally convert these to a ``KVStore``. -## More information +This feature was added by [Greggory Lee](https://github.com/grlee77) with PR [#839](https://github.com/zarr-developers/zarr-python/issues/839), [#789](https://github.com/zarr-developers/zarr-python/issues/789) and [#950](https://github.com/zarr-developers/zarr-python/issues/950). + +Here's a code snippet: + +->Code snippet goes here! + + +### More information + +Details on these features as well as the full list of all changes in 2.11.0 are available on the release notes. Check [here](https://zarr.readthedocs.io/en/stable/release.html#release-2-11-0). + +>Appreciation 🙌🏻 + +Shout-out to all the contributors who made release 2.11.0 possible: +- [Juan Nunez-Iglesias](https://github.com/jni) +- [Davis Bennett](https://github.com/d-v-b) +- [Gregory Lee](https://github.com/grlee77) +- [Ryan Abernathy](https://github.com/rabernat) +- [Matthias Bussonnier](https://github.com/Carreau) +- [Oren Watson](https://github.com/orenwatson) +- [Joe Hamman](https://github.com/jhamman) +- [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos) +- [Ray Bell](https://github.com/raybellwaves) +- [John Kirkham](https://github.com/jakirkham) +- [Mads R. B. Kristensen](https://github.com/madsbk) +- [Josh Moore](https://github.com/joshmoore). + +If you find the above features useful and end up using them, please mention [@zarr_dev](https://twitter.com/zarr_dev) on Twitter and tweet using #zarr or #zarrdata and we'll make sure to get it featured! ✌🏻 -Details on these features as well as the full list of all changes in 2.11.0 -are available on the release notes: -https://zarr.readthedocs.io/en/stable/release.html#release-2-11-0 +~Josh Morre and Sanket Verma \ No newline at end of file From b09dc361447f493ee7c15f536899d246ec96e23f Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Tue, 8 Feb 2022 15:56:00 +0530 Subject: [PATCH 3/8] Update-II Release 2.11.0 Blog --- _posts/2022-02-07-zarr-2.11-release.markdown | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index f33affa..856fc8b 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -23,8 +23,7 @@ time and in filesystem operations in the case of sparse arrays. To revert to the old behaviour, pass the argument ``write_empty_chunks=True`` to the array creation function. -This feature was added by [Juan Nunez-Iglesias](https://github.com/jni) and [Davis Bennett](https://github.com/jni) with PR [#853](https://github.com/zarr-developers/zarr-python/issues/853) and [#738](https://github.com/zarr-developers/zarr-python/issues/738) -respectively. +This feature was added by [Davis Bennett](https://github.com/jni) and [Juan Nunez-Iglesias](https://github.com/jni) with PR [#738](https://github.com/zarr-developers/zarr-python/issues/738) and [#853](https://github.com/zarr-developers/zarr-python/issues/853) respectively. Here are some benchmarks: From 4d7e30988fd682ae06e2091aa0b209fca8f05b3f Mon Sep 17 00:00:00 2001 From: jmoore Date: Tue, 8 Feb 2022 13:06:00 +0100 Subject: [PATCH 4/8] Add chunk benchmark --- _posts/2022-02-07-zarr-2.11-release.markdown | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index 856fc8b..aeb4e3f 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -11,7 +11,7 @@ Version 2.11 of the [Python Zarr package](https://zarr.readthedocs.io/en/stable/ This blog post aims to provide an overview of new features in this release, especially a new parameter that may impact the performance of writing arrays. ->### Empty chunks will no longer be written by default +### Empty chunks will no longer be written by default One of the advantages of the Zarr format is that it is sparse, which means that chunks with no data (more precisely, with data equal to the fill value, which @@ -23,9 +23,18 @@ time and in filesystem operations in the case of sparse arrays. To revert to the old behaviour, pass the argument ``write_empty_chunks=True`` to the array creation function. -This feature was added by [Davis Bennett](https://github.com/jni) and [Juan Nunez-Iglesias](https://github.com/jni) with PR [#738](https://github.com/zarr-developers/zarr-python/issues/738) and [#853](https://github.com/zarr-developers/zarr-python/issues/853) respectively. +This feature was added by [Davis Bennett](https://github.com/jni) and [Juan Nunez-Iglesias](https://github.com/jni) +with PR [#738](https://github.com/zarr-developers/zarr-python/issues/738) and [#853](https://github.com/zarr-developers/zarr-python/issues/853) respectively. -Here are some benchmarks: +Some preliminary benchmark results are shown in [PR 853](https://github.com/zarr-developers/zarr-python/pull/853#issuecomment-988245713). For example, + +![benchmark results](https://user-images.githubusercontent.com/3805136/145100731-95ea83c5-da54-4950-8849-4df157c1a081.png) + +shows how with the previous setting of `write_empty_chunks=True` the speed of +writing a fill-valued chunk is the same as writing a randomly-filled chunk. +With the new default of `write_empty_chunks_False`, writing the fill-valued +chunk is *much* faster while writing a randomly-filled chunk is slightly +slower. ->The benchmark snippet will go here! @@ -82,4 +91,4 @@ Shout-out to all the contributors who made release 2.11.0 possible: If you find the above features useful and end up using them, please mention [@zarr_dev](https://twitter.com/zarr_dev) on Twitter and tweet using #zarr or #zarrdata and we'll make sure to get it featured! ✌🏻 -~Josh Morre and Sanket Verma \ No newline at end of file +~Josh Morre and Sanket Verma From 1e4e27728eff6d3eb42c46feede8bf8b17b6a1cb Mon Sep 17 00:00:00 2001 From: jmoore Date: Tue, 8 Feb 2022 13:06:48 +0100 Subject: [PATCH 5/8] Minor adjustments --- _posts/2022-02-07-zarr-2.11-release.markdown | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index aeb4e3f..a8c86ce 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -11,7 +11,7 @@ Version 2.11 of the [Python Zarr package](https://zarr.readthedocs.io/en/stable/ This blog post aims to provide an overview of new features in this release, especially a new parameter that may impact the performance of writing arrays. -### Empty chunks will no longer be written by default +## Empty chunks will no longer be written by default One of the advantages of the Zarr format is that it is sparse, which means that chunks with no data (more precisely, with data equal to the fill value, which @@ -38,7 +38,7 @@ slower. ->The benchmark snippet will go here! ->### Fancy numpy-style indexing +## Fancy numpy-style indexing Zarr arrays now support NumPy-style fancy indexing with arrays of integer coordinates. This is equivalent to using ``zarr.Array.vindex``. Mixing slices and @@ -50,7 +50,7 @@ Here's a code snippet: ->Code snippet goes here! ->### New base class +## New base class This release of Zarr Python introduces a new ``BaseStore`` class that all provided store classes implemented in Zarr Python now inherit from. This is @@ -69,7 +69,7 @@ Here's a code snippet: ->Code snippet goes here! -### More information +## More information Details on these features as well as the full list of all changes in 2.11.0 are available on the release notes. Check [here](https://zarr.readthedocs.io/en/stable/release.html#release-2-11-0). @@ -89,6 +89,6 @@ Shout-out to all the contributors who made release 2.11.0 possible: - [Mads R. B. Kristensen](https://github.com/madsbk) - [Josh Moore](https://github.com/joshmoore). -If you find the above features useful and end up using them, please mention [@zarr_dev](https://twitter.com/zarr_dev) on Twitter and tweet using #zarr or #zarrdata and we'll make sure to get it featured! ✌🏻 +If you find the above features useful and end up using them, please mention [@zarr_dev](https://twitter.com/zarr_dev) on Twitter and tweet using #ZarrData and we'll make sure to get it featured! ✌🏻 -~Josh Morre and Sanket Verma +~Sanket Verma From 62f18eddb3ac4dbfc55375a6fd3f059f8cf12efd Mon Sep 17 00:00:00 2001 From: jmoore Date: Tue, 8 Feb 2022 13:11:20 +0100 Subject: [PATCH 6/8] Add vindex snippet --- _posts/2022-02-07-zarr-2.11-release.markdown | 28 ++++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index a8c86ce..10dbd05 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -36,19 +36,27 @@ With the new default of `write_empty_chunks_False`, writing the fill-valued chunk is *much* faster while writing a randomly-filled chunk is slightly slower. -->The benchmark snippet will go here! - ## Fancy numpy-style indexing Zarr arrays now support NumPy-style fancy indexing with arrays of integer coordinates. This is equivalent to using ``zarr.Array.vindex``. Mixing slices and integer arrays are not supported. -This feature was added by [Juan Nunez-Iglesias](https://github.com/jni) with PR [#725](https://github.com/zarr-developers/zarr-python/issues/725). +``` + >>> z.vindex[[0, 2], [1, 3]] + array([-1, -2]) + >>> z.vindex[[0, 2], [1, 3]] = [-3, -4] + >>> z[:] + array([[ 0, -3, 2, 3, 4], + [ 5, 6, 7, 8, 9], + [10, 11, 12, -4, 14]]) + >>> z[[0, 2], [1, 3]] + array([-3, -4]) +``` -Here's a code snippet: +See [Advanced indexing](https://zarr.readthedocs.io/en/stable/tutorial.html#advanced-indexing) in the tutorial for more information. -->Code snippet goes here! +This feature was added by [Juan Nunez-Iglesias](https://github.com/jni) with PR [#725](https://github.com/zarr-developers/zarr-python/issues/725). ## New base class @@ -62,12 +70,10 @@ backwards compatibility, various higher-level array creation and convenience functions still accept plain Python dicts or other mutable mappings for the ``store`` argument, but will internally convert these to a ``KVStore``. -This feature was added by [Greggory Lee](https://github.com/grlee77) with PR [#839](https://github.com/zarr-developers/zarr-python/issues/839), [#789](https://github.com/zarr-developers/zarr-python/issues/789) and [#950](https://github.com/zarr-developers/zarr-python/issues/950). - -Here's a code snippet: - -->Code snippet goes here! - +This feature was added by [Greggory Lee](https://github.com/grlee77) with PR +[#839](https://github.com/zarr-developers/zarr-python/issues/839), +[#789](https://github.com/zarr-developers/zarr-python/issues/789) and +[#950](https://github.com/zarr-developers/zarr-python/issues/950). ## More information From 50ad47de77834f5967f085a2a18fe17bc23ccecd Mon Sep 17 00:00:00 2001 From: jmoore Date: Tue, 8 Feb 2022 13:12:15 +0100 Subject: [PATCH 7/8] Use header for 'Appreciation' as well --- _posts/2022-02-07-zarr-2.11-release.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index 10dbd05..0fb6e9d 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -79,7 +79,7 @@ This feature was added by [Greggory Lee](https://github.com/grlee77) with PR Details on these features as well as the full list of all changes in 2.11.0 are available on the release notes. Check [here](https://zarr.readthedocs.io/en/stable/release.html#release-2-11-0). ->Appreciation 🙌🏻 +## Appreciation 🙌🏻 Shout-out to all the contributors who made release 2.11.0 possible: - [Juan Nunez-Iglesias](https://github.com/jni) From 63026a2118eba0df9aae60ee52b1482bbc7f9d00 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Tue, 8 Feb 2022 17:51:13 +0530 Subject: [PATCH 8/8] Minor fix --- _posts/2022-02-07-zarr-2.11-release.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2022-02-07-zarr-2.11-release.markdown b/_posts/2022-02-07-zarr-2.11-release.markdown index 0fb6e9d..2c8b376 100644 --- a/_posts/2022-02-07-zarr-2.11-release.markdown +++ b/_posts/2022-02-07-zarr-2.11-release.markdown @@ -26,7 +26,7 @@ creation function. This feature was added by [Davis Bennett](https://github.com/jni) and [Juan Nunez-Iglesias](https://github.com/jni) with PR [#738](https://github.com/zarr-developers/zarr-python/issues/738) and [#853](https://github.com/zarr-developers/zarr-python/issues/853) respectively. -Some preliminary benchmark results are shown in [PR 853](https://github.com/zarr-developers/zarr-python/pull/853#issuecomment-988245713). For example, +Some preliminary benchmark results are shown in PR [#853](https://github.com/zarr-developers/zarr-python/pull/853#issuecomment-988245713). For example: ![benchmark results](https://user-images.githubusercontent.com/3805136/145100731-95ea83c5-da54-4950-8849-4df157c1a081.png)