numcodecs.zfpy is ready #229

halehawk · 2020-04-21T02:53:17Z

numcodecs.zfpy is ready with pip install from https://test.pypi.org/simple/ zfpy==0.5.5rc1, this package has no py38 build and osx build.

TODO:

Unit tests and/or doctests in docstrings
tox -e py38 passes locally
Docstrings and API docs for any new/modified user-facing classes and functions
Changes documented in docs/release.rst
tox -e docs passes locally
AppVeyor and Travis CI passes
Test coverage to 100% (Coveralls passes)

lindstro · 2021-03-17T22:34:31Z

CI appears to be failing on Python 3.9. Guess it is just due to lack of wheels on that platform. Suggested some changes above to fix that

I can add 3.9 wheels. Someone recently requested this, but at the time I wasn't sure if we require MB_ML_VER=2010 on Linux as we do for 3.8. I can't seem to find the thread where this was discussed. Does anyone know?

jakirkham · 2021-03-17T23:36:52Z

Honestly I think a new release of zfpy is probably more important than Python 3.9 support. I don’t know how much work that would be though

halehawk · 2021-03-18T00:27:28Z

I used zfpy==0.5.5; python_version <'3.9' Looks like it worked, zfpy was ignored. But I got error at Build Docs autodoc: failed to import class 'ZFPY' from module 'numcodecs.zfpy'; the following exception was raised: 93Traceback (most recent call last): 94 File "/opt/hostedtoolcache/Python/3.9.2/x64/lib/python3.9/site-packages/sphinx/util/inspect.py", line 393, in safe_getattr 95 return getattr(obj, name, *defargs) 96 AttributeError: module 'numcodecs.zfpy' has no attribute 'ZFPY' How can I skip it?

…

On Wed, Mar 17, 2021 at 5:37 PM jakirkham ***@***.***> wrote: Honestly I think a new release of zfpy is probably more important than Python 3.9 support. I don’t know how much work that would be though — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFCNNM5KOLN5CJDHRZDTEE4KLANCNFSM4MM553EQ> .

lindstro · 2021-03-18T00:36:19Z

Honestly I think a new release of zfpy is probably more important than Python 3.9 support. I don’t know how much work that would be though

We've essentially frozen new features and are working on tests and documentation for zfp 0.5.6. Shouldn't be too much longer. But once we do the release, it would make sense to include Python 3.9.

jakirkham · 2021-03-18T01:07:15Z

Yep that makes sense. Knew that work towards a release was ongoing. So didn't want to ask for more things, but yeah agree that it makes sense to support Python 3.9 at the same time

halehawk · 2021-03-18T02:40:19Z

@jakirkham, I tried autodoc_mock_imports, it doesn't work. Can I remove zfpy.rst temporarily?

…

On Wed, Mar 17, 2021 at 7:07 PM jakirkham ***@***.***> wrote: Yep that makes sense. Knew that work towards a release was ongoing. So didn't want to ask for more things, but yeah agree that it makes sense to support Python 3.9 at the same time — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFCB26ADWJJRXS2EC3TTEFG5HANCNFSM4MM553EQ> .

jakirkham · 2021-03-18T16:51:33Z

Could we just skip building the docs on Python 3.9? Maybe build them on Python 3.8 instead?

halehawk · 2021-03-18T17:42:59Z

I cannot figure out how to skip building the docs on Python 3.9 only, I added zfpy to MOCK_MODULES = ['msgpack','zfpy'] in conf.py. At least now, all tests passed.

…

On Thu, Mar 18, 2021 at 10:51 AM jakirkham ***@***.***> wrote: Could we just skip building the docs on Python 3.9? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFCLFDWSVZOXA5BTSVDTEIVSTANCNFSM4MM553EQ> .

jakirkham · 2021-03-18T18:13:33Z

Thank you @halehawk! 😄

pjpetersik · 2021-09-30T11:26:45Z

@rabernat and @halehawk What is the reason that the ZFPY Codec actually uses ensure_contiguous_ndarray instead of ensure_ndarray?

When I was playing around with this Codec for compression of weather model data, i.e. 2m temperature from ICON-EU, replacing ensure_contiguous_ndarray with ensure_ndarray did not led to any errors and led to higher compression ratios.

Am I running in some problems if I replace ensure_contiguous_ndarray with ensure_ndarray in a custom ZFPY Codec?

halehawk · 2021-09-30T16:09:43Z

@pjpetersik, do you mean ensure_contiguous_ndarray in zfpy.encode? @pjpetersik if the zfp.compress_numpy which is called in zfpy.encode gets a higher dimension of array, it can get a better compression ratio. From your description, it looks like you got this situation. @jakirkham is ensure_contiguous_ndarray supposed to keep the original shape of the array? Or does it convert the original array to a flat array?

…

On Thu, Sep 30, 2021 at 5:26 AM Paul Petersik ***@***.***> wrote: @rabernat <https://github.com/rabernat> and @halehawk <https://github.com/halehawk> What is the reason that the ZFPY Codec actually uses ensure_contiguous_ndarray instead of ensure_ndarray? When I was playing around with this Codec for compression of weather model data, i.e. 2m temperature from ICON-EU, replacing ensure_contiguous_ndarray with ensure_ndarray did not led to any errors and led to higher compression ratios. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFCEIIPKYMBHCHDR7XTUERCQDANCNFSM4MM553EQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

pjpetersik · 2021-09-30T17:46:10Z

@halehawk I meant ensure_contiguous_ndarray in the ZFPY Codec class:

numcodecs/numcodecs/zfpy.py

Line 58 in e364ceb

buf = ensure_contiguous_ndarray(buf)

jakirkham · 2021-09-30T17:55:00Z

Many of the compressors we use don't know how to handle shape correctly hence why it does that

We could add a flag to ensure_contiguous_ndarray to skip the reshape step. Then use that flag here

halehawk · 2021-09-30T19:06:17Z

@jakirkham Will you add the flag? Then I can use the flag in the encode. Thanks!

…

On Thu, Sep 30, 2021 at 11:55 AM jakirkham ***@***.***> wrote: Many of the compressors we use don't know how to handle shape correctly hence why it does that We could add a flag to ensure_contiguous_ndarray to skip the reshape step. Then use that flag here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFFOCRUFLHVD2HEZHKTUESQABANCNFSM4MM553EQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

jakirkham · 2021-09-30T19:14:25Z

@halehawk it probably will be a while before I have time to do that. In the midst of doing several release atm. If you or @pjpetersik , would like to do a PR though would be happy to review :)

halehawk · 2021-09-30T19:52:35Z

I can try to do a PR, just hope all the build tests can be easily passed. @jakirkham @pjpetersik

…

On Thu, Sep 30, 2021 at 1:14 PM jakirkham ***@***.***> wrote: @halehawk <https://github.com/halehawk> it probably will be a while before I have time to do that. In the midst of doing several release atm. If you or @pjpetersik <https://github.com/pjpetersik> , would like to do a PR though would be happy to review :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFDOGZW4VOB5CMROP6DUESZJZANCNFSM4MM553EQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

halehawk · 2021-11-02T21:37:56Z

@jakirkham I tried to add a flag, then I failed the test on f-contiguous array. Can I flat f-contiguous arrays but not c-contiguous arrays?

…

On Thu, Sep 30, 2021 at 1:52 PM Haiying Xu ***@***.***> wrote: I can try to do a PR, just hope all the build tests can be easily passed. @jakirkham @pjpetersik On Thu, Sep 30, 2021 at 1:14 PM jakirkham ***@***.***> wrote: > @halehawk <https://github.com/halehawk> it probably will be a while > before I have time to do that. In the midst of doing several release atm. > If you or @pjpetersik <https://github.com/pjpetersik> , would like to do > a PR though would be happy to review :) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#229 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACAPEFDOGZW4VOB5CMROP6DUESZJZANCNFSM4MM553EQ> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > >

jakirkham · 2021-11-02T22:03:28Z

Maybe it would be best to move this discussion to a new issue where it is easier to track

halehawk added 30 commits December 21, 2018 15:50

added zfp compressor

80a571c

modified zfp in setup.py for new zfp054 directory

50cfdb5

changed print statement to raise,also changed zfp source path

8b40523

Merge remote-tracking branch 'upstream/master'

dbc2b64

added set_stride, changed zfp source path

33de949

added zfp source as a submodule

035f90d

removed zfp source as a submodule

1adcf4b

added zfp module

a33e947

added zfp docs

6acb24d

used zfp submodule and define type by _type

812441a

removed blanked lines

5972e09

added white space after mode

1a7c105

modified comments to meet numpy docstring

161cbca

updated c-blosc

d44f308

format setup.py

a1674da

added zfp.c

27676f4

add path to bitstream.h

785547f

updated include path

f0361cb

update zfp.c with include path

8cfad3e

changed include path again

e205381

Merge branch 'master' into master

fce55da

sync with latest numcodecs master

fa33fa4

Removed zfp submodule

4f90933

Removed zfp_extension

dc37e5d

Remove test_zfp.py

6ef6096

added zfp.py

fffc757

removed zfp.py

fcf92fa

added zfpy.py

e896edf

add pip install zfpy

ec63632

add pip install modified

1bd37da

Update requirements_dev.txt

3e6bd38

halehawk added 4 commits March 17, 2021 16:58

Update requirements_dev.txt

97abde3

Update requirements_rtfd.txt

3669bd5

Update requirements_rtfd.txt

d561587

Update requirements_dev.txt

c7b699a

halehawk added 3 commits March 17, 2021 19:36

Update conf.py

97848a0

Update conf.py

bd5d835

Update conf.py

fa34826

halehawk added 2 commits March 18, 2021 10:15

Merge branch 'master' into zfpy

7e96469

Update conf.py

0c44788

Update conf.py

44dde11

jakirkham merged commit d16d1ea into zarr-developers:master Mar 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

numcodecs.zfpy is ready #229

numcodecs.zfpy is ready #229

halehawk commented Apr 21, 2020 •

edited

Loading

lindstro commented Mar 17, 2021

jakirkham commented Mar 17, 2021

halehawk commented Mar 18, 2021 via email

lindstro commented Mar 18, 2021

jakirkham commented Mar 18, 2021

halehawk commented Mar 18, 2021 via email

jakirkham commented Mar 18, 2021 •

edited

Loading

halehawk commented Mar 18, 2021 via email

jakirkham commented Mar 18, 2021

pjpetersik commented Sep 30, 2021 •

edited

Loading

halehawk commented Sep 30, 2021 via email

pjpetersik commented Sep 30, 2021

jakirkham commented Sep 30, 2021

halehawk commented Sep 30, 2021 via email

jakirkham commented Sep 30, 2021

halehawk commented Sep 30, 2021 via email

halehawk commented Nov 2, 2021 via email

jakirkham commented Nov 2, 2021

numcodecs.zfpy is ready #229

numcodecs.zfpy is ready #229

Conversation

halehawk commented Apr 21, 2020 • edited Loading

lindstro commented Mar 17, 2021

jakirkham commented Mar 17, 2021

halehawk commented Mar 18, 2021 via email

lindstro commented Mar 18, 2021

jakirkham commented Mar 18, 2021

halehawk commented Mar 18, 2021 via email

jakirkham commented Mar 18, 2021 • edited Loading

halehawk commented Mar 18, 2021 via email

jakirkham commented Mar 18, 2021

pjpetersik commented Sep 30, 2021 • edited Loading

halehawk commented Sep 30, 2021 via email

pjpetersik commented Sep 30, 2021

jakirkham commented Sep 30, 2021

halehawk commented Sep 30, 2021 via email

jakirkham commented Sep 30, 2021

halehawk commented Sep 30, 2021 via email

halehawk commented Nov 2, 2021 via email

jakirkham commented Nov 2, 2021

halehawk commented Apr 21, 2020 •

edited

Loading

jakirkham commented Mar 18, 2021 •

edited

Loading

pjpetersik commented Sep 30, 2021 •

edited

Loading