Provide openeo package on conda-forge

[soxofaan]

We provide openeo python client library already on pypi.org, but it could be beneficial to our adoption rate and installation friendliness to also provide it through conda on conda-forge channel.

I'm not sure at the moment what has to happen to make it possible, how much additional maintenance it would require and if it is all worth the trouble. Just putting it out here to collect some feedback, if any.

[enolfc]

We do manage most packages with conda in the EGI Notebooks service and having packages in conda would simplify the installation of OpenEO.

[sebastian-luna-valero]

Hi,

I am happy to help create a conda package for openeo-python-client. Here is the first attempt:
conda-forge/staged-recipes#15717

@soxofaan could you please have a look and provide feedback? Would you like to be included as a conda package maintaner?

Best regards,
Sebastian

[jdries]

Hi @sebastian-luna-valero , thanks this is very helpfull, as we don't have experience with this ourselves!
Stefaan is on holiday, so he may not reply for a few weeks.
I had a look at the recipe, looks fine for me. You can add me as a maintainer as well.

[sebastian-luna-valero]

Thanks @jdries

Great, I will add you as a maintainer as well.

I have got pytest working on Linux only. Could you have a look at the errors for OSX and Windows here and let me know your thoughts, please?

Best regards,
Sebastian

[jdries]

Hi Sebastian,
seems much needed that we also have some windows and OSX testing.
I'm afraid these point to some actual issues, and seem to remember that they were not super easy to solve.
Is it ok if my colleague Stefaan has a look at this in august?
Otherwise, we may need to come up with some temporary workaround...

[sebastian-luna-valero]

Hi Jeroen,

I am happy to wait until Stefaan can have a look.

It is also possible to just build and release the conda package for Linux, and forget about OSX and Windows for now.

Best regards,
Sebastian

[sebastian-luna-valero]

Hi,

FYI: I will be on leave the last two weeks of August, so I will be available to help with the conda package either next week or after 31st of August.

Best regards,
Sebastian

[soxofaan]

Hi @sebastian-luna-valero thanks for bootstrapping this! I already had a quick look at conda-forge/staged-recipes#15717

I also did a minor tweak that should hopefully fix the failing tests on osx

[sebastian-luna-valero]

Many thanks @soxofaan

I am now updating the recipe. Let's continue the discussion over there, please.

[sebastian-luna-valero]

Hi @soxofaan

Do you need more time before we continue working on conda-forge/staged-recipes#15717? Or can I add h5netcdf back to the list of runtime dependencies?

Best regards,
Sebastian

[soxofaan]

hi @sebastian-luna-valero sorry for the delay.
As noted in conda-forge/staged-recipes#15717 it would be interested to know how you trigger that h5netcdf runtime issue. (We might have to fix that in python client itself, not only the conda recipe)

[sebastian-luna-valero]

Hi @soxofaan

No worries. I added my comments to the conda-forge PR.

[soxofaan]

FYI: under #198 and #351 , windows support was improved and now the test suite should run fully on windows too

[sebastian-luna-valero]

Thanks, following up discussion on conda-forge/staged-recipes#15717

[soxofaan]

Ok it seems openeo is now available for conda (forge): https://anaconda.org/conda-forge/openeo

I just tried to install it in a fresh conda env (with python 3.9) and it gave this package plan:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    brotlipy-0.7.0             |py39hb9d737c_1004         342 KB  conda-forge
    ca-certificates-2022.12.7  |       ha878542_0         143 KB  conda-forge
    certifi-2022.12.7          |     pyhd8ed1ab_0         147 KB  conda-forge
    cffi-1.15.0                |   py39h4bc2ebd_0         227 KB  conda-forge
    charset-normalizer-2.1.1   |     pyhd8ed1ab_0          36 KB  conda-forge
    cryptography-3.4.8         |   py39h95dcef6_1         1.1 MB  conda-forge
    deprecated-1.2.13          |     pyh6c4a22f_0          12 KB  conda-forge
    geos-3.10.2                |       h9c3ff4c_0         1.6 MB  conda-forge
    idna-3.4                   |     pyhd8ed1ab_0          55 KB  conda-forge
    mkl-service-2.4.0          |   py39h7e14d7c_0          61 KB  conda-forge
    mkl_fft-1.3.1              |   py39h0c7bc48_1         213 KB  conda-forge
    mkl_random-1.2.2           |   py39hde0f152_0         379 KB  conda-forge
    numexpr-2.8.4              |   py39he184ba9_0         133 KB
    numpy-1.23.5               |   py39h14f4228_0          10 KB
    numpy-base-1.23.5          |   py39h31eccc5_0         6.7 MB
    openeo-0.14.1              |     pyhd8ed1ab_0         227 KB  conda-forge
    packaging-23.0             |     pyhd8ed1ab_0          40 KB  conda-forge
    pandas-1.5.2               |   py39h417a72b_0        11.9 MB
    pycparser-2.21             |     pyhd8ed1ab_0         100 KB  conda-forge
    pyopenssl-20.0.1           |     pyhd8ed1ab_0          48 KB  conda-forge
    pysocks-1.7.1              |     pyha2e5f31_6          19 KB  conda-forge
    python-dateutil-2.8.2      |     pyhd8ed1ab_0         240 KB  conda-forge
    python_abi-3.9             |           2_cp39           4 KB  conda-forge
    pytz-2022.7.1              |     pyhd8ed1ab_0         182 KB  conda-forge
    requests-2.28.2            |     pyhd8ed1ab_0          55 KB  conda-forge
    shapely-1.8.1              |   py39h73b9895_0         361 KB  conda-forge
    six-1.16.0                 |     pyh6c4a22f_0          14 KB  conda-forge
    urllib3-1.26.14            |     pyhd8ed1ab_0         110 KB  conda-forge
    wrapt-1.14.1               |   py39hb9d737c_0          50 KB  conda-forge
    xarray-2023.2.0            |     pyhd8ed1ab_0         648 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        25.0 MB

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl None
  bottleneck         pkgs/main/linux-64::bottleneck-1.3.5-py39h7deecbd_0 None
  brotlipy           conda-forge/linux-64::brotlipy-0.7.0-py39hb9d737c_1004 None
  cffi               conda-forge/linux-64::cffi-1.15.0-py39h4bc2ebd_0 None
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 None
  cryptography       conda-forge/linux-64::cryptography-3.4.8-py39h95dcef6_1 None
  deprecated         conda-forge/noarch::deprecated-1.2.13-pyh6c4a22f_0 None
  geos               conda-forge/linux-64::geos-3.10.2-h9c3ff4c_0 None
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 None
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561 None
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640 None
  mkl-service        conda-forge/linux-64::mkl-service-2.4.0-py39h7e14d7c_0 None
  mkl_fft            conda-forge/linux-64::mkl_fft-1.3.1-py39h0c7bc48_1 None
  mkl_random         conda-forge/linux-64::mkl_random-1.2.2-py39hde0f152_0 None
  numexpr            pkgs/main/linux-64::numexpr-2.8.4-py39he184ba9_0 None
  numpy              pkgs/main/linux-64::numpy-1.23.5-py39h14f4228_0 None
  numpy-base         pkgs/main/linux-64::numpy-base-1.23.5-py39h31eccc5_0 None
  openeo             conda-forge/noarch::openeo-0.14.1-pyhd8ed1ab_0 None
  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 None
  pandas             pkgs/main/linux-64::pandas-1.5.2-py39h417a72b_0 None
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 None
  pyopenssl          conda-forge/noarch::pyopenssl-20.0.1-pyhd8ed1ab_0 None
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 None
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0 None
  python_abi         conda-forge/linux-64::python_abi-3.9-2_cp39 None
  pytz               conda-forge/noarch::pytz-2022.7.1-pyhd8ed1ab_0 None
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 None
  shapely            conda-forge/linux-64::shapely-1.8.1-py39h73b9895_0 None
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0 None
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 None
  wrapt              conda-forge/linux-64::wrapt-1.14.1-py39hb9d737c_0 None
  xarray             conda-forge/noarch::xarray-2023.2.0-pyhd8ed1ab_0 None

This looks quite a bit more than what a standard pip install drags in:

$ docker run --rm -it python /bin/bash
root@9d6f5489489a:/# pip install openeo
  Downloading openeo-0.14.1-py3-none-any.whl (218 kB) 
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
  Downloading shapely-2.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
  Downloading numpy-1.24.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
  Downloading xarray-2023.2.0-py3-none-any.whl (975 kB)
  Downloading pandas-1.5.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB)
  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
  Downloading wrapt-1.14.1.tar.gz (50 kB)
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
  Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
  Downloading charset_normalizer-3.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (196 kB)
  Downloading idna-3.4-py3-none-any.whl (61 kB)
  Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
  Downloading packaging-23.0-py3-none-any.whl (42 kB)
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)

@sebastian-luna-valero could it be that the dependency listing in the conda recipy needs a bit more fine-tuning or pruning?
Note that it is an explicit goal of the openeo package to be as light as possible dependency-wise

[sebastian-luna-valero]

Hi @soxofaan

I see. However, what we added to the conda recipe:
https://github.com/conda-forge/openeo-feedstock/blob/main/recipe/meta.yaml#L25

Is equal to what we have in:
https://github.com/Open-EO/openeo-python-client/blob/master/setup.py#L54

So the differences are due to the dependency resolution made by conda/mamba vs pip.

What would you modify in the conda recipe then?

[soxofaan]

@sebastian-luna-valero
(FYI: I didn't have time to dive into the details yet, I just wanted to quickly mention this here.)
If I understand correctly these additional dependencies are probably secondary side-effects of depending on the conda variants of numpy/pandas/shapely/xarray. Not much we can do then on the level of the openeo recipe.

Another related question:

... the conda recipe: https://github.com/conda-forge/openeo-feedstock/blob/main/recipe/meta.yaml#L25
... what we have in setup.py: https://github.com/Open-EO/openeo-python-client/blob/master/setup.py#L54

Do you know approaches or tools to keep these things in sync? Yesterday for example, I added a new test dependency, which eventually should be added to the conda recipy too.

Thanks by the way for pushing this ticket, and apologies it took so long to get all the kinks out.

[sebastian-luna-valero]

Do you know approaches or tools to keep these things in sync?

Sorry, I don't know.

Thanks by the way for pushing this ticket, and apologies it took so long to get all the kinks out.

No worries, and thanks for your support!

[soxofaan]

👍

I think we can close this ticket for now.
Thanks again for all the work

[soxofaan]

@sebastian-luna-valero I just released version 0.15.0 of python client. Can you walk me through the steps required to also get that release in conda-forge?

[sebastian-luna-valero]

Hi @soxofaan

conda-forge automatically opens a PR when a new version of the source code is available: conda-forge/openeo-feedstock#1

Since you are listed as a recipe maintainer you are also notified. It's now up to us to review and merge that PR.

Once the PR is merged, the new conda package will be created and released via the conda-forge channel.