Version mismatch causes pip warning and no install

[djhoese]

In my CI build I have:

          python -m pip install \
          --extra-index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ \
          --trusted-host pypi.anaconda.org \
          --no-deps --pre --upgrade \
          matplotlib \
          numpy \
          pandas \
          scipy;

which produces the following:

Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/scipy-wheels-nightly/simple/
Requirement already satisfied: matplotlib in /usr/share/miniconda3/envs/test-environment/lib/python3.8/site-packages (3.3.4)
Collecting matplotlib
  Downloading matplotlib-3.4.0rc1-cp38-cp38-manylinux1_x86_64.whl (10.3 MB)
Requirement already satisfied: numpy in /usr/share/miniconda3/envs/test-environment/lib/python3.8/site-packages (1.20.1)
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2B821.g572d5a1da/numpy-1.21.0.dev0%2B821.g572d5a1da-cp38-cp38-manylinux2010_x86_64.whl (15.5 MB)
Requirement already satisfied: pandas in /usr/share/miniconda3/envs/test-environment/lib/python3.8/site-packages (1.2.2)
Collecting pandas
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/pandas/1.3.0.dev0%2B849.gab687aec38/pandas-1.3.0.dev0%2B849.gab687aec38-cp38-cp38-manylinux1_x86_64.whl (19.1 MB)
Requirement already satisfied: scipy in /usr/share/miniconda3/envs/test-environment/lib/python3.8/site-packages (1.6.0)
Collecting scipy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B5e9eb01/scipy-1.7.0.dev0%2B20210110040759_5e9eb01-cp38-cp38-manylinux1_x86_64.whl (27.3 MB)
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B5e9eb01/scipy-1.7.0.dev0%2B20210110040759_5e9eb01-cp38-cp38-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B5e9eb01/scipy-1.7.0.dev0%2B20210110040759_5e9eb01-cp38-cp38-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20210110040759.5e9eb01', but metadata has '1.7.0.dev0+5e9eb01'
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B9b9f2e8/scipy-1.7.0.dev0%2B20210103040256_9b9f2e8-cp38-cp38-manylinux1_x86_64.whl (27.3 MB)
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B9b9f2e8/scipy-1.7.0.dev0%2B20210103040256_9b9f2e8-cp38-cp38-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B9b9f2e8/scipy-1.7.0.dev0%2B20210103040256_9b9f2e8-cp38-cp38-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20210103040256.9b9f2e8', but metadata has '1.7.0.dev0+9b9f2e8'
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2Bb21b303/scipy-1.7.0.dev0%2B20201227035541_b21b303-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2Bb21b303/scipy-1.7.0.dev0%2B20201227035541_b21b303-cp38-cp38-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2Bb21b303/scipy-1.7.0.dev0%2B20201227035541_b21b303-cp38-cp38-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20201227035541.b21b303', but metadata has '1.7.0.dev0+b21b303'
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B3cc6834/scipy-1.7.0.dev0%2B20201220035335_3cc6834-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B3cc6834/scipy-1.7.0.dev0%2B20201220035335_3cc6834-cp38-cp38-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B3cc6834/scipy-1.7.0.dev0%2B20201220035335_3cc6834-cp38-cp38-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20201220035335.3cc6834', but metadata has '1.7.0.dev0+3cc6834'
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B6148b7c/scipy-1.7.0.dev0%2B20201213035410_6148b7c-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B6148b7c/scipy-1.7.0.dev0%2B20201213035410_6148b7c-cp38-cp38-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B6148b7c/scipy-1.7.0.dev0%2B20201213035410_6148b7c-cp38-cp38-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20201213035410.6148b7c', but metadata has '1.7.0.dev0+6148b7c'
  Downloading scipy-1.6.1-cp38-cp38-manylinux1_x86_64.whl (27.3 MB)
Installing collected packages: scipy, pandas, numpy, matplotlib
  Attempting uninstall: scipy
    Found existing installation: scipy 1.6.0
    Uninstalling scipy-1.6.0:
      Successfully uninstalled scipy-1.6.0

So the other deps are installed but scipy seems to have a version mismatch which causes pip to never install the unstable version. I've also tried --index-url instead of --extra-index-url but the same result. Any ideas?

See corteva/rioxarray#257 and pytroll/satpy#1578

CC @snowman2

[rgommers]

Looks like that's the same issue as scipy/scipy#13196 and/or scipy/scipy#13609. Using pip's legacy resolver is a workaround.

The fix was merged 2 days ago, so I hope the next wheel is fine. If not, we may be building with a numpy version that's still problematic.

[keewis]

this looks more like scipy/scipy#13540: the upload for py38 64bit manylinux fails because of a failing test. Using the legacy resolver (or pip=20.2) works but the installed wheel is from Jan 10.

[keewis]

this looks more like scipy/scipy#13540

Even though that's correct for the initial issue, it looks like upgrading to py39 does not fix this:

  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B83205b1/scipy-1.7.0.dev0%2B20210307035424_83205b1-cp39-cp39-manylinux1_x86_64.whl (28.1 MB)
     |████████████████████████████████| 28.1 MB 80 kB/s 
WARNING: Discarding https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B83205b1/scipy-1.7.0.dev0%2B20210307035424_83205b1-cp39-cp39-manylinux1_x86_64.whl (from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/). Requested scipy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B83205b1/scipy-1.7.0.dev0%2B20210307035424_83205b1-cp39-cp39-manylinux1_x86_64.whl has inconsistent version: filename has '1.7.0.dev0+20210307035424.83205b1', but metadata has '1.7.0.dev0+83205b1'

which means that even versions with the fix (in this case the one from 2021-03-07) are not installable with newer versions of pip (the output above is for pip=21.0.1).

[rgommers]

Looks like we should adopt the same nightly wheel naming scheme as numpy. Compare from https://anaconda.org/scipy-wheels-nightly/numpy/files:

numpy-1.21.0.dev0+1229.g36eb76c9d-cp37-cp37m-manylinux1_x86_64.whl

to those from https://anaconda.org/scipy-wheels-nightly/scipy/files:

scipy-1.7.0.dev0+20210307040256_83205b1-cp37-cp37m-manylinux1_i686.whl

The item right after .dev0+ is regularly incrementing, so pip is able to find the latest wheel.

We should just make this change with relying on versioneer like numpy did, because that has been nothing but trouble.

[charris]

Two things to watch out for using versioneer:

• Make sure the old version macros are defined to maintain CI compatibility with least effort.
  That also means updating the old version.py file (make it static) to use the new _version.py file.
• I had to patch versioneer so dirty was not part of the name. That was because the license
  files were added without a commit.
• You will need to tag the start of the master development branch with something like v1.7.0.dev0.
  You can check this with git describe, which is what versioneer uses to construct the version.
• The wheel build scripts need to stop rewriting the name with the date.

It is a bit of work to make everything go together.

[rgommers]

Sorry, I meant without relying on versioneer. I think it's a terrible and way over-engineered piece of code.

[keewis]

it does come with its own set of issues but setuptools_scm might be a bit easier to use.

[charris]

You could do a pretty good job just using git describe, but it needs the tag for the start. You would still need to fix the fetch depth for travis.

[rgommers]

it does come with its own set of issues but setuptools_scm might be a bit easier to use.

Definitely not touching that, that's even worse:) it's also responsible for still not having a sane pip install . that doesn't copy the whole world.

You could do a pretty good job just using git describe, but it needs the tag for the start. You would still need to fix the fetch depth for travis.

Exactly. The task is to append a simple incrementing integer to a version number. Should be possible to do with 20 lines of code, not with 2000 lines of code that then generate another 500 lines of code and impose extra requirements on git being available.

[charris]

If you don't care about the starting point of the count...

charris@fc [scipy.git (master)]$ git describe --tags
v0.17pre-10973-g1bf7485cb

There 10973-g1bf7485cb would work. The --tags flag is needed because there are no annotated tags reachable from the tip of the scipy master branch. A single incrementing number could be tricky if you would like it to have reasonable tracking of the repository. Versioneer will produce slightly different tags if a commit is made between wheel builds. Counting days would usually be good enough for development wheels if they are built once a day at most, which is not always the case. There is still a git dependency, but I don't think that should cause a lot of trouble.

[djhoese]

it's also responsible for still not having a sane pip install . that doesn't copy the whole world.

@rgommers if you're talking about what I think you're talking about, there is a hack around that:

try:
    # HACK: https://github.com/pypa/setuptools_scm/issues/190#issuecomment-351181286
    # Stop setuptools_scm from including all repository files
    import setuptools_scm.integration
    setuptools_scm.integration.find_files = lambda _: []
except ImportError:
    pass

[rgommers]

Just merged the fix - wheels should appear within a few hours.

[rgommers]

New wheels are up, and this works. However, the newest wheel is not the first one picked. I have asked for admin access to the bucket, but don't have it yet. @mattip or @charris, if you have admin access to https://anaconda.org/scipy-wheels-nightly/scipy/files, could you delete all nightlies for 1.7.0 and 1.6.0 from before today?

[djhoese]

@rgommers Thanks! I was just about to comment about this. Both my CI and local environments don't pick up the new wheels.

[mattip]

@rgommers I granted you owner access to the relevant anaconda.org groups.

[rgommers]

Thanks @mattip.

Fixed now, only recent wheels are left.

[djhoese]

Confirmed. My CI now grabs the new wheels:

 Collecting scipy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy/1.7.0.dev0%2B810.cfb3e80/scipy-1.7.0.dev0%2B810.cfb3e80-cp38-cp38-manylinux1_x86_64.whl (28.0 MB)