pypi package

[louisabraham]

Fix #75

I uploaded a test package to https://test.pypi.org/project/hnswlib/

You can already test it using

pip uninstall hnswlib
pip install -i https://test.pypi.org/simple/ hnswlib

To upload yourself (@yurymalkov) a version to regular pypi:

1. pip install twine
2. Create an account on https://pypi.org/ and verify the email address
3. Create ~/pypirc and put:

[distutils]
index-servers =
  pypi


[pypi]
repository: https://upload.pypi.org/legacy/
username: YOUR_USERNAME

4. cd python_bindings
5. make pypi

After that, people will be able to install it simply with pip install hnswlib. If people still want to install it from the python_bindings folder, they have to cp -r ../hnswlib . before (see travil build).

[yurymalkov]

@louisabraham Many thanks! I'll check it and merge.

[yurymalkov]

Hi @louisabraham ,
I've tried installing it, but it does not seem to work.
It produces an error:

  ERROR: Command errored out with exit status 1:
   command: /home/yurymal/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fex3or7w/hnswlib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fex3or7w/hnswlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-a_5wulmy --python-tag cp36
       cwd: /tmp/pip-install-fex3or7w/hnswlib/
  Complete output (20 lines):
  running bdist_wheel
  running build
  running build_ext
  creating tmp
  gcc -pthread -B /home/yurymal/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yurymal/anaconda3/include/python3.6m -c /tmp/tmpvvimu_px.cpp -o tmp/tmpvvimu_px.o -std=c++14
  gcc: error: unrecognized command line option ‘-std=c++14’
  gcc -pthread -B /home/yurymal/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yurymal/anaconda3/include/python3.6m -c /tmp/tmpeg8chpp3.cpp -o tmp/tmpeg8chpp3.o -std=c++11
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  gcc -pthread -B /home/yurymal/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yurymal/anaconda3/include/python3.6m -c /tmp/tmpl_zllj98.cpp -o tmp/tmpl_zllj98.o -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  building 'hnswlib' extension
  creating build
  creating build/temp.linux-x86_64-3.6
  gcc -pthread -B /home/yurymal/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ihnswlib -I/home/yurymal/anaconda3/include/python3.6m -I/home/yurymal/.local/include/python3.6m -I/home/yurymal/anaconda3/lib/python3.6/site-packages/numpy/core/include -I/home/yurymal/anaconda3/include/python3.6m -c bindings.cpp -o build/temp.linux-x86_64-3.6/bindings.o -O3 -march=native -fopenmp -DVERSION_INFO="0.3" -std=c++11 -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  bindings.cpp:5:29: fatal error: hnswlib/hnswlib.h: Нет такого файла или каталога
   #include "hnswlib/hnswlib.h"
                               ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for hnswlib

It seems that the wheel (https://test-files.pythonhosted.org/packages/81/9f/dba846da717f439f8a85ef1ed9ad31f0847a74017fa46fbc4f7efbec33e1/hnswlib-0.3.tar.gz) does not contain the header files with the library, which leads to this problem.
Can you look at it?

[louisabraham]

Actually, the wheel (precompiled) isn't supposed to contained the headers, but the source distribution is.
On my computer it worked precisely because it used the wheels.

I believe it should be fixed now.

Can you try again?

pip install -i https://test.pypi.org/simple/ hnswlib

I uploaded it as version 3.1

[yurymalkov]

Hi @louisabraham,
Many thanks, now it pip install -i https://test.pypi.org/simple/ hnswlib works, both from Linux and Windows!

However, when I run make pypi, it ends with error:
HTTPError: 400 Client Error: Binary wheel 'hnswlib-0.3-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/
Is it because of the bdist_wheel in setup.py?

Also, I wonder, will it work if a soft link from ../hnswlib to './hnswlib' is created instead of copying the directory?

[louisabraham]

Hi,

Is it because of the bdist_wheel in setup.py?

Indeed, the problem is that pypi doesn't accept linux wheels that are not [manylinux](https://github.com/pypa/manylinux). I'm not sure about the performance implications of such precompilation so it is probably best to only distribute the sources. It should be fixed in my latest commit.

Also, I wonder, will it work if a soft link from ../hnswlib to './hnswlib' is created instead of copying the directory?

When I tested on my computer (macOS), it worked. I don't know if this would work on other file systems. I don't think git handles well symlink in a cross-platform way so the symlink would still have to be created when building the extension. Since you are probably going to be the only one to push to pypi, you can do what you want. You don't even have to put the Makefile online. Le 22 août 2019 à 9:21 PM +0200, Yury Malkov <[email protected]>, a écrit :

…

Hi @louisabraham, Many thanks, now it pip install -i https://test.pypi.org/simple/ hnswlib works, both from Linux and Windows! However, when I run make pypi, it ends with error: HTTPError: 400 Client Error: Binary wheel 'hnswlib-0.3-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/ Is it because of the bdist_wheel in setup.py? Also, I wonder, will it work if a soft link from ../hnswlib to './hnswlib' is created instead of copying the directory? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[yurymalkov]

Great! I've uploaded the library and it works! Thank you!

Can you please redo the PR against the develop branch? I'll merge it right away.

It seems that git supports symlink, as done for nmslib (https://github.com/nmslib/nmslib/blob/master/python_bindings/nmslib), I can test It for this purpose.

[louisabraham]

The use symlink commit failed tests but it has nothing to do with my modifications (see #142)

[yurymalkov]

Thank you so much!