Parseable error message for missing OS/non-pip dependencies?

[oxwivi]

• Pip version: pip 18.1 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
• Python version: Python 2.7.15+
• Operating system: Linux rpi3 4.19.0-1-arm64 #1 SMP Debian 4.19.12-1 (2018-12-22) aarch64 GNU/Linux

For context, I'm not a Python developer, I'm a user of a certain Python app that uses pip to manage its Python dependencies. And I am installing the aforementioned piece of Python software on a deboostrap'd system, which brings me to my pain point: my environment is, as any debootrap user can tell you, very, very bare bones.

And as you might expect, every so often my pip package installation fails, because either itself or one of its many dependencies cannot find libraries and dependencies they expect on the system.

The situation is exacerbated by very bad error messages:

• The main installation stdout fails with referring to the pip installation failure that stopped it, with nary a mention of why:

$ bench init frappe
INFO:bench.utils:virtualenv -q env -p /usr/bin/python
Already using interpreter /usr/bin/python
INFO:bench.utils:env/bin/pip -q install --upgrade pip
INFO:bench.utils:env/bin/pip -q install wheel
INFO:bench.utils:env/bin/pip -q install six
INFO:bench.utils:env/bin/pip -q install -e git+https://github.com/frappe/python-pdfkit.git#egg=pdfkit
INFO:bench.app:getting app frappe
INFO:bench.utils:git clone https://github.com/frappe/frappe.git   --origin upstream
Cloning into 'frappe'...
remote: Enumerating objects: 52, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 160689 (delta 11), reused 16 (delta 3), pack-reused 160637
Receiving objects: 100% (160689/160689), 178.98 MiB | 1.21 MiB/s, done.
Resolving deltas: 100% (122269/122269), done.
Checking out files: 100% (2291/2291), done.
('installing', u'frappe')
INFO:bench.app:installing frappe
INFO:bench.utils:frappe/env/bin/pip install -q  -e frappe/apps/frappe 
  Failed building wheel for Pillow
  Failed building wheel for lxml
Command "/home/frappe/frappe/env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-AneHUg/Pillow/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-qQ63Y3/install-record.txt --single-version-externally-managed --compile --install-headers /home/frappe/frappe/env/include/site/python2.7/Pillow" failed with error code 1 in /tmp/pip-install-AneHUg/Pillow/                                      
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/dist-packages/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bench/commands/make.py", line 30, in init
    python               = python)
  File "/usr/local/lib/python2.7/dist-packages/bench/utils.py", line 75, in init
    get_app(frappe_path, branch=frappe_branch, bench_path=path, build_asset_files=False, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/bench/app.py", line 135, in get_app
    install_app(app=app_name, bench_path=bench_path, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/bench/app.py", line 179, in install_app
    find_links=find_links))
  File "/usr/local/lib/python2.7/dist-packages/bench/utils.py", line 159, in exec_cmd
    raise CommandFailedError(cmd)
bench.utils.CommandFailedError: frappe/env/bin/pip install -q  -e frappe/apps/frappe 

• In the above example, two pip packages were simultaneously mentioned, other times only one, causing me to run through the process after fixing a single dependency issue each time.
• The only way to debug what dependency are missing are my manually running pip install for each packages, and finally I get some verbose error message that shows me what's missing, but it only displays only the first missing dependency the installation process found, leading me to resolve one, then run the command again for the next one and so on and so forth.

In the end, I simply resorted to looking for installation documentation of each of the packages and installing distro-specific packages as noted by them. Even then, it didn't end my problems because I'm on sid and its repo did not carry the exact packages as specified in the docs, so I had to workaround by doing apt build-dep python-lxml python-pil $etc (There's no python-pillow or similar in Debian repos, perhaps there's some story, IDK), and finally get around to getting my actual target to install and compile the pip packages as it saw fit.

This ended being a sort of venting post, please overlook it; I just want to know if there's anyway for pip to check and output OS dependencies of every packages that I may programmatically parse and look up for corresponding packages using my distro's package manager of choice?

If there isn't please feel free to tag this post as feature request.

[oxwivi]

... Just now, even further down the process, I was just faced with this: /home/frappe/frappe/env/bin/python: libjpeg.so.9: cannot open shared object file: No such file or directory

I am on the verge for feeling physical pain, as I've to remove the setup so far that literally continued for at least an hour and half and restart again, and I don't know if there's going to be other errors like this.

[pfmoore]

This ended being a sort of venting post, please overlook it; I just want to know if there's anyway for pip to check and output OS dependencies of every packages that I may programmatically parse and look up for corresponding packages using my distro's package manager of choice?

No problem - it's definitely something painful (and particularly if you're not using pip directly, but it's being used behind the scenes by a tool you're using). I'd argue at least in part that the tool that calls pip should be catching the errors, and presenting them to you in a more helpful way, and not just dumping pip's error output on you (after all, pip is a tool for Python developers, so its output assumes a set of skills and knowledge that you don't have).

Having said that, though, the errors you are seeing are not technically from pip. They are produced by a subprocess that pip runs to "build a wheel" (a "wheel" is the binary artifact that pip actually installs). Of course, the arguments above apply just as well to pip - we shouldn't just dump the build tool's output on our users, we should present it more helpfully - but there's a lot of variation in the build steps and their output, and in our defence, we expect our users to know something about how Python packages are built, so it's not as bad for them as it is for you.

None of which helps you much, though, I'm afraid. The best I can suggest is that you need to get in touch with the projects whose installs are failing - that's visible in the output as

Failed building wheel for Pillow
Failed building wheel for lxml

and ask them to either help diagnose the output, or as a feature request to them, ask them to improve their build scripts to give better error output.

We have considered improving the output in the past, but we've never really found any good solution that doesn't risk hiding vital diagnostic information in at least some cases.

[oxwivi]

Getting the projects to output better error messages will only help the second point I raised:

[...] two pip packages were simultaneously mentioned, other times only one, causing me to run through the process after fixing a single dependency issue each time.

But that does not help me in my ultimate goal of just getting to know what kind of libraries and files the pip package, wheel process are expecting on the underlying system. Like the libjpeg.so.9 that Pillow needed, causing the whole installation to stop midway.

Ideally, pip packagers will mention these as dependencies on requirements.txt so that pip will look for them on $PATH, and either inform the preceding process of these files' absence, write to stdout or do both.

If there's no such existing method, please add feature request label to this issue.

[pfmoore]

Pip doesn't handle non-Python dependencies, sorry. If you want functionality like that, maybe conda is a better fit for your requirements

[oxwivi]

I'll go check out what conda is, but as I was saying, I just want packages to list their non-pip dependencies so that the user may act upon it. I'm not asking pip to handle them.

[pfmoore]

I'll go check out what conda is

I just remembered, you're not calling pip directly but a tool you're using does that for you. So I guess conda won't help you much (it's an alternative to pip).

I just want packages to list their non-pip dependencies so that the user may act upon it.

Well, that was the point I made about asking those packages. Also, if the packages provide a list of their dependencies in machine readable form, maybe the tool you're using could check that the dependencies exist before calling pip. I don't really know, but either way it's not really something for pip but rather for other tools.

[chrahunt]

Hi @oxwivi! I hope things have been easier for you since this issue was created.

One action I think we can take away from this is the need to make it clear (above and below the output) which package failed to install, and that you should be following up with that package author. We're tracking just that in #6959.

As mentioned above, outputting non-Python dependencies is something that should be brought up to package authors. I know several packages that do checks in their setup.py and fail with a helpful error message in case something isn't installed.

I hope that helps or gives some feeling that this will at least get easier to manage in the future. Since we're already tracking the followup activity in the other issue I will close this one. Thanks!