Lots of repositories cause "Argument list too long" for py_binary

[pauldraper]

Problem

The Python wrapper is finding every repository in the runfiles tree, creating a huge PYTHONPATH variable, and then crashing when it exceeds the system's env var limits.

I encountered this on Ubuntu 20.04, Python 3.8.

OSError: [Errno 7] Argument list too long: '/root/.cache/bazel/_bazel_root/70833c7af31404a35dcc581d457bd519/execroot/rivethealth_rivet/bazel-out/k8-fastbuild/bin/api/test_it/bin.runfiles/bazel_tools/tools/python/py3wrapper.sh'

A large number (hundreds) of external repositories is quite possible with Maven and npm projects. In my case, I have utility tools written in Python used in the runtime tree of a Node.js binary (that pulls in npm repositories).

Solution

Ideally, py_binary would only add Python dependencies to the PYTHONPATH. Besides prevent a crash, it's also more efficient for loading modules.

Workarounds

import_all

An undocumented option --experimental_python_import_all_repositories=false prevents automatically adding these repositories.

However, it requires specifying the imports attribute, at least in some executions contexts, and of course the general Bazel ecosystem (e.g. rules_docker) doesn't do that.

usercustomize

Use usercustomize to change PYTHONPATH when execv is run.

~/.local/lib/python3.8/site-packages/usercustomize.py

import os

execv = os.execv

def _execv(*args, **kwargs):
    try:
        pythonpath = os.environ["PYTHONPATH"]
    except KeyError:
        pass
    # remove unnecessary entries from pythonpath
    # ...
    os.environ["PYTHONPATH"] = pythonpath
    return execv(*args, **kwargs)

os.execv = _execv

[hrfuller]

This can also happen when the $TMPDIR location is long so that the absolute paths to the import locations are long. Even with only python repos on the PYTHONPATH you can exceed this limit.

[jbedorf]

This can also happen when you are using the py_test rule when using hermetic Python for the same reason as in the previous post.

[arunkant]

So what would be solution for this?

[hrfuller]

The long term solution is to use the site packages import mechanism for external repos instead of PYTHONPATH.

[pauldraper]

So what would be solution for this?

@arunkant I included two workarounds in my bug report: #14640 (comment)

The first is better than the second, unless you are using third-party Python targets, e.g. in rules_docker.

[arunkant]

Any workaround for rules_docker?

[arunkant]

The long term solution is to use the site packages import mechanism for external repos instead of PYTHONPATH.

How to do that? @hrfuller any docs or config I can use. We are migrating our codebase to monorepo and consolidate all python deps at one location and I don't want all that effort to be blocked by this issue

[pauldraper]

@arunkant in that case, see my second workaround: use usercustomize.py. Monkeypatch os.execv to remove unwanted entries from environ["PYTHONPATH"].

[arunkant]

Thanks @pauldraper. That looks reasonable workaround. I'll try it