Skip to content

Error in framework with latest pytest-xdist:2.0 #583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pranav165 opened this issue Aug 15, 2020 · 6 comments
Closed

Error in framework with latest pytest-xdist:2.0 #583

pranav165 opened this issue Aug 15, 2020 · 6 comments

Comments

@pranav165
Copy link

pranav165 commented Aug 15, 2020 •
edited by nicoddemus
Loading

When trying to build the docker image with latest pytest-xdist , following error is produced. The issue is resolved when reverting to pytest-xdist:1.34.0

Error 
 Running setup.py install for psutil: started
[17:05:25]    Running setup.py install for psutil: finished with status 'error'
[17:05:25]    Complete output from command /home/e2e/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-opgz1ts0/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-6io9j8f9/install-record.txt --single-version-externally-managed --compile --install-headers /home/e2e/venv/include/site/python3.7/psutil:
[17:05:25]    running install
[17:05:25]    running build
[17:05:25]    running build_py
[17:05:25]    creating build
[17:05:25]    creating build/lib.linux-x86_64-3.7
[17:05:25]    creating build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_compat.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_common.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_psposix.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/__init__.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_psaix.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    copying psutil/_psosx.py -> build/lib.linux-x86_64-3.7/psutil
[17:05:25]    creating build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    copying psutil/tests/runner.py -> build/lib.linux-x86_64-3.7/psutil/tests
[17:05:25]    running build_ext
[17:05:25]    building 'psutil._psutil_linux' extension
[17:05:25]    creating build/temp.linux-x86_64-3.7
[17:05:25]    creating build/temp.linux-x86_64-3.7/psutil
[17:05:25]    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/home/e2e/venv/include -I/usr/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
[17:05:25]    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
[17:05:25]    C compiler or Python headers are not installed on this system. Try to run:
[17:05:25]    sudo apt-get install gcc python3-dev
[17:05:25]    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

DockerFile

FROM debian:latest
RUN   set -e \
      && export  DEBIAN_FRONTEND=noninteractive \
      && apt-get update \
      && apt-get -qq install --no-install-recommends \
         python3-dev \
         python3-venv \
      && useradd -m -u 1000 e2e \
      && mkdir /reports \
      && chmod a+rwx /reports \
      && chown e2e:e2e /reports
VOLUME ["/reports"]
COPY entrypoint.sh /usr/local/bin/entrypoint
RUN chmod 755 /usr/local/bin/entrypoint
ENTRYPOINT [ "/usr/local/bin/entrypoint" ]
COPY . /home/e2e/
RUN chown -R e2e:e2e /home/e2e/
RUN chmod -R 0777 /home/e2e/
USER e2e
WORKDIR /home/e2e
RUN set -e \
    && python3 -m venv venv \
    && . venv/bin/activate \
    && pip install --no-cache-dir wheel \
    && pip install --no-cache-dir .\
    && flake8 .

Setup.py

#!/usr/bin/env python

VERSION = '###'
DESCRIPTION = '###'

try:
    from setuptools import setup, find_packages
except ImportError:
    from distutils.core import setup, find_packages

setup(
    name='e2e',
    version=VERSION,
    description=DESCRIPTION,
    license="###",
    long_description="###",
    url='###',
    author="###",
    author_email='###',
    install_requires=[
        'selenium',
        'requests',
        'pytest',
        'nested-lookup==0.2.21',
        'pyhamcrest',
        'retry',
        'pytest-html',
        'pytest-xdist',
        'pytest-rerunfailures',
        'pytest-repeat',
        'wheel',
        'flake8'
    ],
    provides=[],
    packages=find_packages(),
    scripts=[]
)
@nicoddemus
Copy link
Member

Hi @pranav165,

It seems your Docker container is failing to build psutil, a new pytest-xdist dependency in 2.0.

[17:05:25]    running build_ext
[17:05:25]    building 'psutil._psutil_linux' extension
[17:05:25]    creating build/temp.linux-x86_64-3.7
[17:05:25]    creating build/temp.linux-x86_64-3.7/psutil

@pranav165
Copy link
Author

Yes. It also works fine with latest pytest-xdist :2.0 when i use base image as python:3 instead of debian:latest.

So i have workarounds but just not sure why this error.

@nicoddemus
Copy link
Member

Your previous image doesn't seem to have a compiler:

[17:05:25]    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/home/e2e/venv/include -I/usr/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
[17:05:25]    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory

@nicoddemus
Copy link
Member

I'm closing this for now, but feel free to follow up with further questions. 👍

@wrcyyy
Copy link

wrcyyy commented Aug 17, 2020

Similar problem: Ubuntu 20.04 lts, python3.8.2,pip 20.0.2:

FROM ubuntu:20.04
RUN apt-get update \
    && apt-get install -y --no-install-recommends python3 python3-pip \
    && pip3 install --no-cache-dir pytest-xdist

ERROR INFO:


Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py): started
  Building wheel for psutil (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-t421esoa
       cwd: /tmp/pip-install-jqd23vi1/psutil/
  Complete output (44 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/psutil
  copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/__init__.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_psosx.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_psaix.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_common.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_psposix.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.8/psutil
  copying psutil/_compat.py -> build/lib.linux-x86_64-3.8/psutil
  creating build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/runner.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.8/psutil/tests
  copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-3.8/psutil/tests
  running build_ext
  building 'psutil._psutil_linux' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/psutil
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/include/python3.8 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.8/psutil/_psutil_common.o
  unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
  C compiler or Python headers are not installed on this system. Try to run:
  sudo apt-get install gcc python3-dev
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for psutil
  Running setup.py clean for psutil
Failed to build psutil
Installing collected packages: apipkg, execnet, iniconfig, pluggy, more-itertools, attrs, toml, six, pyparsing, packaging, py, pytest, psutil, pytest-forked, pytest-xdist
    Running setup.py install for psutil: started
    Running setup.py install for psutil: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zn5qzuxa/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/psutil
         cwd: /tmp/pip-install-jqd23vi1/psutil/
    Complete output (44 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/__init__.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psosx.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psaix.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_common.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psposix.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_compat.py -> build/lib.linux-x86_64-3.8/psutil
    creating build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/runner.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-3.8/psutil/tests
    running build_ext
    building 'psutil._psutil_linux' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/psutil
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/include/python3.8 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.8/psutil/_psutil_common.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    C compiler or Python headers are not installed on this system. Try to run:
    sudo apt-get install gcc python3-dev
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jqd23vi1/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zn5qzuxa/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/psutil Check the logs for full command output.
The command '/bin/sh -c apt-get update     && apt-get install -y --no-install-recommends python3 python3-pip     && pip3 install --no-cache-dir pytest-xdist' returned a non-zero code: 1

@nicoddemus nicoddemus mentioned this issue Aug 17, 2020
Closed
@simonjayhawkins simonjayhawkins mentioned this issue Aug 19, 2020
Closed
@nicoddemus
Copy link
Member

2.1.0 will drop the hard dependency to psutil: #591

This was referenced Oct 7, 2020
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nicoddemus @pranav165 @wrcyyy