Skip to content

Allow richer expressions in -n auto #477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LivInTheLookingGlass opened this issue Oct 18, 2019 · 5 comments
Closed

Allow richer expressions in -n auto #477

LivInTheLookingGlass opened this issue Oct 18, 2019 · 5 comments

Comments

@LivInTheLookingGlass
Copy link

LivInTheLookingGlass commented Oct 18, 2019

It would be nice if we could put things like pytest -n auto-1 and have it correctly interpret that as "all CPUs except 1". This ensures that the processor is not hogged by tests, but also that it runs at a very fast clip.

Could be done something like:

cores = 1
if 'auto' in args.n:
    cores = cpu_count()
    if '-' in args.n:
        cores -= int(args.n.split('-')[1])
    elif args.n != 'auto':
        raise ValueError("Invalid option given!")
else:
    cores = int(args.n)
@RonnyPfannschmidt
Copy link
Member

im -10 on this - not going to add a mini pseudo programming language into a option

@chrahunt
Copy link

chrahunt commented Nov 8, 2019

This can be done directly in shell, no reason to implement it in xdist.

pytest -n $(($(lscpu | awk '/CPU\(s\)/ {print $2; exit}') - 1))

There's probably similar for Powershell.

@nicoddemus
Copy link
Member

I agree, closing as "won't fix"

@graingert
Copy link
Member

graingert commented Jun 23, 2020

auto_detect_cpus is now a bit more complicated:

def auto_detect_cpus():
try:
from os import sched_getaffinity
except ImportError:
if os.environ.get("TRAVIS") == "true":
# workaround https://bitbucket.org/pypy/pypy/issues/2375
return 2
try:
from os import cpu_count
except ImportError:
from multiprocessing import cpu_count
else:
def cpu_count():
return len(sched_getaffinity(0))
try:
n = cpu_count()
except NotImplementedError:
return 1
return n if n else 1

pytest --numprocesses=$(python -c 'print(__import__("pytest.plugin").plugin.auto_detect_cpus() - 1)')

@stas00
Copy link

stas00 commented Jul 14, 2020

pytest --numprocesses=$(python -c 'print(__import__('pytest.plugin').plugin.auto_detect_cpus() - 1)')

that didn't work.

here is a fixed version

# count
python -c "print(__import__('xdist.plugin').plugin.auto_detect_cpus() - 1)"
# feed that to pytest
pytest --numprocesses=$(python -c "print(__import__('xdist.plugin').plugin.auto_detect_cpus() - 1)")

nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue Aug 24, 2020
…hook

This reverts the behavior we had before psutil was a hard dependency, but
opens up the possibility of customization through the
pytest_xdist_auto_num_workers hook, making things like pytest-dev#477 possible.

Fix pytest-dev#585
nicoddemus added a commit to nicoddemus/pytest-xdist that referenced this issue Aug 25, 2020
This makes using psutil optional and opens up the possibility of customization
through the pytest_xdist_auto_num_workers hook, making things like pytest-dev#477 possible.

Fix pytest-dev#585
toofar added a commit to toofar/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477). So lets just
set it to 2 for now which should be a lot faster than 1, if it works.
toofar added a commit to toofar/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477). So lets just
set it to 2 for now which should be a lot faster than 1, if it works.
toofar added a commit to toofar/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477). So lets just
set it to 2 for now which should be a lot faster than 1, if it works.
toofar added a commit to qutebrowser/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477).
So lets just set it to 2 for now which should be a lot faster than 1, if
it works.

Not sure about using "jobs" as a matrix arg because it kinda shadows the
"jobs" keys at the top level. Maybe for the sake of making it easy to
change it could be defined as an `env` value at the top level.
toofar added a commit to qutebrowser/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477).
So lets just set it to 2 for now which should be a lot faster than 1, if
it works.

Not sure about using "jobs" as a matrix arg because it kinda shadows the
"jobs" keys at the top level. Maybe for the sake of making it easy to
change it could be defined as an `env` value at the top level.
toofar added a commit to qutebrowser/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477).
So lets just set it to 2 for now which should be a lot faster than 1, if
it works.

Not sure about using "jobs" as a matrix arg because it kinda shadows the
"jobs" keys at the top level. Maybe for the sake of making it easy to
change it could be defined as an `env` value at the top level.
toofar added a commit to qutebrowser/qutebrowser that referenced this issue May 25, 2024
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runners have 4 cores (apart from macos-11).

We could use `-n auto`, but I hypothesis that for the e2e tests since we
have one worker process per test process maybe we don't want to actually
use up all the cores with test processes? It doesn't look like it's
possible to tell pytest-xdist to use all core but one, or half the
cores, or anything (issue
herehttps://github.com/pytest-dev/pytest-xdist/issues/477).
So lets just set it to 2 for now which should be a lot faster than 1, if
it works.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants