Test collection of non-python tests

[sashko1988]

Strange performance struggle of pytest collection with non-python tests.

Any help appreciated!

Set up

Versions

Python 3.12.8
pytest 8.3.4

I use non-python tests. Let's assume it's a yaml file for this discussion.

I followed the guide from the docs: https://docs.pytest.org/en/stable/example/nonpython.html, which was helpful. And tests are running without problems.

But collection is where pytest struggles.

A little bit about my setup:

1. Custom collector is part of my plugin. And I load it with -p in the CLI.
2. Two separate folders - one with the test framework (where the plugin is stored) and the other containing only tests.
3. I have a lot of tests of that custom kind (~23k files with that extension) and deep nesting in my test folder.
4. I store my test suites in DBs. And when I resolve the suite, in a pytest_collection hook wrapper, I pass resolved absolute paths into config.args and then run the main collection hook.

Some code context:

@pytest.hookimpl(wrapper=True)
def pytest_collection(session):
    resolved_paths = resolve_suites(session)
    session.config.args.extend(resolved_paths)
    return (yield)
    
def pytest_collect_file(parent, file_path):
    if file_path.suffix == ".yaml":
        return YamlFile.from_parent(parent, path=file_path)

class YamlFile(pytest.File):
    def collect(self) -> Iterable[pytest.Item | pytest.Collector]:
        test_cases = YamlTestResolver().from_file(f"{self.path}")  # leftover from previous runner, but resolves needed stuff.
        for tc in test_cases:
            yield YamlTest.from_parent(self, name=tc.name, tc_spec=tc)
            
class YamlTest(pytest.Item):
    def __init__(self, ptul_tc, **kwargs) -> None:
        super().__init__(**kwargs)
        self.tc_spec = tc_spec

How it struggles:

A lot of unnecessary file checks

Before it even starts creating test items, the main problem is that it iterates through the same files more than once. I added print statements for every file it checks in pytest_collect_file; some paths are repeated 193 times in the produced log for a suite with 1000k test files. Sometimes, those paths are not even tests that need to run.
I run that 1000 suite with the profiler and --collect-only. The whole time was around 2 minutes. And according to the report,
pytest_collect_file ran for 42 seconds, and the actual collection YamlFile.collect took only 6 seconds.
Log file contains 455k lines with paths that were checked in the pytest_collect_file.

• Why does pytest check 455k paths when I give the exact paths of the files?
• How can I optimize the collection here?

Different execution time depending from where pytest was called

The other problem is that I have a different execution time if I run pytest from the framework directory and any other folder.
The flow from the framework directory:

1. Go to the framework dir.
2. Load environment.
3. Start collection

From any other folder:

1. Go to the framework dir.
2. Load environment.
3. Go to the parent folder
4. Start collection

Same args, same paths, but running from the root dir or test dir, it takes 38 seconds without the profiler / around 2 minutes with, and from the framework dir, it takes 23 minutes without the profiler.
The framework and test folders are subfolders of my home folder, on the same disk, which is a pretty fast SSD.
Same struggle with CI - I can't resolve tests, because pytest is called from the framework folder.

What can be the problem?

UPDATED

Profiler results and more info on

Consider this folder structure:

root_working_folder
├── framework_repo
│   └── framework_internal_folder
└── repo_with_tests
    └── tests
        ├── test_folder_1
        │   └── inner_folder
        └── test_folder_2
            └── inner_folder
                └── even_more_depth

But even more subfolders in repo_with_tests

Pytest call is the following: pytest --collect only ${list with 1k non-python tests. (1 test per file)

When I execute the above from framework_internal_folder, the execution time is 56 minutes. When I make the same call from root_working_folder or repo_with_tests, the execution time is ~2 minutes.

The most significant time difference in the two calls is in the cumulative time of that function - nodes.py:546(_check_initialpaths_for_relpath)

# from framework_internal_folder
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   237033   85.508    0.000 3176.004    0.013 ../_pytest/nodes.py:546(_check_initialpaths_for_relpath)

# from root_working_folder
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      135    0.051    0.000    1.772    0.013 ../_pytest/nodes.py:546(_check_initialpaths_for_relpath)

According to stats, when executing from framework_internal_folder, the most struggling function is here:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
206063304  262.471    0.000 1580.672    0.000 ../_pytest/pathlib.py:990(commonpath)

# and stats for callers of that function:
Function                            was called by...
                                        ncalls  tottime  cumtime
pathlib.py:990(commonpath)          <- 205937164/3722648  262.308   28.844  nodes.py:546(_check_initialpaths_for_relpath)

[RonnyPfannschmidt]

That indicates an misunderstanding and/or a missing cache we never knew we needed

[sashko1988]

I did experiment with it.

@lru_cache(maxsize=1000)
def _check_initialpaths_for_relpath(initialpaths: frozenset[Path], path: Path) -> str | None:
    for initial_path in initialpaths:
        if commonpath(path, initial_path) == initial_path:
            rel = str(path.relative_to(initial_path))
            return "" if rel == "." else rel
    return None

That change decreased the overall collection time to 4 minutes.

Stats are also impressive:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     5798    2.109    0.000   79.265    0.014 nodes.py:545(_check_initialpaths_for_relpath)

I'm not sure if commonpath needs caching as well.

[sashko1988]

@RonnyPfannschmidt do I need to create the issue?

[RonnyPfannschmidt]

Yes please

[webknjaz]

You could also go for a PR right away for things that are straightforward..

[sashko1988]

Created an issue - #13420. So we can discuss other optimizations besides adding a cache.

In the meantime, working on PR on adding cache _check_initialpaths_for_relpath