Skip to content

Changing dir in test makes pytest-html fail #732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
adrien-berchet opened this issue Sep 8, 2023 · 7 comments · Fixed by #735
Closed

Changing dir in test makes pytest-html fail #732

adrien-berchet opened this issue Sep 8, 2023 · 7 comments · Fixed by #735

Comments

@adrien-berchet
Copy link
Contributor

I use a fixture like the following:

@pytest.fixture()
def tmp_working_dir(tmp_path, monkeypatch):
    """Change working directory before a test and change it back when the test is finished."""
    monkeypatch.chdir(tmp_path)
    yield tmp_path

It was working well with pytest-html<4 but with pytest-html>=4 I get the following error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 130, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 226, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 113, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 251, in pytest_runtest_logreport
INTERNALERROR>     self._generate_report()
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/selfcontained_report.py", line 39, in _generate_report
INTERNALERROR>     super()._generate_report(self_contained=True)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 65, in _generate_report
INTERNALERROR>     self._write_report(rendered_report)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 134, in _write_report
INTERNALERROR>     with self._report_path.open("w", encoding="utf-8") as f:
INTERNALERROR>   File "/usr/local/lib/python3.8/pathlib.py", line 1222, in open
INTERNALERROR>     return io.open(self, mode, buffering, encoding, errors, newline,
INTERNALERROR>   File "/usr/local/lib/python3.8/pathlib.py", line 1078, in _opener
INTERNALERROR>     return self._accessor.open(self, flags, mode)
INTERNALERROR> FileNotFoundError: [Errno 2] No such file or directory: 'reports/pytest-py38.html'

I use the following command to call pytest in a tox job:

    pytest \
        --basetemp={envtmpdir} \
        --cov={[base]name} \
        --cov-branch \
        --cov-fail-under=64 \
        --no-cov-on-fail \
        --cov-report term-missing \
        --cov-report html:reports/coverage-{envname} \
        --cov-report xml:reports/coverage-{envname}.xml \
        --html reports/pytest-{envname}.html \
        --junit-xml=reports/pytest-{envname}.xml \
        --self-contained-html \
        {posargs}
@BeyondEvil
Copy link
Contributor

Oh, that's interesting!

Can you provide a complete minimal example?

@adrien-berchet
Copy link
Contributor Author

Hi @BeyondEvil
Here is a simple project that shows the issue: pytest-html-issue-732.zip
If you run the command tox run -e py310, you will see the failure.

Please note the following behaviors:

  • if you set pytest-html<4 in setup.py the tests pass.
  • if you change the order of the tests in tests/test_example.py, pytest-html will fail after the first test using the monkeypatch.chdir passes, so it seems that pytest-html is executed after the test passes but before the fixture teardowns are executed, while I think it should be executed after all the fixture teardowns.

@BeyondEvil
Copy link
Contributor

Hi @BeyondEvil Here is a simple project that shows the issue: pytest-html-issue-732.zip If you run the command tox run -e py310, you will see the failure.

Please note the following behaviors:

  • if you set pytest-html<4 in setup.py the tests pass.
  • if you change the order of the tests in tests/test_example.py, pytest-html will fail after the first test using the monkeypatch.chdir passes, so it seems that pytest-html is executed after the test passes but before the fixture teardowns are executed, while I think it should be executed after all the fixture teardowns.

Thanks for the example!

As for your second bullet point. A pytest test (item) has three states: setup, call, teardown. Since we create an entry in the result table depending on where a tests fails - we can't wait until teardown has happened.

Regardless, I'm 99% sure that if we fix the root issue here (which likely has to do with relative vs. absolute paths) that won't be an issue.

@BeyondEvil
Copy link
Contributor

Getting an error when running your example:

☸ deployment (ci) in ~/Downloads/pytest-html-issue-732 via 🐍 v3.9.10 (.venv) 
❯ tox run -e py39       
.pkg: _optional_hooks> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py39: packaging backend failed (code=1), with LookupError: setuptools-scm was unable to detect version for /Users/jimbrannlund/Downloads/pytest-html-issue-732.

@BeyondEvil
Copy link
Contributor

Also, I'd prefer to be able to repro this without tox, which I can't.

@adrien-berchet
Copy link
Contributor Author

As for your second bullet point. A pytest test (item) has three states: setup, call, teardown. Since we create an entry in the result table depending on where a tests fails - we can't wait until teardown has happened.

Regardless, I'm 99% sure that if we fix the root issue here (which likely has to do with relative vs. absolute paths) that won't be an issue.

Indeed it should work with absolute paths and it's simpler.

Getting an error when running your example:

☸ deployment (ci) in ~/Downloads/pytest-html-issue-732 via 🐍 v3.9.10 (.venv) 
❯ tox run -e py39       
.pkg: _optional_hooks> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py39: packaging backend failed (code=1), with LookupError: setuptools-scm was unable to detect version for /Users/jimbrannlund/Downloads/pytest-html-issue-732.

Ah yes I guess you have to initialize a git repo and commit everything to make setuptools-scm work, I forgot about this...

Also, I'd prefer to be able to repro this without tox, which I can't.

To run it without tox you can just create a virtualenv and do:

pip install ".[test]"
pytest --html reports/pytest-py39.html

@adrien-berchet
Copy link
Contributor Author

Here is a new version to fix the version stuff, so it should be simpler to install it: pytest-html-issue-732.zip

@adrien-berchet adrien-berchet mentioned this issue Sep 12, 2023
Merged
adrien-berchet added a commit to adrien-berchet/pytest-html that referenced this issue Sep 12, 2023
@adrien-berchet
Fix: Use absolute path for the report
9ae689f 
Fixes pytest-dev#732
@adrien-berchet adrien-berchet mentioned this issue Sep 12, 2023
Merged
BeyondEvil pushed a commit that referenced this issue Sep 14, 2023
@adrien-berchet
Fix: Use absolute path for the report (#735)
82d08c4 
* Fix: Use absolute path for the report

Fixes #732
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: Use absolute path for the report adrien-berchet/pytest-html
2 participants
@adrien-berchet @BeyondEvil