npm should not be required to install pytest-html from pypi

[thebaptiste]

Since release 4.0.0rc5, npm is required to install pytest-html from pypi

I understand npm is only used for tests purpose (maybe I'm wrong...) ? If it is the case I think it should not be necessary to have npm pre-installed to install pytest-html

Below the error I have installing pytest-html >= 4.0.0rc5 from pypi (same thing with 4.0.0, 4.0.1 and 4.0.2) :

  Downloading pytest_html-4.0.0rc5.tar.gz (145 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 145.8/145.8 kB 20.8 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      /bin/sh: npm: command not found
      Traceback (most recent call last):
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/hatchling/build.py", line 56, in build_wheel
          return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 150, in build
          build_hook.initialize(version, build_data)
        File "/home/baptiste/mfext/src/layers/layer7_python3_devtools/0500_extra_python_packages/tempolayer3830361/tmp/pip-wheel-dl81rpb1/pytest-html_2cea07a17db443aebda5c2ba1b9efeb4/scripts/npm.py", line 8, in initialize
          subprocess.check_output("npm ci", shell=True)
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'npm ci' returned non-zero exit status 127.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

[BeyondEvil]

hmm... that's weird. Like you say, it shouldn't be required. It's only required when building from source.

It's somehow related to Hatch and/or pyproject.toml - not sure how or why tho.

[BeyondEvil]

pytest-html installs just fine on a system without npm.

jbrwe6@MAC-6CY79W0M dev % mkdir testing-npm
jbrwe6@MAC-6CY79W0M dev % cd testing-npm
jbrwe6@MAC-6CY79W0M testing-npm % python3 -m venv .venv
jbrwe6@MAC-6CY79W0M testing-npm % source .venv/bin/activate
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip freeze
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % npm
zsh: command not found: npm
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip install pytest-html
Collecting pytest-html
  Downloading pytest_html-4.0.2-py3-none-any.whl (23 kB)
*snip*
Installing collected packages: tomli, pluggy, packaging, iniconfig, exceptiongroup, pytest, MarkupSafe, pytest-metadata, jinja2, pytest-html
Successfully installed MarkupSafe-2.1.3 exceptiongroup-1.1.3 iniconfig-2.0.0 jinja2-3.1.2 packaging-23.2 pluggy-1.3.0 pytest-7.4.3 pytest-html-4.0.2 pytest-metadata-3.0.0 tomli-2.0.1
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip freeze
exceptiongroup==1.1.3
iniconfig==2.0.0
Jinja2==3.1.2
MarkupSafe==2.1.3
packaging==23.2
pluggy==1.3.0
pytest==7.4.3
pytest-html==4.0.2
pytest-metadata==3.0.0
tomli==2.0.1

So unless you can provide steps to reproduce, I'm closing the ticket.

[RonnyPfannschmidt]

It seems like the sdist needs it still, I'd recommend checking if the build results can be included and used for the sdist as well to ease things for people that work with broken ci setups or repackage

[BeyondEvil]

It seems like the sdist needs it still, I'd recommend checking if the build results can be included and used for the sdist as well to ease things for people that work with broken ci setups or repackage

Aha, so something (likely in the pyproject.toml) is missing specifically for the sdist?

Into the Hatch documentation I go then...

[BeyondEvil]

Hmm... when I build with Hatch, the app.js is included in both sdist and wheel.

However, it seems that the zipped and tarred assets in the Github releases does not.

Something I need to test, is what the result of the way CI (using hynek/build-and-inspect-python-package@v1) builds the dist.

[BeyondEvil]

Building with SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) python -m build --outdir dist also shows app.js as part of the sdist.

I'm out of ideas. Without steps to reproduce, I can't fix this.

[RonnyPfannschmidt]

$ podman run --rm python pip install --no-binary pytest-html pytest-html
Collecting pytest-html
  Downloading pytest_html-4.0.2.tar.gz (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.1/149.1 kB 2.7 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      /tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/setuptools_scm/_integration/dump_version.py:65: UserWarning: template='' looks like a error, using default instead
        warnings.warn(f"{template=} looks like a error, using default instead")
      /bin/sh: 1: npm: not found
      Traceback (most recent call last):
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/hatchling/build.py", line 56, in build_wheel
          return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 150, in build
          build_hook.initialize(version, build_data)
        File "/tmp/pip-install-_hpom3az/pytest-html_facb36f6a4cc48a083187845766b40bf/scripts/npm.py", line 8, in initialize
          subprocess.check_output("npm ci", shell=True)
        File "/usr/local/lib/python3.12/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'npm ci' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1

[RonnyPfannschmidt]

so the build hook is always used

@ofek is there builtin hatch helper to have certain build steps happen for from source but not for from sdist

[BeyondEvil]

If the response from Ofek is negative, what's the solution?

Remove the build-hook and build it "manually" in CI and make sure app.js is added to wheel and sdist?

[ofek]

The fix should be pretty simple https://github.com/pytest-dev/pytest-html/blob/master/scripts/npm.py

Just gate those two statements by a check for something that is not shipped in either artifact like /.git/

[BeyondEvil]

The fix should be pretty simple https://github.com/pytest-dev/pytest-html/blob/master/scripts/npm.py

Just gate those two statements by a check for something that is not shipped in either artifact like /.git/

You mean like:

import subprocess
from hatchling.builders.hooks.plugin.interface import BuildHookInterface
from pathlib import Path


class NpmBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        app_js = Path("src/pytest_html/resources/app.js")
        if not app_js.exists():
            subprocess.check_output("npm ci", shell=True)
            subprocess.check_output("npm run build", shell=True)

[ofek]

Yes but just for correctness/to be pedantic even though it's the same path:

app_js = Path(self.root, "src", "pytest_html", "resources", "app.js")

Also I do understand what you are going for there but that would not work for repeat builds like running it twice locally which is why I recommended something that is not shipped at all if possible.

[BeyondEvil]

Also I do understand what you are going for there but that would not work for repeat builds like running it twice locally which is why I recommended something that is not shipped at all if possible.

Ah, I see what you mean now! If /.git/ is present we can assume we're building from source.

Can there be cases where someones building from source but .git is not present?

import subprocess
from hatchling.builders.hooks.plugin.interface import BuildHookInterface
from pathlib import Path


class NpmBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        building_from_source = Path(self.root, ".git").exists()
        if building_from_source:
            subprocess.check_output("npm ci", shell=True)
            subprocess.check_output("npm run build", shell=True)

[ofek]

Oh yeah definitely you're right, forget what I said and continue with the path that you chose. A situation would be an archive from a GitHub release. You build artifacts in CI the proper way and it would only run once so it's fine.