From a268885fbe2deafd7069be2211ed8fa7b2251268 Mon Sep 17 00:00:00 2001 From: Dominik Gresch Date: Wed, 9 Feb 2022 14:04:54 +0100 Subject: [PATCH 1/3] Update packaging guide to refer to 'flit' --- doc/source/library_description/packaging.rst | 87 +++++++++++++------- 1 file changed, 57 insertions(+), 30 deletions(-) diff --git a/doc/source/library_description/packaging.rst b/doc/source/library_description/packaging.rst index 08d633d2b..b98b38762 100644 --- a/doc/source/library_description/packaging.rst +++ b/doc/source/library_description/packaging.rst @@ -11,13 +11,10 @@ Namespace Packaging ------------------- A PyAnsys library uses `namespace packaging`_. Namespace packages allow a user to easily split subpackages from a package into -a single, independent distribution. +single, independent distributions. -Three different approaches are currently available for creating a namespace package: - -* `native namespace packages`_ -* pkgutil-style namespace packages -* pkg_resources-style namespace packages +There are different approaches available for creating a namespace package. For the +``ansys`` namespace, we use the `PEP 420`_ `native namespace packages`_ approach. Required Files -------------- @@ -27,48 +24,71 @@ Required Files * LICENSE file: Specifies copyrights and required authorization. -* setup.py file: Provides package information. - The presence of this file indicate that the package was likely created using ``disutils``, - which is the Python standard for building and distributing a Python package. +* pyproject.toml file: Provides package information. + This file provides the package metadata, and defines how it is built. + There are different build backends available, such as `setuptools`_, + `poetry`_ and `flit`_. -Setup File ----------- -The `setup.py`_ file is the build script for ``setuptools``. It exposes dynamic metadata and contains -package information, such as a description, author, and version. -In this file, the ``setuptools`` module is used to configure the metadata (as opposed to ``distutils``). +Project Configuration File +-------------------------- -.. code:: python +The ``pyproject.toml`` file is the standardized build configuration file for Python +projects. It needs to at least contain a ``[build-system]`` section, which determines +how the project is built. Some commonly used packaging tools are `setuptools`_, +`poetry`_, or `flit`_. + +When writing a *library*, `flit`_ is a good default choice. For *applications*, +`poetry`_ is a good default as it provides features to pin dependency versions. +We use `flit`_ in the template repository and the description below. + +To use `flit`_ as a packaging tool, the ``pyproject.toml`` should contain + +.. code:: toml - import setuptools - setuptools.setup(...) + [build-system] + requires = ["flit_core >=3.2,<4"] + build-backend = "flit_core.buildapi" -This file gathers all namespace packages and files that must be included in the distributed -package. +The ``[project]`` section contains metadata, and defines the project's dependencies. Refer to the +`flit metadata documentation`_ for details. + +Flit can automatically determine the project's version from the source code. +In the ``[project]`` section, add + +.. code:: toml + + dynamic = ["version"] + +The version is parsed from the ``ansys/package/library/__init__.py`` file, which must +contain a statement .. code:: python - packages = [] - for package in setuptools.find_namespace_packages(include='ansys*'): - if package.startswith('ansys.tools.example_coverage'): - packages.append(package) + __version__ = "0.1.0" + +Where supported, we aim to put all tooling-related configuration into ``pyproject.toml``. +For example, it can also be used to configure the code formatter `black`_ or the static +type checker `mypy`_. +.. note:: -It also extracts the version number from the ``_version.py`` file located in the -``ansys//library`` directory of the source code. + When using `setuptools`_ as a build backend, providing the metadata in ``pyproject.toml`` is not yet fully supported. + Instead, it also requires a ``setup.cfg`` and / or ``setup.py`` file. Generate the Package and Upload It on PyPI ------------------------------------------ -The first time that you want to upload a package on PyPI under the `ansys `_ +The first time that you want to upload a package on PyPI under the `ansys `_ account, you must perform the following process manually. Create the python package. .. code:: - python setup.py sdist + pip install build + python -m build Verify the distribution's long description rendering with ``twine``. @@ -77,7 +97,7 @@ Verify the distribution's long description rendering with ``twine``. pip install twine twine check dist/* -Upload the package to PyPI using ``twine`` and the upload token generated for the ``ansys`` PyPI account. +Upload the package to PyPI using ``twine`` and the upload token generated for the ``ansys`` PyPI account. Contact alexander.kaszynski@ansys.com for the token. .. code:: @@ -127,7 +147,7 @@ Install a package with: .. code:: - pip install ansys.. + pip install ansys-- To create a package complying with the above standards, here is the minimal content of your PyAnsys library: @@ -136,12 +156,19 @@ To create a package complying with the above standards, here is the minimal cont ansys///__init__.py LICENSE README.rst - setup.py + pyproject.toml tests/ .. _namespace packaging: https://packaging.python.org/guides/packaging-namespace-packages/ .. _native namespace packages: https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages +.. _PEP 420: https://www.python.org/dev/peps/pep-0420/ +.. _setuptools: https://setuptools.pypa.io +.. _poetry: https://python-poetry.org/docs/ +.. _flit: https://flit.readthedocs.io +.. _flit metadata documentation: https://flit.readthedocs.io/en/latest/pyproject_toml.html#new-style-metadata +.. _black: https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-via-a-file +.. _mypy: https://mypy.readthedocs.io/en/stable/config_file.html#the-mypy-configuration-file .. _trunk-based development: https://trunkbaseddevelopment.com/ .. _secret: https://docs.github.com/en/actions/reference/encrypted-secrets .. _setup.py: https://packaging.python.org/tutorials/packaging-projects/#configuring-metadata From 3a7b5cbb53df6293db9f1ebeaa0a5f47b4d6b023 Mon Sep 17 00:00:00 2001 From: Dominik Gresch Date: Tue, 22 Feb 2022 17:22:45 +0100 Subject: [PATCH 2/3] Switch from 'flit' to 'poetry' --- doc/source/library_description/packaging.rst | 42 +++++++++++--------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/doc/source/library_description/packaging.rst b/doc/source/library_description/packaging.rst index b98b38762..716280077 100644 --- a/doc/source/library_description/packaging.rst +++ b/doc/source/library_description/packaging.rst @@ -38,34 +38,38 @@ projects. It needs to at least contain a ``[build-system]`` section, which deter how the project is built. Some commonly used packaging tools are `setuptools`_, `poetry`_, or `flit`_. -When writing a *library*, `flit`_ is a good default choice. For *applications*, -`poetry`_ is a good default as it provides features to pin dependency versions. -We use `flit`_ in the template repository and the description below. +We use `poetry`_ as a default choice in the `PyAnsys template`_, for the following reasons: +* it supports pinning dependency versions, which we use for testing / CI +* downstream packages can still consume a loose dependency specification +* it integrates with `dependabot`_ to update the pinned version -To use `flit`_ as a packaging tool, the ``pyproject.toml`` should contain +Feel free to use any one of the packaging tools mentioned above that best suits +your needs. The advantage of `flit`_ is its simplicity, while `setuptools`_ is most useful +when custom build steps need to be implemented as Python code. + +To use `poetry`_ as a packaging tool, the ``pyproject.toml`` should contain .. code:: toml [build-system] - requires = ["flit_core >=3.2,<4"] - build-backend = "flit_core.buildapi" - -The ``[project]`` section contains metadata, and defines the project's dependencies. Refer to the -`flit metadata documentation`_ for details. + requires = ["poetry-core>=1.0.0"] + build-backend = "poetry.core.masonry.api" -Flit can automatically determine the project's version from the source code. -In the ``[project]`` section, add +The ``[tool.poetry]`` section contains metadata, and defines the project's dependencies. Refer to the +`poetry pyproject.toml documentation`_ for details. -.. code:: toml +Since poetry cannot automatically determine a package's version, we instead specify it in the ``[tool.poetry]`` +section, and add code to ``__init__.py`` which obtains the version from the installation metadata: - dynamic = ["version"] +.. code:: python -The version is parsed from the ``ansys/package/library/__init__.py`` file, which must -contain a statement + try: + import importlib.metadata as importlib_metadata + except ModuleNotFoundError: + import importlib_metadata -.. code:: python + __version__ = importlib_metadata.version(__name__.replace(".", "-")) - __version__ = "0.1.0" Where supported, we aim to put all tooling-related configuration into ``pyproject.toml``. For example, it can also be used to configure the code formatter `black`_ or the static @@ -166,7 +170,9 @@ To create a package complying with the above standards, here is the minimal cont .. _setuptools: https://setuptools.pypa.io .. _poetry: https://python-poetry.org/docs/ .. _flit: https://flit.readthedocs.io -.. _flit metadata documentation: https://flit.readthedocs.io/en/latest/pyproject_toml.html#new-style-metadata +.. _dependabot: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/about-dependabot-version-updates +.. _PyAnsys template: https://github.com/pyansys/template +.. _poetry pyproject.toml documentation: https://python-poetry.org/docs/pyproject/ .. _black: https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-via-a-file .. _mypy: https://mypy.readthedocs.io/en/stable/config_file.html#the-mypy-configuration-file .. _trunk-based development: https://trunkbaseddevelopment.com/ From 38a1ae93e291934e0cef40aaada8cae94a8b9546 Mon Sep 17 00:00:00 2001 From: Jorge Martinez Date: Mon, 7 Mar 2022 22:59:21 +0100 Subject: [PATCH 3/3] Update to pyansys-templates --- doc/source/library_description/packaging.rst | 231 +++++++++++++++---- 1 file changed, 187 insertions(+), 44 deletions(-) diff --git a/doc/source/library_description/packaging.rst b/doc/source/library_description/packaging.rst index 716280077..99a7008fd 100644 --- a/doc/source/library_description/packaging.rst +++ b/doc/source/library_description/packaging.rst @@ -2,64 +2,177 @@ Packaging ######### -A Python package organizes and structures a Python library, which contains several -modules and assets such as examples or binary extensions. A Python package -offers an easy, reliable, and comprehensive way to distribute and install -a Python library on a variety of platforms and environments. + +A Python package organizes and structures a Python library, which contains +several modules and assets such as examples or binary extensions. A Python +package offers an easy, reliable, and comprehensive way to distribute and +install a Python library on a variety of platforms and environments. + +.. note:: + + If you want to create a new PyAnsys project according to the guidelines + presented in the following lines, consider using the `ansys-templates tool`_. + + +Python Scripts, Modules, Sub-packages, and Packages +--------------------------------------------------- + +It is important to understand the difference between Python scripts, modules, +sub-packages, and packages: + +* ``Script``: Any Python file with logic source code. +* ``Module``: Any Python script hosted next to an ``__init__.py`` file. +* ``Sub-package``: Any directory containing various Python modules. +* ``Package``: Any directory containing Python modules and sub-packages. + +The following structure is shown to better explain previous concepts: + +.. code:: bash + + . + ├── src + │   └── package + │   ├── subpackage_a + │   │   ├── __init__.py + │   │   └── module_c.py + │   ├── __init__.py + │   ├── module_a.py + │   └── module_b.py + ├── LICENSE + ├── README.rst + └── pyproject.toml + Namespace Packaging ------------------- -A PyAnsys library uses `namespace packaging`_. -Namespace packages allow a user to easily split subpackages from a package into -single, independent distributions. +A PyAnsys library uses `namespace packaging`_. Namespace packages allow you +to easily split sub-packages from a package into single, independent +distributions. + +There are different approaches available for creating a namespace package. For +the ``ansys`` namespace, we use the `PEP 420`_ `native namespace packages`_ +approach. + +Therefore, the source directory of any `PyAnsys library` should look like this: + +.. code:: bash + + . + └── src + └── ansys + └── product + └── library + └── __init__.py -There are different approaches available for creating a namespace package. For the -``ansys`` namespace, we use the `PEP 420`_ `native namespace packages`_ approach. Required Files -------------- -* README.rst file: Describes the purpose of the package. +* ``README.rst`` file: Describes the purpose of the package. *The format of this file must be reStructuredText.* -* LICENSE file: Specifies copyrights and required authorization. +* ``LICENSE`` file: Specifies copyrights and required authorization. -* pyproject.toml file: Provides package information. - This file provides the package metadata, and defines how it is built. - There are different build backends available, such as `setuptools`_, - `poetry`_ and `flit`_. +* ``pyproject.toml`` file: Provides package metadata and defines how the package + is built. There are different build backends available, such as `setuptools`_, + `poetry`_, and `flit`_. +* ``src/ansys/product/library/__init__.py`` file: Usually contains the + version of the package in a variable named ``__version__``. The value of this + variable can be parsed from the ``pyproject.toml`` file so that the version + is only specified in one location. -Project Configuration File --------------------------- + +Additional Directories +---------------------- + +The following directories may be specified at the same level as the ``src/`` one: + +* ``tests/``: Contains all unit tests for the package. It is + likely that these tests take advantage of the `pytest`_ framework. + +* ``doc/``: Contain all documentation files and examples on + how to use the package. + + +Project File and Build System +------------------------------ The ``pyproject.toml`` file is the standardized build configuration file for Python -projects. It needs to at least contain a ``[build-system]`` section, which determines +projects. It must contain at least a ``[build-system]`` section, which determines how the project is built. Some commonly used packaging tools are `setuptools`_, -`poetry`_, or `flit`_. +`poetry`_, and `flit`_. All three of these packaging tools are currently supported by +the ``pyansys-advanced`` template, which is included in the `ansys-templates tool`_. + + +Flit +^^^^ -We use `poetry`_ as a default choice in the `PyAnsys template`_, for the following reasons: -* it supports pinning dependency versions, which we use for testing / CI -* downstream packages can still consume a loose dependency specification -* it integrates with `dependabot`_ to update the pinned version +Flit is a modern and lightweight build system that requires developers +to manage virtual environments on their own. Developers must: -Feel free to use any one of the packaging tools mentioned above that best suits -your needs. The advantage of `flit`_ is its simplicity, while `setuptools`_ is most useful -when custom build steps need to be implemented as Python code. +* Create a virtual environment and activate it. +* Install the package in editable mode. -To use `poetry`_ as a packaging tool, the ``pyproject.toml`` should contain +Flit is the default tool for creating a new ``pyansys`` project when using the +`ansys-templates tool`_. -.. code:: toml +The ``[project]`` section specifies the project's metadata and required +dependencies. For more information, see `flit pyproject.toml +guidelines`_. - [build-system] - requires = ["poetry-core>=1.0.0"] - build-backend = "poetry.core.masonry.api" -The ``[tool.poetry]`` section contains metadata, and defines the project's dependencies. Refer to the -`poetry pyproject.toml documentation`_ for details. +Poetry +^^^^^^ + +Because of its ``poetry.lock`` file, Poetry provides strong dependency pinning. When +installing a package, poetry creates a virtual environment, thus ensuring an isolated +package development environment. + +Nevertheless, it is possible to make Poetry ignore the `poetry.lock` file by running: + +.. code:: bash + + poetry config virtualenvs.create false --local + +Using `poetry`_ is popular because it: + +* Supports pinning dependency versions via a ``poetry.lock`` file that can be + used for testing and CI +* Allows downstream packages to still consume a loose dependency specification +* Integrates with `dependabot`_ to update the pinned version + +The ``[tool.poetry]`` section contains metadata and defines the project's +dependencies. For more information, see `poetry pyproject.toml documentation`_. + + +Setuptools +^^^^^^^^^^ + +Setuptools is a very well known build system in the Python ecosystem. It is used +in projects requiring a ``setup.py`` file and can be used in projects with a +``pyproject.toml`` file, although not all metadata in this second file +is fully supported yet. + +The main advantage of this build system is the ability to create custom build +steps in the form of Python code. + + +Specifying Package Version +-------------------------- + +It is very common for packages to specify their current version in the +``__version__`` variable. This variable is usually declared in the +``__init__.py`` file included in the ``library`` directory. + +However, it is also required to specify the version in the ``pyproject.toml`` or +``setup.py`` file. This leads to a duplicate declaration of the project's version, +which could lead to a potential mismatch between both. + +Therefore, a good practice is to take advantage of the `importlib.metadata package`_ +for parsing the version from package metadata. This guarantees that there is no mismatch +between both version declarations. -Since poetry cannot automatically determine a package's version, we instead specify it in the ``[tool.poetry]`` -section, and add code to ``__init__.py`` which obtains the version from the installation metadata: .. code:: python @@ -71,20 +184,32 @@ section, and add code to ``__init__.py`` which obtains the version from the inst __version__ = importlib_metadata.version(__name__.replace(".", "-")) -Where supported, we aim to put all tooling-related configuration into ``pyproject.toml``. -For example, it can also be used to configure the code formatter `black`_ or the static -type checker `mypy`_. +Extra Tools Configuration +------------------------- + +There are plenty of tools in the Python ecosystem that enable developers to +write clean code according to different coding style guidelines. Some of these +tools are `black`_, `isort`_, `flake8`_, and `mypy`_. + +Some of these tools can be configured. This configuration might be specified in +custom files required by the tool or in the ``pyproject.toml`` file, thus reducing the +number of files in the project directory. .. note:: - When using `setuptools`_ as a build backend, providing the metadata in ``pyproject.toml`` is not yet fully supported. - Instead, it also requires a ``setup.cfg`` and / or ``setup.py`` file. + When using `setuptools`_ as a build backend, providing the metadata in + the ``pyproject.toml`` file is not yet fully supported. Instead, it also + requires a ``setup.cfg`` file, ``setup.py`` file, or both files. + +In the `pyansys template`, all these configurations are included by default in +the ``.pre-commit-config.yaml`` file because ``pre-commit`` is not able to parse the +``pyproject.toml`` file nor the ``setup.py`` file. Generate the Package and Upload It on PyPI ------------------------------------------ -The first time that you want to upload a package on PyPI under the `ansys `_ +The first time that you want to upload a package on PyPI under `ansys `_ account, you must perform the following process manually. Create the python package. @@ -94,6 +219,13 @@ Create the python package. pip install build python -m build +If using flit or poetry, you can also run: + +.. code:: + + flit build + poetry build + Verify the distribution's long description rendering with ``twine``. .. code:: @@ -101,8 +233,14 @@ Verify the distribution's long description rendering with ``twine``. pip install twine twine check dist/* -Upload the package to PyPI using ``twine`` and the upload token generated for the ``ansys`` PyPI account. -Contact alexander.kaszynski@ansys.com for the token. + +Upload the package to PyPI using ``twine`` and the upload token generated for +the ``ansys`` PyPI account. As soon as the package has been released for the +first time, it is possible to create an independent token dedicated to this +package. This way the token stored in the GitHub secrets and used in the +release's workflow is only related to that specific package. This limits the +exposure to any potential token security flaws. Contact +alexander.kaszynski@ansys.com for the token. .. code:: @@ -169,12 +307,17 @@ To create a package complying with the above standards, here is the minimal cont .. _PEP 420: https://www.python.org/dev/peps/pep-0420/ .. _setuptools: https://setuptools.pypa.io .. _poetry: https://python-poetry.org/docs/ +.. _flit pyproject.toml guidelines: https://flit.readthedocs.io/en/latest/pyproject_toml.html .. _flit: https://flit.readthedocs.io .. _dependabot: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/about-dependabot-version-updates -.. _PyAnsys template: https://github.com/pyansys/template +.. _ansys-templates tool: https://github.com/pyansys/pyansys-templates .. _poetry pyproject.toml documentation: https://python-poetry.org/docs/pyproject/ .. _black: https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-via-a-file .. _mypy: https://mypy.readthedocs.io/en/stable/config_file.html#the-mypy-configuration-file .. _trunk-based development: https://trunkbaseddevelopment.com/ .. _secret: https://docs.github.com/en/actions/reference/encrypted-secrets .. _setup.py: https://packaging.python.org/tutorials/packaging-projects/#configuring-metadata +.. _importlib.metadata package: https://docs.python.org/3/library/importlib.metadata.html +.. _isort: https://github.com/PyCQA/isort +.. _flake8: https://flake8.pycqa.org/en/latest/ +.. _pytest: https://docs.pytest.org/en/latest/