Update README/docs to use pyproject.toml + separate setup.py docs

README.md

@@ -10,83 +10,159 @@
 Compile and distribute Python extensions written in Rust as easily as if
 they were written in C.
 
-## Setup
+## Quickstart
 
-For a complete example, see
-[html-py-ever](https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever).
+The following is a very basic tutorial that shows how to use `setuptools-rust` in `pyproject.toml`.
+It assumes that you already have a bunch of Python and Rust files that you want
+to distribute. You can see examples for these files in the 
+[`examples/hello-world`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
+directory in the [github repository](https://github.com/PyO3/setuptools-rust).
+The [PyO3 docs](https://pyo3.rs) have detailed information on how to write Python
+modules in Rust.
 
-First, you need to create a bunch of files:
-
-### setup.py
-
-```python
-from setuptools import setup
-from setuptools_rust import Binding, RustExtension
-
-setup(
-    name="hello-rust",
-    version="1.0",
-    rust_extensions=[RustExtension("hello_rust.hello_rust", binding=Binding.PyO3)],
-    packages=["hello_rust"],
-    # rust extensions are not zip safe, just like C-extensions.
-    zip_safe=False,
-)
+```
+hello-world
+├── python
+│   └── hello_world
+│       └── __init__.py
+└── rust
+    └── lib.rs
 ```
 
-For a complete reference of the options supported by the `RustExtension` class, see the
-[API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
+Once the implementation files are in place, we need to add a `pyproject.toml`
+file that tells anyone that wants to use your project how to build it.
+In this file, we use an [array of tables](https://toml.io/en/v1.0.0#array-of-tables)
+(TOML jargon equivalent to Python's list of dicts) for ``[[tool.setuptools-rust.ext-modules]]``,
+to specify different extension modules written in Rust:
 
-### pyproject.toml
 
 ```toml
+# pyproject.toml
 [build-system]
-requires = ["setuptools", "wheel", "setuptools-rust"]
+requires = ["setuptools", "setuptools-rust"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hello-world"
+version = "1.0"
+
+[tool.setuptools.packages]
+# Pure Python packages/modules
+find = { where = ["python"] }
+
+[[tool.setuptools-rust.ext-modules]]
+# Private Rust extension module to be nested into the Python package
+target = "hello_world._lib"  # The last part of the name (e.g. "_lib") has to match lib.name in Cargo.toml,
+                             # but you can add a prefix to nest it inside of a Python package.
+path = "Cargo.toml"      # Default value, can be omitted
+binding = "PyO3"         # Default value, can be omitted
+py-limited-api = "auto"  # Default value, can be omitted
 ```
 
-### MANIFEST.in
+Each extension module should map directly into the corresponding `[lib]` table on the
+[Cargo manifest file](https://doc.rust-lang.org/cargo/reference/manifest.html):
 
-This file is required for building source distributions
-
-```text
-include Cargo.toml
-recursive-include src *
+```toml
+# Cargo.toml
+[package]
+name = "hello-world"
+version = "0.1.0"
+edition = "2018"
+
+[dependencies]
+pyo3 = { version = "0.19.2", features = ["extension-module"] }
+
+[lib]
+name = "_lib"  # private module to be nested into Python package,
+               # needs to match the name of the function with the `[#pymodule]` attribute
+path = "rust/lib.rs"
+crate-type = ["cdylib"]  # required for shared library for Python to import from.
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+# See also PyO3 docs on writing Cargo.toml files at https://pyo3.rs
 ```
 
-## Usage
-
-You can use same commands as for c-extensions. For example:
+You will also need to tell Setuptools that the Rust files are required to build your
+project from the [source distribution](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html).
+That can be done either via `MANIFEST.in` (see example below) or via a plugin like
+[`setuptools-scm`](https://pypi.org/project/setuptools-scm/).
 
 ```
->>> python ./setup.py develop
-running develop
-running egg_info
-writing hello-rust.egg-info/PKG-INFO
-writing top-level names to hello_rust.egg-info/top_level.txt
-writing dependency_links to hello_rust.egg-info/dependency_links.txt
-reading manifest file 'hello_rust.egg-info/SOURCES.txt'
-writing manifest file 'hello_rust.egg-info/SOURCES.txt'
-running build_ext
-running build_rust
-cargo build --manifest-path extensions/Cargo.toml --features python3
-    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
-
-Creating /.../lib/python3.6/site-packages/hello_rust.egg-link (link to .)
-
-Installed hello_rust
-Processing dependencies for hello_rust==1.0
-Finished processing dependencies for hello_rust==1.0
+# MANIFEST.in
+include Cargo.toml
+recursive-include rust *.rs
 ```
 
-Or you can use commands like `bdist_wheel` (after installing `wheel`). See also [the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
-
-Cross-compiling is also supported, using one of [`crossenv`](https://github.com/benfogle/crossenv), [`cross`](https://github.com/rust-embedded/cross) or [`cargo-zigbuild`](https://github.com/messense/cargo-zigbuild).
-For examples see the `test-crossenv` and `test-cross` and `test-zigbuild` Github actions jobs in [`ci.yml`](https://github.com/PyO3/setuptools-rust/blob/main/.github/workflows/ci.yml).
-
-By default, `develop` will create a debug build, while `install` will create a release build.
-
-## Commands
+With these files in place, you can install the project in a virtual environment
+for testing and making sure everything is working correctly:
+
+```powershell
+# cd hello-world
+python3 -m venv .venv
+source .venv/bin/activate  # on Linux or macOS
+.venv\Scripts\activate     # on Windows
+python -m pip install -e .
+python
+>>> import hello_world
+# ... try running something from your new extension module ...
+# ... better write some tests with pytest ...
+```
 
-  - `build` - Standard build command will also build all rust extensions.
-  - `build_rust` - Command builds all rust extensions.
-  - `clean` - Standard clean command executes cargo clean for all rust
-    extensions.
+## Next steps and final remarks
+
+- When you are ready to distribute your project, have a look on
+  [the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
+
+- Cross-compiling is also supported, using one of
+  [`crossenv`](https://github.com/benfogle/crossenv),
+  [`cross`](https://github.com/rust-embedded/cross) or
+  [`cargo-zigbuild`](https://github.com/messense/cargo-zigbuild).
+  For examples see the `test-crossenv` and `test-cross` and `test-zigbuild` Github actions jobs in
+  [`ci.yml`](https://github.com/PyO3/setuptools-rust/blob/main/.github/workflows/ci.yml).
+
+- You can also use `[[tool.setuptools-rust.bins]]` (instead of `[[tool.setuptools-rust.ext-modules]]`),
+  if you want to distribute a binary executable written in Rust (instead of a library that can be imported by the Python runtime).
+  Note however that distributing both library and executable (or multiple executables),
+  may significantly increase the size of the
+  [wheel](https://packaging.python.org/en/latest/glossary/#term-Wheel)
+  file distributed by the
+  [package index](https://packaging.python.org/en/latest/glossary/#term-Package-Index)
+  and therefore increase build, download and installation times.
+  Another approach is to use a Python entry-point that calls the Rust
+  implementation (exposed via PyO3 bindings).
+  See the [hello-world](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
+  example for more insights.
+
+- For a complete reference of the configuration options, see the
+  [API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
+  You can use any parameter defined by the `RustExtension` class with
+  `[[tool.setuptools-rust.ext-modules]]` and any parameter defined by the
+  `RustBin` class with `[[tool.setuptools-rust.bins]]`; just remember to replace
+  underscore characters `_` with dashes `-` in your `pyproject.toml` file.
+
+- `Cargo.toml` allow only one `[lib]` table per file.
+  If you require multiple extension modules you will need to write multiple `Cargo.toml` files.
+  Alternatively you can create a single private Rust top-level module that exposes
+  multiple submodules (using [PyO3's submodules](https://pyo3.rs/v0.19.2/module#python-submodules)),
+  which may also reduce the size of the build artifacts.
+  You can always keep your extension modules private and wrap them in pure Python
+  to have fine control over the public API.
+
+- If want to include both `[[tool.setuptools-rust.bins]]` and `[[tool.setuptools-rust.ext-modules]]`
+  in the same macOS wheel, you might have to manually add an extra `build.rs` file,
+  see [PyO3/setuptools-rust#351](https://github.com/PyO3/setuptools-rust/pull/351)
+  for more information about the workaround.
+
+- For more examples, see:
+  - [`hello-world`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world):
+    a more complete version of the code used in this tutorial that mixes both
+    `[[tool.setuptools-rust.ext-modules]]` and `[[tool.setuptools-rust.bins]]`
+    in a single distribution.
+  - [`html-py-ever`](https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever):
+    a more advanced example that uses Rust crates as dependencies.
+  - [`rust_with_cffi`](https://github.com/PyO3/setuptools-rust/tree/main/examples/rust_with_cffi):
+    uses both Rust and [CFFI](https://cffi.readthedocs.io/en/latest/).
+  - [`namespace_package`](https://github.com/PyO3/setuptools-rust/tree/main/examples/namespace_package):
+    integrates Rust-written modules into PEP 420 namespace packages.
+  - [`hello-world-script`](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world-script):
+    uses Rust only for creating binary executables, not library modules.

docs/building_wheels.md

@@ -1,8 +1,12 @@
 # Building wheels
 
-Because `setuptools-rust` is an extension to `setuptools`, the standard `setup.py bdist_wheel` command is used to build distributable wheels. These wheels can be uploaded to PyPI using standard tools such as [twine](https://github.com/pypa/twine).
+Because `setuptools-rust` is an extension to `setuptools`, the standard [`python -m build`](https://pypa-build.readthedocs.io/en/stable/) command
+(or [`pip wheel --no-deps . --wheel-dir dist`](https://pip.pypa.io/en/stable/cli/pip_wheel/)) can be used to build distributable wheels.
+These wheels can be uploaded to PyPI using standard tools such as [twine](https://github.com/pypa/twine).
 
-`setuptools-rust` supports building for the [PEP 384](https://www.python.org/dev/peps/pep-0384/) "stable" (aka "limited") API when the `--py-limited-api` option is passed to `setup.py bdist_wheel`. If using PyO3 bindings for `RustExtension`, then the correct [`pyo3/abi3`](https://pyo3.rs/v0.14.5/features.html#abi3) sub-feature is automatically enabled. In this way, abi3 wheels can be uploaded to make package distributors' roles easier, and  package users installing from source with `python setup.py install` can use optimizations specific to their Python version.
+`setuptools-rust` supports building for the [PEP 384](https://www.python.org/dev/peps/pep-0384/) "stable" (aka "limited") API when the `py_limited_api` option is set on the `[bdist_wheel]` section of `setup.cfg`.
+If using PyO3 bindings for `RustExtension`, then the correct [`pyo3/abi3`](https://pyo3.rs/v0.14.5/features.html#abi3) sub-feature is automatically enabled.
+In this way, abi3 wheels can be uploaded to make package distributors' roles easier, and package users installing from source with `pip install .` can use optimizations specific to their Python version.
 
 This chapter of the documentation explains two possible ways to build wheels for multiple Python versions below.
 
@@ -52,7 +56,7 @@ It is possible to use any of the `manylinux` docker images: `manylinux1`, `manyl
 
 ### Binary wheels on macOS
 
-For building wheels on macOS it is sufficient to run the `bdist_wheel` command, i.e. `setup.py bdist_wheel`.
+For building wheels on macOS it is sufficient to use one of the default `python -m build` or `pip wheel --no-deps . --wheel-dir dist` commands.
 
 To build `universal2` wheels set the `ARCHFLAGS` environment variable to contain both `x86_64` and `arm64`, for example `ARCHFLAGS="-arch x86_64 -arch arm64"`. Wheel-building solutions such as [`cibuildwheel`][cibuildwheel] set this environment variable automatically.
 

docs/index.md

@@ -4,6 +4,7 @@
    :hidden:
 
    README.md
+   setuppy_tutorial
    building_wheels
    reference
 ```

docs/setuppy_tutorial.md

@@ -0,0 +1,130 @@
+# Usage with `setup.py`
+
+While `pyproject.toml`-based configuration will be enough for most projects,
+sometimes you may need to use custom logic and imperative programming during the build.
+For those scenarios, `setuptools` also allows you to specify project configuration
+via `setup.py` in addition to `pyproject.toml`.
+
+The following is a very basic tutorial that shows how to use `setuptools-rust` in
+your `setup.py`.
+
+
+## Basic implementation files
+
+Let's start by assuming that you already have a bunch of Python and Rust files[^1]
+that you would like to package for distribution in PyPI inside of a project directory
+named `hello-world-setuppy`[^2][^3]:
+
+[^1]: To know more about how to write Rust to be integrated into Python packages,
+      please have a look on the [PyO3 docs](https://pyo3.rs)
+[^2]: You can have a look on the
+      [examples/hello-world-setuppy](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world-setuppy)
+      directory in the `setuptools-rust` repository.
+[^3]: If you are an experienced Python or Rust programmer, you may notice that we
+      avoid using the `src` directory and explicitly instruct Setuptools and Cargo to
+      look into the `python` and `rust` directories respectively.
+      Since both Python and Rust ecosystem will try to claim the `src` directory as
+      their default, we prefer to be explicit and avoid confusion.
+
+
+```
+hello-world-setuppy
+├── Cargo.lock
+├── Cargo.toml
+├── python
+│   └── hello_world
+│       └── __init__.py
+└── rust
+    └── lib.rs
+```
+
+```{literalinclude} ../examples/hello-world-setuppy/python/hello_world/__init__.py
+   :language: python
+```
+
+```{literalinclude} ../examples/hello-world-setuppy/rust/lib.rs
+   :language: rust
+```
+
+```{literalinclude} ../examples/hello-world-setuppy/Cargo.toml
+   :language: toml
+```
+
+
+## Adding files to support packaging
+
+Now we start by adding a `pyproject.toml` which tells anyone that wants to use
+our project to use `setuptools` and `setuptools-rust` to build it:
+
+```{literalinclude} ../examples/hello-world-setuppy/pyproject.toml
+   :language: toml
+```
+
+… and a [`setup.py` configuration file](https://setuptools.pypa.io/en/latest/references/keywords.html)
+that tells Setuptools how to build the Rust extensions using our `Cargo.toml` and `setuptools-rust`:
+
+```{literalinclude} ../examples/hello-world-setuppy/setup.py
+   :language: python
+```
+
+For a complete reference of the options supported by the `RustExtension` class, see the
+[API reference](https://setuptools-rust.readthedocs.io/en/latest/reference.html).
+
+
+We also add a [`MANIFEST.in` file](https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html)
+to control which files we want in the source distribution[^4]:
+
+```{literalinclude} ../examples/hello-world-setuppy/MANIFEST.in
+```
+
+[^4]: Alternatively you can also use `setuptools-scm` to add all the files under revision control
+      to the `sdist`, see the [docs](https://pypi.org/project/setuptools-scm/) for more information.
+
+
+## Testing the extension
+
+With these files in place, you can install the project in a virtual environment
+for testing and making sure everything is working correctly:
+
+
+```powershell
+# cd hello-world-setuppy
+python3 -m venv .venv
+source .venv/bin/activate  # on Linux or macOS
+.venv\Scripts\activate     # on Windows
+python -m pip install -e .
+python -c 'import hello_world; print(hello_world.sum_as_string(5, 7))'  # => 12
+# ... better write some tests with pytest ...
+```
+
+
+## Next steps and final remarks
+
+- When you are ready to distribute your project, have a look on
+  [the notes in the documentation about building wheels](https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html).
+
+- You can also use a [`RustBin`](https://setuptools-rust.readthedocs.io/en/latest/reference.html) object
+  (instead of a `RustExtension`), if you want to distribute a binary executable
+  written in Rust (instead of a library that can be imported by the Python runtime).
+  Note however that distributing both library and executable (or multiple executables),
+  may significantly increase the size of the
+  [wheel](https://packaging.python.org/en/latest/glossary/#term-Wheel)
+  file distributed by the
+  [package index](https://packaging.python.org/en/latest/glossary/#term-Package-Index)
+  and therefore increase build, download and installation times.
+  Another approach is to use a Python entry-point that calls the Rust
+  implementation (exposed via PyO3 bindings).
+  See the [hello-world](https://github.com/PyO3/setuptools-rust/tree/main/examples/hello-world)
+  example for more insights.
+
+- If want to include both `RustBin` and `RustExtension` same macOS wheel, you might have
+  to manually add an extra `build.rs` file, see [PyO3/setuptools-rust#351](https://github.com/PyO3/setuptools-rust/pull/351)
+  for more information about the workaround.
+
+- Since the adoption of {pep}`517`, running `python setup.py ...` directly as a CLI tool is
+  [considered deprecated](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html).
+  Nevertheless, `setup.py` can be safely used as a configuration file
+  (the same way `conftest.py` is used by `pytest` or `noxfile.py` is used by `nox`).
+  There is a different mindset that comes with this change, though:
+  for example, it does not make sense to use `sys.exit(0)` in a `setup.py` file
+  or use a overarching `try...except...` block to re-run a failed build with different parameters.