refactor: Address some issues with enums and overhaul documentation

Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]>

Author: narendasan

dev_dep_versions.yml

@@ -1,3 +1,2 @@
-__version__: "2.5.0.dev0"
 __cuda_version__: "12.4"
 __tensorrt_version__: "10.0.1"

docsrc/conf.py

@@ -25,7 +25,7 @@
 # -- Project information -----------------------------------------------------
 
 project = "Torch-TensorRT"
-copyright = "2022, NVIDIA Corporation"
+copyright = "2024, NVIDIA Corporation"
 author = "NVIDIA Corporation"
 
 version = f"v{torch_tensorrt.__version__}"
@@ -151,6 +151,9 @@
     "master_doc": True,
     "version_info": {
         "main": "https://pytorch.org/TensorRT/",
+        "v2.3.0": "https://pytorch.org/TensorRT/v2.3.0",
+        "v2.2.0": "https://pytorch.org/TensorRT/v2.2.0",
+        "v2.1.0": "https://pytorch.org/TensorRT/v2.1.0",
         "v1.4.0": "https://pytorch.org/TensorRT/v1.4.0",
         "v1.3.0": "https://pytorch.org/TensorRT/v1.3.0",
         "v1.2.0": "https://pytorch.org/TensorRT/v1.2.0",
@@ -186,6 +189,8 @@
 
 nbsphinx_execute = "never"
 
+autodoc_member_order = "groupwise"
+
 # -- A patch that prevents Sphinx from cross-referencing ivar tags -------
 # See http://stackoverflow.com/a/41184353/3343043
 

docsrc/getting_started/getting_started_with_windows.rst

@@ -1,15 +1,15 @@
 .. _installation:
 
 Installation
-=============
+##################
 
 Precompiled Binaries
-*********************
+---------------------
 
-Torch-TensorRT 2.x is centered primarily around Python. As such, precompiled releases can be found on pypi.org
+Torch-TensorRT 2.x is centered primarily around Python. As such, precompiled releases can be found on `pypi.org <https://pypi.org/project/torch-tensorrt/>`_
 
 Dependencies
----------------
+~~~~~~~~~~~~~~
 
 You need to have CUDA, PyTorch, and TensorRT (python package is sufficient) installed to use Torch-TensorRT
 
@@ -18,16 +18,18 @@ You need to have CUDA, PyTorch, and TensorRT (python package is sufficient) inst
 
 
 Installing Torch-TensorRT
----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 You can install the python package using
 
 .. code-block:: sh
 
     python -m pip install torch torch-tensorrt tensorrt
 
+Packages are uploaded for Linux on x86 and Windows
+
 Installing Torch-TensorRT for a specific CUDA version
---------------------------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Similar to PyTorch, Torch-TensorRT has builds compiled for different versions of CUDA. These are distributed on PyTorch's package index
 
@@ -38,7 +40,7 @@ For example CUDA 11.8
     python -m pip install torch torch-tensorrt tensorrt --extra-index-url https://download.pytorch.org/whl/cu118
 
 Installing Nightly Builds
----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Torch-TensorRT distributed nightlies targeting the PyTorch nightly. These can be installed from the PyTorch nightly package index (separated by CUDA version)
 
@@ -51,19 +53,22 @@ Torch-TensorRT distributed nightlies targeting the PyTorch nightly. These can be
 .. _bin-dist:
 
 C++ Precompiled Binaries (TorchScript Only)
---------------------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Precompiled tarballs for releases are provided here: https://github.com/pytorch/TensorRT/releases
 
 .. _compile-from-source:
 
 Compiling From Source
-******************************************
+------------------------
+
+Building on Linux
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. _installing-deps:
 
-Dependencies for Compilation
--------------------------------
+Dependencies
+^^^^^^^^^^^^^^
 
 * Torch-TensorRT is built with **Bazel**, so begin by installing it.
 
@@ -120,7 +125,7 @@ If you have a local version of TensorRT installed, this can be used as well by c
 
 
 Building the Package
----------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Once the WORKSPACE has been configured properly, all that is required to build torch-tensorrt is the following command
 
@@ -135,12 +140,41 @@ To build the wheel file
 
         python -m pip wheel --no-deps --pre . --extra-index-url https://download.pytorch.org/whl/nightly/cu124 -w dist
 
+Additional Build Options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Some features in the library are optional and allow builds to be lighter or more portable.
+
+Python Only Distribution
+............................
+
+There are multiple features of the library which require C++ components to be enabled. This includes both the TorchScript frontend which accepts TorchScript modules for compilation
+and the Torch-TensorRT runtime, the default executor for modules compiled with Torch-TensorRT, be it with the TorchScript or Dynamo frontend.
+
+In the case you may want a build which does not require C++ you can disable these features and avoid building these compoents. As a result, the only available runtime will be the Python based on
+which has implications for features like serialization.
+
+.. code-block:: sh
+
+    PYTHON_ONLY=1 python -m pip install --pre . --extra-index-url https://download.pytorch.org/whl/nightly/cu124
+
+
+No TorchScript Frontend
+............................
+
+The TorchScript frontend is a legacy feature of Torch-TensorRT which is now in maintance as TorchDynamo has become the prefered compiler technology for this project. It contains quite a bit
+of C++ code that is no longer necessary for most users. Therefore you can exclude this component from your build to speed up build times. The C++ based runtime will still be available to use.
+
+.. code-block:: sh
+
+    NO_TORCHSCRIPT=1 python -m pip install --pre . --extra-index-url https://download.pytorch.org/whl/nightly/cu124
+
 
-Building the C++ Library (TorchScript Only)
-------------------------------
+Building the C++ Library Standalone (TorchScript Only)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Release Build
-^^^^^^^^^^^^^^^^^^^^^^^^
+............................
 
 .. code-block:: shell
 
@@ -151,7 +185,7 @@ A tarball with the include files and library can then be found in ``bazel-bin``
 .. _build-from-archive-debug:
 
 Debug Build
-^^^^^^^^^^^^^^^^^^^^^^^^
+............................
 
 To build with debug symbols use the following command
 
@@ -162,7 +196,7 @@ To build with debug symbols use the following command
 A tarball with the include files and library can then be found in ``bazel-bin``
 
 Pre CXX11 ABI Build
-^^^^^^^^^^^^^^^^^^^^^^^^
+............................
 
 To build using the pre-CXX11 ABI use the ``pre_cxx11_abi`` config
 
@@ -204,8 +238,45 @@ recommended commands:
 
     NOTE: For all of the above cases you must correctly declare the source of PyTorch you intend to use in your WORKSPACE file for both Python and C++ builds. See below for more information
 
-**Building with CMake** (TorchScript Only)
--------------------------------------------
+
+
+Building on Windows
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+* Microsoft VS 2022 Tools
+* Bazelisk
+* CUDA
+
+
+Build steps
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Open the app "x64 Native Tools Command Prompt for VS 2022" - note that Admin priveleges may be necessary
+* Ensure Bazelisk (Bazel launcher) is installed on your machine and available from the command line. Package installers such as Chocolatey can be used to install Bazelisk
+* Install latest version of Torch (i.e. with ``pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu124``)
+* Clone the Torch-TensorRT repository and navigate to its root directory
+* Run ``pip install ninja wheel setuptools``
+* Run ``pip install --pre -r py/requirements.txt``
+* Run ``set DISTUTILS_USE_SDK=1``
+* Run ``python setup.py bdist_wheel``
+* Run ``pip install dist/*.whl``
+
+Advanced setup and Troubleshooting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the ``WORKSPACE`` file, the ``cuda_win``, ``libtorch_win``, and ``tensorrt_win`` are Windows-specific modules which can be customized. For instance, if you would like to build with a different version of CUDA, or your CUDA installation is in a non-standard location, update the `path` in the `cuda_win` module.
+
+Similarly, if you would like to use a different version of pytorch or tensorrt, customize the `urls` in the ``libtorch_win`` and ``tensorrt_win`` modules, respectively.
+
+Local versions of these packages can also be used on Windows. See ``toolchains\\ci_workspaces\\WORKSPACE.win.release.tmpl`` for an example of using a local version of TensorRT on Windows.
+
+
+Alternative Build Systems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Building with CMake (TorchScript Only)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 It is possible to build the API libraries (in cpp/) and the torchtrtc executable using CMake instead of Bazel.
 Currently, the python API and the tests cannot be built with CMake.
@@ -233,11 +304,12 @@ A few useful CMake options include:
             [-DCMAKE_BUILD_TYPE=Debug|Release]
         cmake --build <build directory>
 
-**Building Natively on aarch64 (Jetson)**
--------------------------------------------
+
+Building Natively on aarch64 (Jetson)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Prerequisites
-^^^^^^^^^^^^^^
+............................
 
 Install or compile a build of PyTorch/LibTorch for aarch64
 
@@ -247,7 +319,7 @@ NVIDIA hosts builds the latest release branch for Jetson here:
 
 
 Enviorment Setup
-^^^^^^^^^^^^^^^^^
+............................
 
 To build natively on aarch64-linux-gnu platform, configure the ``WORKSPACE`` with local available dependencies.
 
@@ -279,7 +351,7 @@ use that library, set the paths to the same path but when you compile make sure
 
 
 Compile C++ Library and Compiler CLI
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+........................................................
 
     NOTE: Due to shifting dependency locations between Jetpack 4.5 and 4.6 there is a now a flag to inform bazel of the Jetpack version
 
@@ -295,7 +367,7 @@ Compile Torch-TensorRT library using bazel command:
    bazel build //:libtorchtrt --platforms //toolchains:jetpack_5.0
 
 Compile Python API
-^^^^^^^^^^^^^^^^^^^^
+............................
 
     NOTE: Due to shifting dependencies locations between Jetpack 4.5 and newer Jetpack verisons there is now a flag for ``setup.py`` which sets the jetpack version (default: 5.0)
 
@@ -307,4 +379,4 @@ Compile the Python API using the following command from the ``//py`` directory:
 
 If you have a build of PyTorch that uses Pre-CXX11 ABI drop the ``--use-cxx11-abi`` flag
 
-If you are building for Jetpack 4.5 add the ``--jetpack-version 5.0`` flag
+If you are building for Jetpack 4.5 add the ``--jetpack-version 5.0`` flag

docsrc/getting_started/installation.rst

@@ -0,0 +1,74 @@
+.. _quick_start:
+
+Quick Start
+##################
+
+Option 1: torch.compile
+-------------------------
+
+You can use Torch-TensorRT anywhere you use torch.compile:
+
+.. code-block:: py
+
+    import torch
+    import torch_tensorrt
+
+    model = MyModel().eval().cuda() # define your model here
+    x = torch.randn((1, 3, 224, 224)).cuda() # define what the inputs to the model will look like
+
+    optimized_model = torch.compile(model, backend="tensorrt")
+    optimized_model(x) # compiled on first run
+
+    optimized_model(x) # this will be fast!
+
+
+Option 2: Export
+-------------------------
+
+If you want to optimize your model ahead-of-time and/or deploy in a C++ environment, Torch-TensorRT provides an export-style workflow that serializes an optimized module. This module can be deployed in PyTorch or with libtorch (i.e. without a Python dependency).
+
+Step 1: Optimize + serialize
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: py
+
+    import torch
+    import torch_tensorrt
+
+    model = MyModel().eval().cuda() # define your model here
+    inputs = [torch.randn((1, 3, 224, 224)).cuda()] # define a list of representative inputs here
+
+    trt_gm = torch_tensorrt.compile(model, ir="dynamo", inputs)
+    torch_tensorrt.save(trt_gm, "trt.ep", inputs=inputs) # PyTorch only supports Python runtime for an ExportedProgram. For C++ deployment, use a TorchScript file
+    torch_tensorrt.save(trt_gm, "trt.ts", output_format="torchscript", inputs=inputs)
+
+Step 2: Deploy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deployment in Python:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+.. code-block:: py
+
+    import torch
+    import torch_tensorrt
+
+    inputs = [torch.randn((1, 3, 224, 224)).cuda()] # your inputs go here
+
+    # You can run this in a new python session!
+    model = torch.export.load("trt.ep").module()
+    # model = torch_tensorrt.load("trt.ep").module() # this also works
+    model(*inputs)
+
+Deployment in C++:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: c++
+
+    #include "torch/script.h"
+    #include "torch_tensorrt/torch_tensorrt.h"
+
+    auto trt_mod = torch::jit::load("trt.ts");
+    auto input_tensor = [...]; // fill this with your inputs
+    auto results = trt_mod.forward({input_tensor});