Can please somebody give a clear explanation of how to install torch-tensorrt on Windows?

[ninono12345]

❓ Question

Hello,

I've encountered problems installing torch-tensorrt on Windows 10

No matter how I try, how many sources I look up to, there is no clear explanation on how to do everything. The documentation is vague, and because I am used to working with python code, which does everything for you, that is pip install... python code.py, and nothing more is required, I do not have as much experience with cmake, building libraries, files, and c++, which makes it very difficult to follow along the installation process.

Now I've tried to follow along instructions from the main page

pip install torch-tensorrt doesn't work
downloaded zip file of this repository; python setup.py install also doesn't work

installed bazel
modified the workspace, still nothing

tried to directly import into code py/torch-tensorrt - nothing

then inside the py folder opened command prompt ant typed in:

bazel build //:libtorchtrt --compilation_mode=dbg

and received this error:

`Starting local Bazel server and connecting to it...
INFO: Repository libtorch instantiated at:
D:/pyth/tensorrt-main/WORKSPACE:53:13: in
Repository rule http_archive defined at:
C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in
WARNING: Download from https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip failed: class com.google.devtools.build.lib.bazel.repository.downloader.ContentLengthMismatchException Bytes read 2210658461 but wanted 2501377827
ERROR: An error occurred during the fetch of repository 'libtorch':
Traceback (most recent call last):
File "C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827
ERROR: D:/pyth/tensorrt-main/WORKSPACE:53:13: fetching http_archive rule //external:libtorch: Traceback (most recent call last):
File "C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827
ERROR: D:/pyth/tensorrt-main/core/util/logging/BUILD:13:11: //core/util/logging:logging depends on @libtorch//:libtorch in repository @libtorch which failed to fetch. no such package '@libtorch//': java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827
ERROR: Analysis of target '//:libtorchtrt' failed; build aborted:
INFO: Elapsed time: 458.697s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (64 packages loaded, 413 targets configured)
Fetching https://download.pytorch.org/...orch-cxx11-abi-shared-with-deps-latest.zip; 2.1 GiB (2,210,121,825B) 446s

And also tried some other things, I cannot remember, but unsuccessfully.

THANK YOU FOR YOUR HELP IN ADVANCE

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• PyTorch Version: 2.1.2+cu121
• OS : Windows 10
• I am running python and pytorch straight from Windows, without any environment
• Python version: 3.10.13
• CUDA version: 12.1 update 1
• GPU models and configuration: GTX 1660 TI

[gs-olive]

Hello - we are working on a pip-installable Windows distribution for which a sample can be found on the windows_CI branch. This branch currently only supports our Dynamo paths (ir=dynamo, ir=torch_compile) and a limited TensorRT runtime (though #1943 will improve this runtime constraint). It does not require bazel to install.

In order to use it, the Torch distribution requires this patch: pytorch/pytorch#111313, to enable Dynamo experimentally on Windows, since Torch does not yet fully support Dynamo on Windows. Then, once cloned, running python setup.py install from the root of the directory would begin the package installation. I have tested it successfully on Python 3.10.

[ninono12345]

Hello again, thank you for your reply, @gs-olive, and I apologize, for replying so late. Thank you for pointing me to the windows_CI branch, I haven't noticed it then.

Now if I use git clone https://github.com/pytorch/TensorRT.git (even though this is the same link to non windows_CI, it's on windows_CI branch) and run python setup.py install I get Could not find bazel in PATH . This tells me that it is not the windows_CI being downloaded.

Now I downloaded the zip file format, I get the TensorRT windows_CI, when i run setup.py, i get an error, that this is not a git repository. I ran the command git init, that error is gone, but now I receive this error:

fatal: Needed a single revision Traceback (most recent call last): File "D:\pyth\TensorRT-windows_CI\setup.py", line 104, in <module> __version__ = f"{get_base_version()}.dev0+{get_git_revision_short_hash()}" File "D:\pyth\TensorRT-windows_CI\setup.py", line 43, in get_git_revision_short_hash subprocess.check_output(["git", "rev-parse", "--short", "HEAD"]) File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['git', 'rev-parse', '--short', 'HEAD']' returned non-zero exit status 128

specifically the line fatal: Needed a single revision I do not understand, and once again, can't find any help online...

Thank you for your help

[gs-olive]

Hello - thanks for the follow-up. To clone a specific branch from the command line, I use:

git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git

This should resolve the git issues related to that repository. Then, python setup.py install should avoid bazel.

[ninono12345]

Hi again @gs-olive, I've successfully installed it like you said, but now when i try to import torch_tensorrt, I get this error:

C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable.
warnings.warn(
Traceback (most recent call last):
File "D:\pyth\pytracking-master\band5.py", line 22, in
import torch_tensorrt
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_init_.py", line 84, in
from torch_tensorrt._compile import * # noqa: F403
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_compile.py", line 24, in
from torch_tensorrt.dynamo.compiler import compile as dynamo_compile
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo_init.py", line 10, in
from .compiler import compile
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo_compiler.py", line 45, in
from torch_tensorrt.dynamo.lowering import apply_lowering_passes, get_decompositions
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering_init.py", line 7, in
from .repair_input_aliasing import repair_input_aliasing
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering_repair_input_aliasing.py", line 4, in
from torch_tensorrt.dynamo.lowering.passes.pass_utils import get_tensor_placeholders
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes_init.py", line 1, in
from ._aten_lowering_pass import *
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes_aten_lowering_pass.py", line 6, in
from .constant_folding import constant_fold
File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes\constant_folding.py", line 16, in
from torch._inductor.constant_folding import (
ModuleNotFoundError: No module named 'torch._inductor.constant_folding'

[gs-olive]

Thanks for testing that out - I have seen this error before; it relates to the torch version being used. The branch is based on the latest Torch nightly, which has the torch._inductor.constant_folding module, but Torch 2.1.2 does not have that module. I just updated the branch to fix that issue, so windows_CI should get past that issue now if you pull the latest changes and reinstall.

[ninono12345]

Thank you @gs-olive for your fast reply. Now I'm really confused. I installed torch_tensorrt, now when I try to import torch_tensorrt I get this error and the program stops:
C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable. warnings.warn(

for some reason at first when I installed it I was having some problems, so I did pip uninstall torch_tensorrt, and then ran again python setup.py install, and I get the error

Or for it to work ok, I need to download the pytorch 2.2.0 nightly?

[jensdraht1999]

@gs-olive Hello, I cloned it again as well and it has the same error with "returned non-zero exit status 128" in the end. Can we have some kind of a step by step guide, what exactly we have to do?

1.) Download this that and that.
2.) Add this that and that to the PATH
3.) Then install Python 3.10
4.) Then run setup.py

Something like this I mean.

Thanks a lot.

[HolyWu]

I still get the following error using the latest commit (3c099ef at the time of writing) from the windows_CI branch when trying to import torch_tensorrt.

Warning (from warnings module):
  File "C:\Python311\Lib\site-packages\torch_tensorrt\_Device.py", line 19
    warnings.warn(
UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable.
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    import torch_tensorrt
  File "C:\Python311\Lib\site-packages\torch_tensorrt\__init__.py", line 90, in <module>
    from torch_tensorrt.logging import *
  File "C:\Python311\Lib\site-packages\torch_tensorrt\logging.py", line 4, in <module>
    from torch_tensorrt._C import (
ModuleNotFoundError: No module named 'torch_tensorrt._C'

[jensdraht1999]

@HolyWu I don't know exactly why the problem of you is happening, because I'm not the dev of this, but perhaps you could try Python 3.10 because the dev said it.

[jensdraht1999]

@gs-olive Hello, I cloned it again as well and it has the same error with "returned non-zero exit status 128" in the end. Can we have some kind of a step by step guide, what exactly we have to do?

1.) Download this that and that. 2.) Add this that and that to the PATH 3.) Then install Python 3.10 4.) Then run setup.py

Something like this I mean.

Thanks a lot.

By the way I have solved it myself. Here is how:

1.) Before all of this, please install Python 3.11.
2.) And then do this: "pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121"
3.) Download Cuda 12.1 (only this)
4.) Download Cudnn
5.) Download Nvida Tensorrt
6.) Install Cuda
7.) Copy the cudnn content(whats in it) to the folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1.
8.) Put Nvidia Tensorrt to the Enviroment Path.
9.) Clone the Pytorch/Tensorrt via this command ONLY (nothing else does not work):
git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git
10.) Then you have to do: "cd folderwhereithasbeencopied"
11.) Then you have to do: "python setup.py"

Setup should install the library.

However I have not tried more than this for now. Some kind of example would be helpful?

[gs-olive]

Hello - thanks for the comments on this thread. To address them:

@ninono12345 - the UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. is expected currently, since the version of Torch-TensorRT in the windows_CI branch does not support TorchScript; it only supports the ir="dynamo" and ir="torch_compile" IRs when using this patch: pytorch/pytorch#111313.

@HolyWu - thanks for this catch/report - torch_tensorrt._C is not installed in the Windows build on windows_CI, so that import should not be present in this experimental branch. I've added a commit to remove it.

@jensdraht1999 - the steps provided are very close to what I am using on this experimental branch to get a working build. For a detailed set of steps I am currently using with this experimental branch, here is what I have:

Steps for Dynamo-Only, Python-Runtime-Only Windows build using windows_CI Branch

1. Ensure CUDA is Installed (preferably 12.x). See official CUDA installation steps for Windows if not.
2. Install your preferred TensorRT Python version. Currently, I use pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com tensorrt==9.2.0.post12.dev5. You can also install other TensorRT versions with the Python bindings using the official TensorRT Windows installation docs, which currently do not use pip.
3. Install the nightly distribution of Torch, for instance: pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121
4. Apply this patch: Remove dynamo suported check for Windows. pytorch#111313. Since Torch does not officially support Windows for Dynamo, but most of the required functionality is there, we use this patch as an experimental workaround while the pending issues for torch.compile on Windows are addressed. I currently apply the patch by modifying the torch/_dynamo/eval_frame.py file, for which the location can be found by running pip show torch.
5. Clone the branch: git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git
6. cd into the directory containing the cloned repo. For instance, cd TensorRT
7. Run python setup.py install to install torch_tensorrt
8. Open a Python interpreter and run import torch; import torch_tensorrt. There may be warnings about TorchScript and Torchvision, but the import should succeed.
9. Try an example run, for instance python examples/dynamo/torch_compile_resnet_example.py, which runs ResNet using this tutorial.

Note: Windows support for the Dynamo paths is in an experimental state, and there are no guarantees on functionality + it may not work in some compilation cases since it is under active development.
Note: To run torch_tensorrt.compile on Windows using the windows_CI branch, it is required to use ir="dynamo" or ir="torch_compile" and use_python_runtime=True.

[jensdraht1999]

@gs-olive I have done all of this. But Step 8 is a problem. Step 9 doesn't really work I think. Please look at the pictures. Would you recommend me to use Python 3.10? By the way I have totally reinstalled Python 3.11 deleted all the sitepackage folder and more to have it all clean and also cuda 12.1.

Step 8:

Step 9: If I run the example file without changing anything:

log.txt

What could be the problem?

[gs-olive]

Hi - thanks for the follow-up. I do not think Python 3.10 vs 3.11 should make a difference here, though I am using Python 3.10. Regarding the attached image, in the IDE on the left, it seems that there is an error finding a cudnn file, whereas in the IDE on the right, there is just the expected warning. Did anything change between the two runs to make the import succeed, as it does on the left?

Regarding the error logs, that is due to the version of the example which I had linked in the original message. I have fixed the link now, and it uses the correct configuration parameters for Windows. It is linked here as well for convenience.