Description
Bug Description
Hi, based on some past posts like this and this and this, I was able to get a bazel build working for windows for the c++ library, with both model compilation and inference, by building torch_tensorrt.dll, which I believe includes both the runtime and the compilation plugins.
I am wondering what it would take for official windows support. I've attached a full patch, I think the declspec may not be needed.
Once this builds, all one has to do to use torch_tensorrt in their project is, link to torch_tensorrt.dll.if.lib and ensure torch_tensorrt.dll is loaded up with LoadLibrary as mentioned in other issue reports.
HMODULE hLib = LoadLibrary(TEXT("torch_tensorrt"));
if (hLib == NULL) {
std::cerr << "Library torch_tensorrt.dll not found" << std::endl;
exit(1);
}
More info about exact versions of libraries used:
cuda_11.8.0_522.06_windows.exe
TensorRT-8.6.1.6.Windows10.x86_64.cuda-11.8
libtorch-win-shared-with-deps-2.0.1+cu118
cudnn-windows-x86_64-8.8.0.121_cuda11-archive
Build command:
bazel-6.3.2-windows-x86_64.exe build //:libtorchtrt --compilation_mode opt
Environment
Build information about Torch-TensorRT can be found by turning on debug messages
- Torch-TensorRT Version (e.g. 1.0.0): 8.6.1.6
- PyTorch Version (e.g. 1.0): 2.0.1
- CPU Architecture:
- OS (e.g., Linux): Windows
- How you installed PyTorch (
conda
,pip
,libtorch
, source): libtorch, source - Build command you used (if compiling from source): bazel
- Are you using local sources or building from archives: latest github?
- Python version:
- CUDA version: 11.8
- GPU models and configuration: 3080Ti
- Any other relevant information:
Activity
jensdraht1999 commentedon Oct 16, 2023
@markivfb Can you use this with Pytorch? Or just native C/C++?
If yes, can you tell me, what you did with the dlls and where did you put them?
Justin62628 commentedon Oct 19, 2023
This script works on my cuda 11.7 too with torch 2.0.1.
torch_tensorrt.dll
is successfully built. Thanks for your marvelous work.jensdraht1999 commentedon Oct 19, 2023
@Justin62628 I have build torch_tensorrt.dll successfully too, but something more complicated. However: What do I actually do with these dll files for Pytorch?
Justin62628 commentedon Oct 21, 2023
so far I can only work with these dll files with libtorch.
Normally we should compile wheels of torch_tensorrt for python first, which means you are to exec
python setup.py install
from this repo to get the_C.pyd
extension for torch_tensorrt(the setup script will automatically get all those done for you, which installs torch_tensorrt python api to your site-pakcages), then you are able to import torch_tensorrt in your python project,I have managed to compile torch_tensorrt
_C.pyd
with a modified compilation scriptbazel build //:libtorchtrt --compilation_mode opt --features=windows_export_all_symbols
, but there seems to be some linking and symbols exporting mistakes, where I encountersexternal dlls not found
error when trying to import torch_tensorrt into python project like yoursjensdraht1999 commentedon Oct 23, 2023
@Justin62628 Could you share your wheel file for Python, so I could install this? You can upload as a image file here and then I can rename the extension back to .py and install it.
phineas-pta commentedon Nov 24, 2023
for me i also have to edit file
.bazelrc
: change-std=c++17
to/std:c++17
jensdraht1999 commentedon Dec 1, 2023
Did you manage to somehow work it with pytorch with python?
phineas-pta commentedon Dec 2, 2023
@jensdraht1999 wheel built successfully but unusable (dll linking error like #856)