fix(//core/runtime): Support more delimiter variants #1004

narendasan · 2022-04-27T03:31:05Z

Description

Fixes an issue where different classes of delimiters would not properly be handled when deserializing TRT engines.

Fixes #930
Fixes #982

Type of change

Please delete options that are not relevant and/or add your own.

Bug fix (non-breaking change which fixes an issue)

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 5ba3a56..70b51e0 100644
--- a/workspace/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -63,7 +63,10 @@ TRTEngine::TRTEngine(std::string mod_name, std::string serialized_engine, CudaDe
    auto delim = bind_name.find(".");
    if (delim == std::string::npos) {
      delim = bind_name.find("_");
-      TORCHTRT_CHECK(delim != std::string::npos, "Unable to determine binding index for input " << bind_name << "\nEnsure module was compile with Torch-TensorRT.ts");
+      TORCHTRT_CHECK(
+          delim != std::string::npos,
+          "Unable to determine binding index for input " << bind_name
+                                                         << "\nEnsure module was compile with Torch-TensorRT.ts");
    }

    std::string idx_s = bind_name.substr(delim + 1);
@@ -97,16 +100,16 @@ std::string TRTEngine::to_str() const {
  ss << "  Name: " << name << std::endl;
  ss << "  Inputs: [" << std::endl;
  for (uint64_t i = 0; i < num_io.first; i++) {
-  ss << "    id: " << i << std::endl;
-  ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
-  ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
+    ss << "    id: " << i << std::endl;
+    ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
+    ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Outputs: [" << std::endl;
  for (uint64_t o = 0; o < num_io.second; o++) {
-  ss << "    id: " << o << std::endl;
-  ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
-  ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
+    ss << "    id: " << o << std::endl;
+    ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
+    ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Device: " << device_info << std::endl;
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 5ba3a56..70b51e0 100644
--- a/workspace/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -63,7 +63,10 @@ TRTEngine::TRTEngine(std::string mod_name, std::string serialized_engine, CudaDe
    auto delim = bind_name.find(".");
    if (delim == std::string::npos) {
      delim = bind_name.find("_");
-      TORCHTRT_CHECK(delim != std::string::npos, "Unable to determine binding index for input " << bind_name << "\nEnsure module was compile with Torch-TensorRT.ts");
+      TORCHTRT_CHECK(
+          delim != std::string::npos,
+          "Unable to determine binding index for input " << bind_name
+                                                         << "\nEnsure module was compile with Torch-TensorRT.ts");
    }

    std::string idx_s = bind_name.substr(delim + 1);
@@ -97,16 +100,16 @@ std::string TRTEngine::to_str() const {
  ss << "  Name: " << name << std::endl;
  ss << "  Inputs: [" << std::endl;
  for (uint64_t i = 0; i < num_io.first; i++) {
-  ss << "    id: " << i << std::endl;
-  ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
-  ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
+    ss << "    id: " << i << std::endl;
+    ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
+    ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Outputs: [" << std::endl;
  for (uint64_t o = 0; o < num_io.second; o++) {
-  ss << "    id: " << o << std::endl;
-  ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
-  ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
+    ss << "    id: " << o << std::endl;
+    ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
+    ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Device: " << device_info << std::endl;
ERROR: Some files do not conform to style guidelines

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 5ba3a56..70b51e0 100644
--- a/workspace/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -63,7 +63,10 @@ TRTEngine::TRTEngine(std::string mod_name, std::string serialized_engine, CudaDe
    auto delim = bind_name.find(".");
    if (delim == std::string::npos) {
      delim = bind_name.find("_");
-      TORCHTRT_CHECK(delim != std::string::npos, "Unable to determine binding index for input " << bind_name << "\nEnsure module was compile with Torch-TensorRT.ts");
+      TORCHTRT_CHECK(
+          delim != std::string::npos,
+          "Unable to determine binding index for input " << bind_name
+                                                         << "\nEnsure module was compile with Torch-TensorRT.ts");
    }

    std::string idx_s = bind_name.substr(delim + 1);
@@ -97,16 +100,16 @@ std::string TRTEngine::to_str() const {
  ss << "  Name: " << name << std::endl;
  ss << "  Inputs: [" << std::endl;
  for (uint64_t i = 0; i < num_io.first; i++) {
-  ss << "    id: " << i << std::endl;
-  ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
-  ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
+    ss << "    id: " << i << std::endl;
+    ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
+    ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Outputs: [" << std::endl;
  for (uint64_t o = 0; o < num_io.second; o++) {
-  ss << "    id: " << o << std::endl;
-  ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
-  ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
+    ss << "    id: " << o << std::endl;
+    ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
+    ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Device: " << device_info << std::endl;
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 5ba3a56..70b51e0 100644
--- a/workspace/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -63,7 +63,10 @@ TRTEngine::TRTEngine(std::string mod_name, std::string serialized_engine, CudaDe
    auto delim = bind_name.find(".");
    if (delim == std::string::npos) {
      delim = bind_name.find("_");
-      TORCHTRT_CHECK(delim != std::string::npos, "Unable to determine binding index for input " << bind_name << "\nEnsure module was compile with Torch-TensorRT.ts");
+      TORCHTRT_CHECK(
+          delim != std::string::npos,
+          "Unable to determine binding index for input " << bind_name
+                                                         << "\nEnsure module was compile with Torch-TensorRT.ts");
    }

    std::string idx_s = bind_name.substr(delim + 1);
@@ -97,16 +100,16 @@ std::string TRTEngine::to_str() const {
  ss << "  Name: " << name << std::endl;
  ss << "  Inputs: [" << std::endl;
  for (uint64_t i = 0; i < num_io.first; i++) {
-  ss << "    id: " << i << std::endl;
-  ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
-  ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
+    ss << "    id: " << i << std::endl;
+    ss << "      shape: " << exec_ctx->getBindingDimensions(i) << std::endl;
+    ss << "      dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(i)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Outputs: [" << std::endl;
  for (uint64_t o = 0; o < num_io.second; o++) {
-  ss << "    id: " << o << std::endl;
-  ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
-  ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
+    ss << "    id: " << o << std::endl;
+    ss << "    shape: " << exec_ctx->getBindingDimensions(o) << std::endl;
+    ss << "    dtype: " << util::TRTDataTypeToScalarType(exec_ctx->getEngine().getBindingDataType(o)) << std::endl;
  }
  ss << "  ]" << std::endl;
  ss << "  Device: " << device_info << std::endl;
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to Python style guidelines

bowang007 · 2022-04-27T21:33:32Z

@narendasan looks good, however, I rebased locally it still cannot resolve the issue that I have now. There is no delimiter such as '.' or '_' in the bind_name that I get in the model, maybe I still need to look into what happens in this model.
At least the error message is clearer for me now.

narendasan · 2022-04-27T21:35:19Z

We need inputs to have an index because tensorrt does not guarantee binding order. So you will need to modify whatever is coming in to have indexes

bowang007 · 2022-04-27T23:14:10Z

I checked the log where we have this in conversion process:

DEBUG: [Torch-TensorRT TorchScript Conversion Context] - Input Dimension Specs: {
    argument_1.1 : Input(shape: [3, 1024, 1024], dtype: Float32, format: NCHW\Contiguous\Linear),}
INFO: [Torch-TensorRT TorchScript Conversion Context] - Adding Input argument_1.1 (named: input_0): Input(shape: [3, 1024, 1024], dtype: Float32, format: NCHW\Contiguous\Linear) in engine (conversion.AddInputs)

it seems that we have input_0, then there comes serialization and deserialization, after deserialization the binding name became 0 for input_0.
How can we modify this? since this whole serialization and deserialization are done through TensorRT. @narendasan

narendasan · 2022-04-28T19:19:45Z

The binding is just 0?

bowang007 · 2022-04-28T21:49:40Z

The binding is just 0?

yes.

narendasan · 2022-04-29T00:32:24Z

hmm there is something odd there i feel. I would expect that the name would be input_0 or something

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

narendasan · 2022-05-06T01:38:47Z

/blossom-ci

deserialization NOTE: This does not fully address the deserialization issue as the root cause is TensorRT modifies the input binding names which is leading to these cases of stoi errors. Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

Code conforms to Python style guidelines

github-actions

There are some changes that do not conform to C++ style guidelines:

diff --git a/workspace/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index dbef11b..26c755c 100644
--- a/workspace/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -66,8 +66,9 @@ TRTEngine::TRTEngine(std::string mod_name, std::string serialized_engine, CudaDe
      delim = bind_name.find("_");
      TORCHTRT_CHECK(
          delim != std::string::npos,
-          "Unable to determine binding index for input " << bind_name
-                                                         << "\nEnsure module was compiled with Torch-TensorRT.ts or follows Torch-TensorRT Runtime conventions");
+          "Unable to determine binding index for input "
+              << bind_name
+              << "\nEnsure module was compiled with Torch-TensorRT.ts or follows Torch-TensorRT Runtime conventions");
    }

    std::string idx_s = bind_name.substr(delim + 1);
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

fix(//core/runtime): Support more delimiter variants

819c911

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

narendasan requested a review from bowang007 April 27, 2022 03:31

github-actions bot added component: core Issues re: The core compiler component: runtime labels Apr 27, 2022

github-actions bot requested changes Apr 27, 2022

View reviewed changes

github-actions bot approved these changes Apr 27, 2022

View reviewed changes

github-actions bot requested changes Apr 27, 2022

View reviewed changes

github-actions bot added the component: api [C++] Issues re: C++ API label Apr 27, 2022

docs: update docs with binding name restrictions

1c3c779

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

narendasan force-pushed the support_multiple_delimiters branch from 78f808b to 1c3c779 Compare April 27, 2022 03:39

github-actions bot added the component: api [Python] Issues re: Python API label Apr 27, 2022

github-actions bot requested changes Apr 27, 2022

View reviewed changes

github-actions bot approved these changes Apr 27, 2022

View reviewed changes

github-actions bot requested changes Apr 27, 2022

View reviewed changes

github-actions bot approved these changes Apr 27, 2022

View reviewed changes

refactor: apply linting

49d367d

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions bot added the component: tests Issues re: Tests label May 5, 2022

github-actions bot approved these changes May 5, 2022

View reviewed changes

github-actions bot approved these changes May 7, 2022

View reviewed changes

github-actions bot requested changes May 7, 2022

View reviewed changes

narendasan force-pushed the support_multiple_delimiters branch from 30cb548 to 65af9d1 Compare May 7, 2022 01:43

github-actions bot approved these changes May 7, 2022

View reviewed changes

narendasan merged commit 8af0422 into master May 7, 2022

bowang007 mentioned this pull request Jul 6, 2022

fix: converter renaming already named tensors #1167

Merged

7 tasks

bowang007 mentioned this pull request Jul 18, 2022

fix: fix the renaming error in squeeze converter #1187

Closed

7 tasks

fix(//core/runtime): Support more delimiter variants #1004

fix(//core/runtime): Support more delimiter variants #1004

Uh oh!

Conversation

narendasan commented Apr 27, 2022

Description

Type of change

Checklist:

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

bowang007 commented Apr 27, 2022

Uh oh!

narendasan commented Apr 27, 2022

Uh oh!

bowang007 commented Apr 27, 2022

Uh oh!

narendasan commented Apr 28, 2022

Uh oh!

bowang007 commented Apr 28, 2022

Uh oh!

narendasan commented Apr 29, 2022

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

narendasan commented May 6, 2022

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!