[Caffe2] Investigate the performance gap between Caffe2 and Tensorflow port

[petrex]

The goal is to eliminate the overhead in the current implementation. Expect to see on par performance between different frameworks.

[petrex]

@ashishfarmer Please make sure we are running exactly the same thing in caffe2 and Tensorflow before the investigation. Try ONNX and share your knowledge with Perf Lab.

[petrex]

seeing on par performance locally. need to enable perf lab

[ashishfarmer]

Closing this issue.
The perf enhancement will be an ongoing work