MobileNetV3 quantization bug

[datumbox]

🐛 Describe the bug

This issue documents an bug discovered on February 2021 while quantizing the MobileNetV3. We worked around the issue but it probably remains unresolved on the eager quantization API. Here I'm gathering all the information exchanged with the quantization team so that we can investigate on the future.

---

TLDR: During the QAT training of MobileNetV3 the accuracies are at the expected levels. After serializing and loading the weights of the quantized weights, the accuracies appear similar to a randomly initialized model. The workaround is to instead load the QAT fine-tuned weights and convert the model.

Originally to quantize the MobileNetV3 Large we took the standard approach (see #3323). Our initial setup was extremely similar to the one of MobileNetV2. We trained a model with the reference script and we got good accuracies. Unfortunately when we tried to deploy the weights of the quantized model we found out that loading them and doing predictions leads to extremely low accuracy (less than 1%).

After trial and error, we found implemented the following workaround that allows to load the weights, quantize the model and get good accuracies: 274c6a1. Effectively we create a QAT model, load the fine-tuned weights and then quantize it.

The above workaround should be unnecessary. Instead MobileNetV3 should follow the same approach as with every other QAT model (such as in the case of MobileNetV2).