-
Notifications
You must be signed in to change notification settings - Fork 7.1k
Add a test that compares the output of our quantized models against expected cached values #4597
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a test that compares the output of our quantized models against expected cached values #4597
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @jdsgomes
Overall it looks great, I just left a couple of minor remarks for you. Let me know your thoughts.
Edit: Also there seems to be a conflict with latest main, could you please resolve?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are failing tests that seem related. It might be due to flakiness but worth checking the following:
Co-authored-by: Vasilis Vryniotis <[email protected]>
…ision into output-checks-on-quant-models
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @jdsgomes!
Thanks for the review @datumbox ! |
…against expected cached values (#4597) Summary: * adding tests to check output of quantized models * adding test quantized model weights * merge test_new_quantized_classification_model with test_quantized_classification_model * adding skipif removed by mistake * addressing comments from PR * removing unused argument * fixing lint errors * changing model to eval model and updating weights * Update test/test_models.py * enforce single test in circleci * changing random seed * updating weights for new seed * adding missing empty line * try 128 random seed * try 256 random seed * try 16 random seed * disable inception_v3 input/output quantization tests * removing ModelTester.test_inception_v3_quantized_expect.pkl * reverting temporary ci run_test.sh changes Reviewed By: fmassa Differential Revision: D31649962 fbshipit-source-id: 35a0cb4d8d3564c88dabc09e750d5ad0a281431a Co-authored-by: Vasilis Vryniotis <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]>
…xpected cached values (pytorch#4597) * adding tests to check output of quantized models * adding test quantized model weights * merge test_new_quantized_classification_model with test_quantized_classification_model * adding skipif removed by mistake * addressing comments from PR * removing unused argument * fixing lint errors * changing model to eval model and updating weights * Update test/test_models.py Co-authored-by: Vasilis Vryniotis <[email protected]> * enforce single test in circleci * changing random seed * updating weights for new seed * adding missing empty line * try 128 random seed * try 256 random seed * try 16 random seed * disable inception_v3 input/output quantization tests * removing ModelTester.test_inception_v3_quantized_expect.pkl * reverting temporary ci run_test.sh changes Co-authored-by: Vasilis Vryniotis <[email protected]>
…xpected cached values (pytorch#4597) * adding tests to check output of quantized models * adding test quantized model weights * merge test_new_quantized_classification_model with test_quantized_classification_model * adding skipif removed by mistake * addressing comments from PR * removing unused argument * fixing lint errors * changing model to eval model and updating weights * Update test/test_models.py Co-authored-by: Vasilis Vryniotis <[email protected]> * enforce single test in circleci * changing random seed * updating weights for new seed * adding missing empty line * try 128 random seed * try 256 random seed * try 16 random seed * disable inception_v3 input/output quantization tests * removing ModelTester.test_inception_v3_quantized_expect.pkl * reverting temporary ci run_test.sh changes Co-authored-by: Vasilis Vryniotis <[email protected]>
closes #4502
cc @datumbox @pmeier