USPS dataset #5647

lezwon · 2022-03-19T14:06:27Z

Add USPS dataset to torchvision. Fixes #5352

facebook-github-bot · 2022-03-19T14:06:34Z

💊 CI failures summary and remediations

As of commit d23863c (more details on the Dr. CI page):

1/4 failures introduced in this PR
3/4 broken upstream at merge base 1af20e8 since Mar 21

1 failure not recognized by patterns:

Job	Step	Action
^{cmake_linux_gpu}	^{Build torchvision C++ distribution and test}	🔁 rerun

🚧 3 ongoing upstream failures:

These were probably caused by upstream breakages that are not fixed yet.

binary_linux_conda_py3.7_cu115 since Mar 21 (fbc8ea4)
- 🔁 rerun
binary_linux_conda_py3.10_cu115 since Mar 21 (fbc8ea4)
- 🔁 rerun
binary_linux_conda_py3.8_cu115 since Mar 21 (fbc8ea4)
- 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

pmeier

Thanks a lot for the PR @lezwon! I have some comments inline. When they are addressed, we also need to add tests. I'll add another section to the README to guide you.

torchvision/prototype/datasets/_builtin/usps.py

* xfail mobilnet norm layer test * fix test

To mitigate missing `libcupti.so` dependency

* update urls for kinetics dataset * update urls for kinetics dataset * remove errors * update the changes and add test option to split * added test to valid values for split arg * change .txt to .csv for annotation url of k600 Co-authored-by: Nicolas Hug <[email protected]>

* Moving basefiles outside of prototype and porting Alexnet, ConvNext, Densenet and EfficientNet. * Porting googlenet * Porting inception * Porting mnasnet * Porting mobilenetv2 * Porting mobilenetv3 * Porting regnet * Porting resnet * Porting shufflenetv2 * Porting squeezenet * Porting vgg * Porting vit * Fix docstrings * Fixing imports * Adding missing import * Fix mobilenet imports * Fix tests * Fix prototype tests * Exclude get_weight from models on test * Fix init files * Porting googlenet * Porting inception * porting mobilenetv2 * porting mobilenetv3 * porting resnet * porting shufflenetv2 * Fix test and linter * Fixing docs. * Porting Detection models (pytorch#5617) * fix inits * fix docs * Port faster_rcnn * Port fcos * Port keypoint_rcnn * Port mask_rcnn * Port retinanet * Port ssd * Port ssdlite * Fix linter * Fixing tests * Fixing tests * Fixing vgg test * Porting Optical Flow, Segmentation, Video models (pytorch#5619) * Porting raft * Porting video resnet * Porting deeplabv3 * Porting fcn and lraspp * Fixing the tests and linter * Porting docs, examples, tutorials and galleries (pytorch#5620) * Fix examples, tutorials and gallery * Update gallery/plot_optical_flow.py Co-authored-by: Nicolas Hug <[email protected]> * Fix import * Revert hardcoded normalization * fix uncommitted changes * Fix bug * Fix more bugs * Making resize optional for segmentation * Fixing preset * Fix mypy * Fixing documentation strings * Fix flake8 * minor refactoring Co-authored-by: Nicolas Hug <[email protected]> * Resolve conflict * Porting model tests (pytorch#5622) * Porting tests * Remove unnecessary variable * Fix linter * Move prototype to extended tests * Fix download models job * Update CI on Multiweight branch to use the new weight download approach (pytorch#5628) * port Pad to prototype transforms (pytorch#5621) * port Pad to prototype transforms * use literal * Bump up LibTorchvision version number for Podspec to release Cocoapods (pytorch#5624) Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]> * pre-download model weights in CI docs build (pytorch#5625) * pre-download model weights in CI docs build * move changes into template * change docs image * Regenerated config.yml Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Anton Thomma <[email protected]> * Porting reference scripts and updating presets (pytorch#5629) * Making _preset.py classes * Remove support of targets on presets. * Rewriting the video preset * Adding tests to check that the bundled transforms are JIT scriptable * Rename all presets from *Eval to *Inference * Minor refactoring * Remove --prototype and --pretrained from reference scripts * remove pretained_backbone refs * Corrections and simplifications * Fixing bug * Fixing linter * Fix flake8 * restore documentation example * minor fixes * fix optical flow missing param * Fixing commands * Adding weights_backbone support in detection and segmentation * Updating the commands for InceptionV3 * Setting `weights_backbone` to its fully BC value (pytorch#5653) * Replace default `weights_backbone=None` with its BC values. * Fixing tests * Fix linter * Update docs. * Update preprocessing on reference scripts. * Change qat/ptq to their full values. * Refactoring preprocessing * Fix video preset * No initialization on VGG if pretrained * Fix warning messages for backbone utils. * Adding star to all preset constructors. * Fix mypy. Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Anton Thomma <[email protected]>

Co-authored-by: Philip Meier <[email protected]>

pmeier

Thanks @lezwon this is looking pretty good. I've added some suggestions inline. Apart from that, we just merged #5666 that added instructions how to add tests. Please follow them and let me know if something is unclear.

torchvision/prototype/datasets/_builtin/usps.py

Co-authored-by: Philip Meier <[email protected]>

lezwon · 2022-03-25T01:02:40Z

@pmeier I've added the tests. Do let me know if any changes are required. :)

pmeier

Almost there!

test/test_prototype_builtin_datasets.py

test/builtin_dataset_mocks.py

Co-authored-by: Philip Meier <[email protected]>

pmeier

LGTM, thanks a lot @lezwon for the continued work on this!

test/test_prototype_builtin_datasets.py

NicolasHug

Thanks a lot @lezwon and @pmeier , LGTM . (Ideally we'd change the test name to avoid confusion before merging https://github.com/pytorch/vision/pull/5647/files#r835158538)

Co-authored-by: Nicolas Hug <[email protected]>

github-actions · 2022-03-25T12:54:17Z

Hey @pmeier!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

Summary: * added usps dataset * fixed type issues * fix mobilnet norm layer test (#5643) * xfail mobilnet norm layer test * fix test * More robust check in tests for 16 bits images (#5652) * Prefer nvidia channel for conda builds (#5648) To mitigate missing `libcupti.so` dependency * fix torchdata CI installation (#5657) * update urls for kinetics dataset (#5578) * update urls for kinetics dataset * update urls for kinetics dataset * remove errors * update the changes and add test option to split * added test to valid values for split arg * change .txt to .csv for annotation url of k600 * Port Multi-weight support from prototype to main (#5618) * Moving basefiles outside of prototype and porting Alexnet, ConvNext, Densenet and EfficientNet. * Porting googlenet * Porting inception * Porting mnasnet * Porting mobilenetv2 * Porting mobilenetv3 * Porting regnet * Porting resnet * Porting shufflenetv2 * Porting squeezenet * Porting vgg * Porting vit * Fix docstrings * Fixing imports * Adding missing import * Fix mobilenet imports * Fix tests * Fix prototype tests * Exclude get_weight from models on test * Fix init files * Porting googlenet * Porting inception * porting mobilenetv2 * porting mobilenetv3 * porting resnet * porting shufflenetv2 * Fix test and linter * Fixing docs. * Porting Detection models (#5617) * fix inits * fix docs * Port faster_rcnn * Port fcos * Port keypoint_rcnn * Port mask_rcnn * Port retinanet * Port ssd * Port ssdlite * Fix linter * Fixing tests * Fixing tests * Fixing vgg test * Porting Optical Flow, Segmentation, Video models (#5619) * Porting raft * Porting video resnet * Porting deeplabv3 * Porting fcn and lraspp * Fixing the tests and linter * Porting docs, examples, tutorials and galleries (#5620) * Fix examples, tutorials and gallery * Update gallery/plot_optical_flow.py * Fix import * Revert hardcoded normalization * fix uncommitted changes * Fix bug * Fix more bugs * Making resize optional for segmentation * Fixing preset * Fix mypy * Fixing documentation strings * Fix flake8 * minor refactoring * Resolve conflict * Porting model tests (#5622) * Porting tests * Remove unnecessary variable * Fix linter * Move prototype to extended tests * Fix download models job * Update CI on Multiweight branch to use the new weight download approach (#5628) * port Pad to prototype transforms (#5621) * port Pad to prototype transforms * use literal * Bump up LibTorchvision version number for Podspec to release Cocoapods (#5624) * pre-download model weights in CI docs build (#5625) * pre-download model weights in CI docs build * move changes into template * change docs image * Regenerated config.yml * Porting reference scripts and updating presets (#5629) * Making _preset.py classes * Remove support of targets on presets. * Rewriting the video preset * Adding tests to check that the bundled transforms are JIT scriptable * Rename all presets from *Eval to *Inference * Minor refactoring * Remove --prototype and --pretrained from reference scripts * remove pretained_backbone refs * Corrections and simplifications * Fixing bug * Fixing linter * Fix flake8 * restore documentation example * minor fixes * fix optical flow missing param * Fixing commands * Adding weights_backbone support in detection and segmentation * Updating the commands for InceptionV3 * Setting `weights_backbone` to its fully BC value (#5653) * Replace default `weights_backbone=None` with its BC values. * Fixing tests * Fix linter * Update docs. * Update preprocessing on reference scripts. * Change qat/ptq to their full values. * Refactoring preprocessing * Fix video preset * No initialization on VGG if pretrained * Fix warning messages for backbone utils. * Adding star to all preset constructors. * Fix mypy. * Apply suggestions from code review * use decompressor for extracting bz2 * Apply suggestions from code review * Apply suggestions from code review * fixed lint fails * added tests for USPS * check image shape * fix tests * check shape on image directly * Apply suggestions from code review * removed test and comments * Update test/test_prototype_builtin_datasets.py (Note: this ignores all push blocking failures!) Reviewed By: datumbox Differential Revision: D35216783 fbshipit-source-id: 556a63a89f15d1541ac2b479244a7b6c564eff14 Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Nicolas Hug <[email protected]> Co-authored-by: Nikita Shulga <[email protected]> Co-authored-by: Sahil Goyal <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]> Co-authored-by: Anton Thomma <[email protected]> Co-authored-by: Anton Thomma <[email protected]>

added usps dataset

0606674

pytorch-bot bot added the ciflow/default label Mar 19, 2022

facebook-github-bot added the cla signed label Mar 19, 2022

fixed type issues

ed267f6

lezwon changed the title ~~[WIP] USPS dataset~~ USPS dataset Mar 21, 2022

pmeier requested changes Mar 23, 2022

View reviewed changes

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

pmeier and others added 8 commits March 23, 2022 16:40

fix mobilnet norm layer test (pytorch#5643)

16123ef

* xfail mobilnet norm layer test * fix test

More robust check in tests for 16 bits images (pytorch#5652)

a37b3e9

Prefer nvidia channel for conda builds (pytorch#5648)

7fd2ea0

To mitigate missing `libcupti.so` dependency

fix torchdata CI installation (pytorch#5657)

2efd0f2

Apply suggestions from code review

f7d0a50

Co-authored-by: Philip Meier <[email protected]>

use decompressor for extracting bz2

85bc5fd

lezwon force-pushed the 5352-usps-dataset branch from 8df24ed to 85bc5fd Compare March 23, 2022 11:11

Merge branch 'main' into 5352-usps-dataset

0b46877

pmeier reviewed Mar 23, 2022

View reviewed changes

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

torchvision/prototype/datasets/_builtin/usps.py Outdated Show resolved Hide resolved

lezwon and others added 10 commits March 23, 2022 22:16

Apply suggestions from code review

68f15ba

Co-authored-by: Philip Meier <[email protected]>

Apply suggestions from code review

aefad05

Co-authored-by: Philip Meier <[email protected]>

Merge branch 'main' into 5352-usps-dataset

e1ec56c

fixed lint fails

81cfba9

added tests for USPS

e847fde

Merge branch 'main' into 5352-usps-dataset

96da894

check image shape

fe7c573

fix tests

b1deb63

Merge branch 'main' into 5352-usps-dataset

9a08f51

check shape on image directly

0624de9

pmeier reviewed Mar 25, 2022

View reviewed changes

test/test_prototype_builtin_datasets.py Outdated Show resolved Hide resolved

test/test_prototype_builtin_datasets.py Outdated Show resolved Hide resolved

test/builtin_dataset_mocks.py Outdated Show resolved Hide resolved

lezwon and others added 2 commits March 25, 2022 15:06

Apply suggestions from code review

d5d9386

Co-authored-by: Philip Meier <[email protected]>

removed test and comments

7324861

pmeier approved these changes Mar 25, 2022

View reviewed changes

pmeier requested a review from NicolasHug March 25, 2022 09:58

NicolasHug reviewed Mar 25, 2022

View reviewed changes

test/test_prototype_builtin_datasets.py Outdated Show resolved Hide resolved

NicolasHug approved these changes Mar 25, 2022

View reviewed changes

Update test/test_prototype_builtin_datasets.py

d23863c

Co-authored-by: Nicolas Hug <[email protected]>

pmeier merged commit 71907be into pytorch:main Mar 25, 2022

pmeier added module: datasets prototype labels Mar 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

USPS dataset #5647

USPS dataset #5647

lezwon commented Mar 19, 2022

facebook-github-bot commented Mar 19, 2022 •

edited

Loading

pmeier left a comment

pmeier left a comment

lezwon commented Mar 25, 2022

pmeier left a comment

pmeier left a comment

NicolasHug left a comment

github-actions bot commented Mar 25, 2022

USPS dataset #5647

USPS dataset #5647

Conversation

lezwon commented Mar 19, 2022

facebook-github-bot commented Mar 19, 2022 • edited Loading

💊 CI failures summary and remediations

1 failure not recognized by patterns:

🚧 3 ongoing upstream failures:

pmeier left a comment

Choose a reason for hiding this comment

pmeier left a comment

Choose a reason for hiding this comment

lezwon commented Mar 25, 2022

pmeier left a comment

Choose a reason for hiding this comment

pmeier left a comment

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 25, 2022

facebook-github-bot commented Mar 19, 2022 •

edited

Loading