[DataPipe] DataPipe Deprecation Tracker #163

NivekT · 2022-01-11T18:18:36Z

We have a number of DataPipes that are being deprecated. Our general policy is that we first mark the DataPipe as deprecated with a warning, and wait at least one release cycle (~3 months) before removing it. Note that some DataPipes will be removed from the PyTorch Core library but will remain in TorchData, and some others are renamed.

Status Types:

Deprecated - marked as deprecated with a warning
Removed - removed from repository

`DataLoader2` Tracker

Name	Deprecation Date	Status	Earliest Removal Version
`PrototypeMultiProcessingReadingService` -> `MultiProcessingReadingService`	0.6	Deprecated	0.8

`IterDataPipe` Tracker

Name	Functional API	Module	Deprecation Date	Status	Earliest Removal Version
BucketBatcher	NA	Core	Sep 30th, 2021	Removed (moved to TorchData)
HTTPReader	NA	Core	Sep 30th, 2021	Removed (moved to TorchData)
LineReader	NA	Core	Sep 30th, 2021	Removed (moved to TorchData)
TarArchiveReader	NA	Core	Sep 30th, 2021	Removed (moved to TorchData)
ZipArchiveReader	NA	Core	Sep 30th, 2021	Removed (moved to TorchData)
FileLoader	NA	Core	Jan 5th, 2022	Removed (use FileOpener)	1.13 (Sept 2022)
FileLoader	NA	Data	Jan 5th, 2022	Removed (use FileOpener)
IoPathFileLoader	load_file_by_iopath	Data	Jan 5th, 2022	Removed (use IoPathFileOpener)
RoutedDecoder	routed_decode	Core	Jan 10th, 2022	Deprecated	1.13 (Sept 2022)
TarArchiveReader	read_from_tar	Data	Feb 22th, 2022	Removed (use TarArchiveLoader)	0.5 (Sept 2022)
XzFileReader	read_from_xz	Data	Feb 22th, 2022	Removed (use XzFileLoader)	0.5 (Sept 2022)
ZipArchiveReader	read_from_zip	Data	Feb 22th, 2022	Removed (use ZipArchiveLoader)	0.5 (Sept 2022)
Filter	filter	Core	1.12	Removed argument (drop_empty_batches)	2.0 (Nov 2022)
FSSpecFileOpener	open_files_by_fsspec	Data	0.4	`open_file_by_fsspec` is Removed	0.6 (Nov 2022)
IoPathFileOpener	open_files_by_fsspec	Data	0.4	`open_file_by_iopath` is Removed	0.6 (Nov 2022)

`MapDataPipe` Tracker

Nothing for now

cc: @ejguan @VitalyFedyunin @NivekT

The text was updated successfully, but these errors were encountered:

@VitalyFedyunin

We labeled these DataPipes as deprecated on Sep 30th, 2021 (#65827). Users should import these DataPipes from [TorchData](https://github.com/pytorch/data) to continue using them. We will be checking for any downstream library usage before landing this PR. All deprecation related to DataPipes are tracked in pytorch/data#163 DataPipes impacted by this PR: | Name | Functional API | Deprecation Date | What Happens Now | ------------- | ------------- | ------------- | ------------- | | BucketBatcher | NA | Sep 30th, 2021 | Remove (moved to TorchData) | HTTPReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | LineReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | TarArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | ZipArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) Lastly, tests related to those DataPipes will also be removed or migrated to TorchData (if they do not already exist there). Differential Revision: [D33532272](https://our.internmc.facebook.com/intern/diff/D33532272) cc ezyang gchanan @VitalyFedyunin ejguan @NivekT [ghstack-poisoned]

@VitalyFedyunin

We labeled these DataPipes as deprecated on Sep 30th, 2021 (#65827). Users should import these DataPipes from [TorchData](https://github.com/pytorch/data) to continue using them. We will be checking for any downstream library usage before landing this PR. All deprecation related to DataPipes are tracked in pytorch/data#163 DataPipes impacted by this PR: | Name | Functional API | Deprecation Date | What Happens Now | ------------- | ------------- | ------------- | ------------- | | BucketBatcher | NA | Sep 30th, 2021 | Remove (moved to TorchData) | HTTPReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | LineReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | TarArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | ZipArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) Lastly, tests related to those DataPipes will also be removed or migrated to TorchData (if they do not already exist there). Differential Revision: [D33532272](https://our.internmc.facebook.com/intern/diff/D33532272) cc ezyang gchanan @VitalyFedyunin ejguan @NivekT [ghstack-poisoned]

@VitalyFedyunin

We labeled these DataPipes as deprecated on Sep 30th, 2021 (#65827). Users should import these DataPipes from [TorchData](https://github.com/pytorch/data) to continue using them. We will be checking for any downstream library usage before landing this PR. All deprecation related to DataPipes are tracked in pytorch/data#163 DataPipes impacted by this PR: | Name | Functional API | Deprecation Date | What Happens Now | ------------- | ------------- | ------------- | ------------- | | BucketBatcher | NA | Sep 30th, 2021 | Remove (moved to TorchData) | HTTPReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | LineReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | TarArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | ZipArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) Lastly, tests related to those DataPipes will also be removed or migrated to TorchData (if they do not already exist there). Differential Revision: [D33532272](https://our.internmc.facebook.com/intern/diff/D33532272) cc ezyang gchanan @VitalyFedyunin ejguan @NivekT [ghstack-poisoned]

@VitalyFedyunin

We labeled these DataPipes as deprecated on Sep 30th, 2021 (#65827). Users should import these DataPipes from [TorchData](https://github.com/pytorch/data) to continue using them. We will be checking for any downstream library usage before landing this PR. All deprecation related to DataPipes are tracked in pytorch/data#163 DataPipes impacted by this PR: | Name | Functional API | Deprecation Date | What Happens Now | ------------- | ------------- | ------------- | ------------- | | BucketBatcher | NA | Sep 30th, 2021 | Remove (moved to TorchData) | HTTPReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | LineReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | TarArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | ZipArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) Lastly, tests related to those DataPipes will also be removed or migrated to TorchData (if they do not already exist there). Differential Revision: [D33532272](https://our.internmc.facebook.com/intern/diff/D33532272) cc ezyang gchanan @VitalyFedyunin ejguan @NivekT [ghstack-poisoned]

@VitalyFedyunin

We labeled these DataPipes as deprecated on Sep 30th, 2021 (#65827). Users should import these DataPipes from [TorchData](https://github.com/pytorch/data) to continue using them. We will be checking for any downstream library usage before landing this PR. All deprecation related to DataPipes are tracked in pytorch/data#163 DataPipes impacted by this PR: | Name | Functional API | Deprecation Date | What Happens Now | ------------- | ------------- | ------------- | ------------- | | BucketBatcher | NA | Sep 30th, 2021 | Remove (moved to TorchData) | HTTPReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | LineReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | TarArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) | ZipArchiveReader | NA | Sep 30th, 2021 | Remove (moved to TorchData) Lastly, tests related to those DataPipes will also be removed or migrated to TorchData (if they do not already exist there). Differential Revision: [D33532272](https://our.internmc.facebook.com/intern/diff/D33532272) cc ezyang gchanan @VitalyFedyunin ejguan @NivekT [ghstack-poisoned]

ejguan · 2022-03-01T19:52:53Z

For TarArchiveReader, should we add a deprecation warning in main branch as 0.3.0 branch cut has been finished.

Summary: Pull Request resolved: #272 Tracked by #163 Test Plan: Imported from OSS Reviewed By: ejguan Differential Revision: D34562086 Pulled By: NivekT fbshipit-source-id: c930c190da57a82c491da265da5848f94aadd85e

…taPipe arguments Last patch to align DataPipe API with TorchArrow DataFrame For deprecation warning of DataPipe argument: ``` The argument `drop_empty_batches` of `FilterIterDataPipe()` is deprecated since 1.12 and will be removed in 1.14. See pytorch/data#163 for details. ``` Pull Request resolved: #76060 Approved by: https://github.com/NivekT

…taPipe arguments (#76060) Summary: Last patch to align DataPipe API with TorchArrow DataFrame For deprecation warning of DataPipe argument: ``` The argument `drop_empty_batches` of `FilterIterDataPipe()` is deprecated since 1.12 and will be removed in 1.14. See pytorch/data#163 for details. ``` Pull Request resolved: #76060 Approved by: https://github.com/NivekT Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/ec591087fb95525b430f2bbcdd023567e2fb1cc9 Reviewed By: seemethere, osalpekar Differential Revision: D35874488 Pulled By: ejguan fbshipit-source-id: 83f1cfc6430e6294b332195e9f056af4d2405dce

…Pipe names" Fixes pytorch/data#480. For the list of deprecated DataPipes (or functional names), see pytorch/data#163 Testing: ```python IterableWrapper(range(10)).open_file_by_iopath() ``` Returns: ``` /Users/.../pytorch/torch/utils/data/datapipes/utils/common.py:171: FutureWarning: `IoPathFileOpener()`'s functional API `.open_file_by_iopath()` is deprecated since 1.12 and will be removed in 1.14. See pytorch/data#163 for details. Please use `.open_files_by_iopath()` instead. warnings.warn(msg, FutureWarning) ``` [ghstack-poisoned]

ejguan · 2022-09-19T20:50:00Z

Another Misc tracker:

Name	Module	Deprecation Version	Status	Earliest Removal Version
torch.utils.data.graph.traverse	Core	1.13	Deprecating	1.15 / 2.1

Summary: - Continue to deprecate stuff based on #163 - Fix a typo Pull Request resolved: #890 Reviewed By: NivekT Differential Revision: D41130080 Pulled By: ejguan fbshipit-source-id: d8c1a6a4732211fcf75a08d5cb5e090a587b555b

…ional APIs (#88693) - Deprecating based on pytorch/data#163 Corresponding PRs from TorchData: pytorch/data#890 Pull Request resolved: #88693 Approved by: https://github.com/NivekT

BlueskyFR · 2022-11-23T16:39:29Z

I see RoutedDecoder has been marked as deprecated: what is it going to be replaced by?

ejguan · 2022-11-23T16:56:33Z

I see RoutedDecoder has been marked as deprecated: what is it going to be replaced by?

@BlueskyFR
IIRC, we plan to remove this DataPipe in the future. The general reason is that we think this can be easily achieved by using a demux based on file types then decode each datapipe correspondingly then mux them together. Glad to hear your use case.

BlueskyFR · 2022-11-23T18:41:08Z

I see RoutedDecoder has been marked as deprecated: what is it going to be replaced by?

@BlueskyFR
IIRC, we plan to remove this DataPipe in the future. The general reason is that we think this can be easily achieved by using a demux based on file types then decode each datapipe correspondingly then mux them together. Glad to hear your use case.

I don't understand: how should I proceed to decode a PNG image in the current state then?

ejguan · 2022-11-23T19:06:15Z

You can use a map function like datapipe.map(decode_fn) to decode the PNG image

BlueskyFR · 2022-11-23T19:34:08Z

You can use a map function like datapipe.map(decode_fn) to decode the PNG image

Okay, but why was support for decoding dropped then?

ejguan · 2022-11-23T19:40:24Z

Okay, but why was support for decoding dropped then?

decoding didn't do more things like a map function, except we provided a few decoding functions for convenient. And, in order to support routed_decode, we need to add lots of decoding functions to cover the general file decoding, which is not sustainable for us to maintain and it makes the routed_decode more complicated and redundant. For example of your use case (decoding PNG), the routed_decode would add more decoding handlers such as json, pickle, etc. into this DataPipe.

As, TorchData provides composable way to construct pipeline, users should be able to create a pipeline to handle specific decoding mechanism

BlueskyFR · 2022-11-23T19:46:47Z

Okay, but why was support for decoding dropped then?

decoding didn't do more things like a map function, except we provided a few decoding functions for convenient. And, in order to support routed_decode, we need to add lots of decoding functions to cover the general file decoding, which is not sustainable for us to maintain and it makes the routed_decode more complicated and redundant. For example of your use case (decoding PNG), the routed_decode would add more decoding handlers such as json, pickle, etc. into this DataPipe.

As, TorchData provides composable way to construct pipeline, users should be able to create a pipeline to handle specific decoding mechanism

Okay. What is the preferred mechanism to decode images? Ideally I think it should be done in batches if performance is needed

ejguan · 2022-11-23T19:55:28Z

Okay. What is the preferred mechanism to decode images? Ideally I think it should be done in batches if performance is needed

It depends on if your decode_fn supports batched decoding in high performance (multithreading). Otherwise, I think it's going to be similar to do decoding per image.

…ional APIs (pytorch#88693) - Deprecating based on pytorch/data#163 Corresponding PRs from TorchData: pytorch/data#890 Pull Request resolved: pytorch#88693 Approved by: https://github.com/NivekT

NivekT mentioned this issue Jan 11, 2022

[DataPipe] Removing deprecated DataPipes pytorch/pytorch#71161

Closed

NivekT mentioned this issue Mar 1, 2022

Adding deprecation warnings for archive readers #272

Closed

This was referenced Mar 24, 2022

Warnings for deprecated datapipes are insufficient #322

Closed

improve datapipe deprecation warnings pytorch/pytorch#74685

Closed

Updating TorchData DataPipe API usages pytorch/text#1663

Merged

ejguan mentioned this issue Apr 19, 2022

[DataPipe] Add input_col to filter and add deprecation warning for DataPipe arguments pytorch/pytorch#76060

Closed

NivekT mentioned this issue Jun 6, 2022

[DataPipe] Add function for deprecation of functional DataPipe names pytorch/pytorch#78970

Closed

This was referenced Nov 8, 2022

Deprecating certain functional APIs and fix an Error msg #890

Closed

[DataPipe] Deprecating drop_empty_batches from Filter and other functional APIs pytorch/pytorch#88693

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DataPipe] DataPipe Deprecation Tracker #163

[DataPipe] DataPipe Deprecation Tracker #163

NivekT commented Jan 11, 2022 •

edited

Loading

ejguan commented Mar 1, 2022

Uh oh!

ejguan commented Sep 19, 2022 •

edited

Loading

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022 •

edited

Loading

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022 •

edited

Loading

Uh oh!

[DataPipe] DataPipe Deprecation Tracker #163

[DataPipe] DataPipe Deprecation Tracker #163

Comments

NivekT commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DataLoader2 Tracker

IterDataPipe Tracker

MapDataPipe Tracker

ejguan commented Mar 1, 2022

Uh oh!

ejguan commented Sep 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022

Uh oh!

BlueskyFR commented Nov 23, 2022

Uh oh!

ejguan commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NivekT commented Jan 11, 2022 •

edited

Loading

`DataLoader2` Tracker

`IterDataPipe` Tracker

`MapDataPipe` Tracker

ejguan commented Sep 19, 2022 •

edited

Loading

ejguan commented Nov 23, 2022 •

edited

Loading

ejguan commented Nov 23, 2022 •

edited

Loading