Adds TextStreamingDecoder

[keunwoochoi]

This is a specialized file opener + decoder that

• works for various types of sources (s3, gcs, local path)
• open any text file and stream the content line by line.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1500

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit 8832f44 with merge base 9295079 ():

NEW FAILURES - The following jobs have failed:

• Run Nodes Tests / test (windows-latest, 3.10) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.11) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.12) (gh)
  Process completed with exit code 1.
• Run Nodes Tests / test (windows-latest, 3.9) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ramanishsingh]

We can add a shutdown method

[keunwoochoi]

ok, added in the new commit.

[ramanishsingh]

shall we also save METADATA_KEY?

[keunwoochoi]

i) like self.CURRENT_FILE_KEY, i added self.CURRENT_LINE_KEY.

ii) are you sure you meant saving METADATA_KEY? DATA_KEY and METADATA_KEY play similar roles which i don't think needs to be saved.

[keunwoochoi]

thanks for the review. made some changes & pushed three commits.

[keunwoochoi]

(note to myself)

(1018 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================== short test summary info ===========================
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_metadata - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpcyw34h51\\test1.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_state_management - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpe4dio47s\\test1.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_empty_file - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpivu3sp7w\\normal.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_text_stream_encoding - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp73b21cs9\\utf8.txt'
FAILED test/nodes/io/test_text_streaming_decoder.py::test_error_handling - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp4oq398ov\\valid.txt'
===== 5 failed, 383 passed, 15 skipped, 27 warnings in 1320.71s (0:22:00) =====
Error: Process completed with exit code 1.