HADOOP-18189 S3APrefetchingInputStream to support status probes when closed #5036

virajjasani · 2022-10-16T04:29:35Z

No description provided.

…closed

virajjasani · 2022-10-16T04:30:26Z

Tested against us-west-2 with:

mvn -Dparallel-tests -DtestsThreadCount=8 -Dscale clean verify

mvn -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale clean verify

hadoop-yetus · 2022-10-16T06:13:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 54s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 16s		trunk passed
+1 💚	shadedclient	23m 53s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 14s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 10s		the patch passed
+1 💚	shadedclient	23m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 26s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		102m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/1/artifact/out/Dockerfile
GITHUB PR	#5036
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6a911c8ecd7b 4.15.0-192-generic #203-Ubuntu SMP Wed Aug 10 17:40:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e49f8a6`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/1/testReport/
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-10-17T08:02:46Z

@steveloughran, requesting your review

steveloughran

production code all good, minor tweaks to the test cases needed

steveloughran · 2022-10-17T17:20:33Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java

+        + " should still support some status probes");
+
+    byte[] data = ContractTestUtils.dataset(SMALL_FILE_SIZE, 'a', 26);
+    Path smallFile = path("testStatusProbesAfterClosingStream");


you can just use path() and one is built up from the method name. this is better as it avoids cut and paste bugs and if a build is parameterized, the parameterized string is used in the path (though tests fail if that string isn't a valid path any more)

Ah my bad, should have used methodPath(). You meant methodPath() only, correct?

steveloughran · 2022-10-17T17:22:00Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java

+    assertEquals("Stream stats retrieved through stream before and after closing should match",
+        inputStreamStatistics, newInputStreamStatistics);
+
+    assertFalse("Not supported with prefetch", in.seekToNewSource(10));


nit: can you have the error message include "seekToNewSource()".

hadoop-yetus · 2022-10-17T23:34:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 1s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 16s		trunk passed
+1 💚	shadedclient	23m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 14s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	23m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 27s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		101m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/2/artifact/out/Dockerfile
GITHUB PR	#5036
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2918fd61f970 4.15.0-192-generic #203-Ubuntu SMP Wed Aug 10 17:40:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f367cf6`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/2/testReport/
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5036/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-10-18T18:40:05Z

Addressed both of above comments

steveloughran

+1, merging. thanks

…closed (apache#5036) Contributed by Viraj Jasani

…closed (#5036) Contributed by Viraj Jasani

HADOOP-18189 S3APrefetchingInputStream to support status probes when …

e49f8a6

…closed

steveloughran requested changes Oct 17, 2022

View reviewed changes

addendum

f367cf6

virajjasani requested a review from steveloughran October 18, 2022 18:39

steveloughran approved these changes Oct 19, 2022

View reviewed changes

steveloughran merged commit 8aa04b0 into apache:trunk Oct 19, 2022

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18189 S3APrefetchingInputStream to support status probes when …

2c844aa

…closed (apache#5036) Contributed by Viraj Jasani

ahmarsuhail mentioned this pull request Apr 17, 2023

HADOOP-18028. High performance S3A input stream (#4752) #5559

Closed

4 tasks

asfgit pushed a commit that referenced this pull request Apr 26, 2023

HADOOP-18189 S3APrefetchingInputStream to support status probes when …

b031fc8

…closed (#5036) Contributed by Viraj Jasani

asfgit pushed a commit that referenced this pull request Apr 28, 2023

HADOOP-18189 S3APrefetchingInputStream to support status probes when …

a71c708

…closed (#5036) Contributed by Viraj Jasani

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18189 S3APrefetchingInputStream to support status probes when closed #5036

HADOOP-18189 S3APrefetchingInputStream to support status probes when closed #5036

Uh oh!

virajjasani commented Oct 16, 2022

Uh oh!

virajjasani commented Oct 16, 2022

Uh oh!

hadoop-yetus commented Oct 16, 2022

Uh oh!

virajjasani commented Oct 17, 2022

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Oct 17, 2022

Uh oh!

virajjasani Oct 17, 2022

Uh oh!

steveloughran Oct 17, 2022

Uh oh!

hadoop-yetus commented Oct 17, 2022

Uh oh!

virajjasani commented Oct 18, 2022

Uh oh!

steveloughran left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18189 S3APrefetchingInputStream to support status probes when closed #5036

HADOOP-18189 S3APrefetchingInputStream to support status probes when closed #5036

Uh oh!

Conversation

virajjasani commented Oct 16, 2022

Uh oh!

virajjasani commented Oct 16, 2022

Uh oh!

hadoop-yetus commented Oct 16, 2022

Uh oh!

virajjasani commented Oct 17, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

virajjasani Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Oct 17, 2022

Uh oh!

virajjasani commented Oct 18, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants