HDFS-17506. [FGL] Performance for phase 1 #6806

ZanderXu · 2024-05-09T08:32:26Z

Add some benchmarks

ferhui · 2024-05-10T01:32:40Z

Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally?

ZanderXu · 2024-05-10T02:23:36Z

Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally?

FSNLockBenchmarkThroughput is a tool, each one can do the performance tests through this tool after deploying FGL.
TestFSNLockBenchmarkThroughput is a UT, it mocks a MiniQJMHACluster and do some performance tests locally.

ferhui · 2024-05-10T03:41:18Z

Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally?

FSNLockBenchmarkThroughput is a tool, each one can do the performance tests through this tool after deploying FGL. TestFSNLockBenchmarkThroughput is a UT, it mocks a MiniQJMHACluster and do some performance tests locally.

Got it, thanks.

ferhui

It's OK to me as an initial version.
Just left minor comments.

...dfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/fgl/FSNLockBenchmarkThroughput.java

ferhui · 2024-05-10T06:08:02Z

Can also add usage referring to hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java

hadoop-yetus · 2024-05-10T13:20:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ HDFS-17384 Compile Tests _
-1 ❌	mvninstall	35m 34s	/branch-mvninstall-root.txt	root in HDFS-17384 failed.
+1 💚	compile	0m 44s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 40s		HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 43s		HDFS-17384 passed
+1 💚	mvnsite	0m 45s		HDFS-17384 passed
+1 💚	javadoc	0m 43s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 9s		HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 56s		HDFS-17384 passed
+1 💚	shadedclient	21m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 30s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 43s		the patch passed
+1 💚	shadedclient	21m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	212m 26s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		306m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/artifact/out/Dockerfile
GITHUB PR	#6806
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3535f61db013 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `3b26cc4`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/testReport/
Max. process+thread count	4548 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-05-10T18:52:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 01s		No case conflicting files found.
+0 🆗	spotbugs	0m 01s		spotbugs executables are not available.
+0 🆗	codespell	0m 01s		codespell was not available.
+0 🆗	detsecrets	0m 01s		detect-secrets was not available.
+1 💚	@author	0m 00s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 00s		The patch appears to include 2 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	128m 34s		HDFS-17384 passed
+1 💚	compile	10m 30s		HDFS-17384 passed
+1 💚	checkstyle	7m 37s		HDFS-17384 passed
+1 💚	mvnsite	10m 54s		HDFS-17384 passed
+1 💚	javadoc	9m 03s		HDFS-17384 passed
+1 💚	shadedclient	218m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	7m 15s		the patch passed
+1 💚	compile	5m 24s		the patch passed
+1 💚	javac	5m 24s		the patch passed
+1 💚	blanks	0m 01s		The patch has no blanks issues.
+1 💚	checkstyle	3m 41s		the patch passed
+1 💚	mvnsite	6m 18s		the patch passed
+1 💚	javadoc	5m 27s		the patch passed
+1 💚	shadedclient	224m 07s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	8m 28s		The patch does not generate ASF License warnings.
		614m 22s

Subsystem	Report/Notes
GITHUB PR	#6806
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 ab4e39a29189 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `3b26cc4`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/1/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/1/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-05-10T21:52:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 00s		No case conflicting files found.
+0 🆗	spotbugs	0m 01s		spotbugs executables are not available.
+0 🆗	codespell	0m 01s		codespell was not available.
+0 🆗	detsecrets	0m 01s		detect-secrets was not available.
+1 💚	@author	0m 00s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 00s		The patch appears to include 2 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	88m 04s		HDFS-17384 passed
+1 💚	compile	6m 18s		HDFS-17384 passed
+1 💚	checkstyle	4m 39s		HDFS-17384 passed
+1 💚	mvnsite	6m 24s		HDFS-17384 passed
+1 💚	javadoc	5m 43s		HDFS-17384 passed
+1 💚	shadedclient	144m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 44s		the patch passed
+1 💚	compile	3m 26s		the patch passed
+1 💚	javac	3m 26s		the patch passed
+1 💚	blanks	0m 01s		The patch has no blanks issues.
+1 💚	checkstyle	2m 17s		the patch passed
+1 💚	mvnsite	3m 59s		the patch passed
+1 💚	javadoc	3m 26s		the patch passed
+1 💚	shadedclient	155m 07s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	5m 14s		The patch does not generate ASF License warnings.
		415m 09s

Subsystem	Report/Notes
GITHUB PR	#6806
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 45a322de9117 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `3b26cc4`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/2/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/2/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ZanderXu · 2024-05-11T07:01:12Z

The failed UT hadoop.hdfs.tools.TestDFSAdmin is not caused by this PR, I will create a ticket to fix it.

hadoop-yetus · 2024-05-11T11:10:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	33m 21s		HDFS-17384 passed
+1 💚	compile	0m 43s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 43s		HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 40s		HDFS-17384 passed
+1 💚	mvnsite	0m 47s		HDFS-17384 passed
+1 💚	javadoc	0m 44s		HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 7s		HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 53s		HDFS-17384 passed
+1 💚	shadedclient	21m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 31s		the patch passed
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 6s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 51s		the patch passed
+1 💚	shadedclient	21m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	213m 38s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		306m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/artifact/out/Dockerfile
GITHUB PR	#6806
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6412d6991c5d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `db92d26`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/testReport/
Max. process+thread count	4448 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-05-11T15:16:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 00s		No case conflicting files found.
+0 🆗	spotbugs	0m 01s		spotbugs executables are not available.
+0 🆗	codespell	0m 01s		codespell was not available.
+0 🆗	detsecrets	0m 01s		detect-secrets was not available.
+1 💚	@author	0m 00s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 01s		The patch appears to include 2 new or modified test files.
			_ HDFS-17384 Compile Tests _
+1 💚	mvninstall	108m 02s		HDFS-17384 passed
+1 💚	compile	7m 17s		HDFS-17384 passed
+1 💚	checkstyle	5m 38s		HDFS-17384 passed
+1 💚	mvnsite	8m 07s		HDFS-17384 passed
+1 💚	javadoc	7m 02s		HDFS-17384 passed
+1 💚	shadedclient	177m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	5m 46s		the patch passed
+1 💚	compile	4m 29s		the patch passed
+1 💚	javac	4m 29s		the patch passed
+1 💚	blanks	0m 00s		The patch has no blanks issues.
+1 💚	checkstyle	2m 53s		the patch passed
+1 💚	mvnsite	5m 08s		the patch passed
+1 💚	javadoc	4m 19s		the patch passed
+1 💚	shadedclient	193m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	6m 26s		The patch does not generate ASF License warnings.
		512m 36s

Subsystem	Report/Notes
GITHUB PR	#6806
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 a4a169f40776 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	HDFS-17384 / `db92d26`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/3/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/3/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ferhui · 2024-05-13T02:52:02Z

Thanks for contribution. Merged.

HDFS-17506. [FGL] Performance for phase 1

dbd8a7c

github-actions bot added HDFS trunk labels May 9, 2024

ferhui reviewed May 10, 2024

View reviewed changes

...dfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/fgl/FSNLockBenchmarkThroughput.java Outdated Show resolved Hide resolved

HDFS-17506. Add Usage for FSNLockBenchmarkThroughput

3b26cc4

HDFS-17506. Fix checkstyle

db92d26

ZanderXu mentioned this pull request May 11, 2024

HDFS-17520. [BugFix] TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed #6812

Merged

ferhui approved these changes May 11, 2024

View reviewed changes

ferhui merged commit 141b190 into apache:HDFS-17384 May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-17506. [FGL] Performance for phase 1 #6806

HDFS-17506. [FGL] Performance for phase 1 #6806

Uh oh!

ZanderXu commented May 9, 2024

Uh oh!

ferhui commented May 10, 2024

Uh oh!

ZanderXu commented May 10, 2024

Uh oh!

ferhui commented May 10, 2024

Uh oh!

ferhui left a comment

Uh oh!

Uh oh!

ferhui commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

ZanderXu commented May 11, 2024

Uh oh!

hadoop-yetus commented May 11, 2024

Uh oh!

hadoop-yetus commented May 11, 2024

Uh oh!

ferhui commented May 13, 2024

Uh oh!

Uh oh!

HDFS-17506. [FGL] Performance for phase 1 #6806

HDFS-17506. [FGL] Performance for phase 1 #6806

Uh oh!

Conversation

ZanderXu commented May 9, 2024

Uh oh!

ferhui commented May 10, 2024

Uh oh!

ZanderXu commented May 10, 2024

Uh oh!

ferhui commented May 10, 2024

Uh oh!

ferhui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ferhui commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

hadoop-yetus commented May 10, 2024

Uh oh!

ZanderXu commented May 11, 2024

Uh oh!

hadoop-yetus commented May 11, 2024

Uh oh!

hadoop-yetus commented May 11, 2024

Uh oh!

ferhui commented May 13, 2024

Uh oh!

Uh oh!