HDFS-16488. [SPS]: Expose metrics to JMX for external SPS #4035

tomscut · 2022-02-26T14:15:00Z

Currently, external SPS has no monitoring metrics. We do not know how many blocks are waiting to be processed, how many blocks are waiting to be retried, and how many blocks have been migrated.

We can expose these metrics in JMX for easy collection and display by monitoring systems.

For example, in our cluster, we exposed these metrics to JMX, collected by JMX-Exporter and combined with Prometheus, and finally display by Grafana.

hadoop-yetus · 2022-02-26T22:29:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	37m 46s		trunk passed
+1 💚	compile	1m 43s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 26s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 38s		trunk passed
+1 💚	javadoc	1m 10s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 48s		trunk passed
+1 💚	shadedclient	27m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 30s		the patch passed
+1 💚	compile	1m 34s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
-1 ❌	javac	1m 34s	/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt	hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 466 unchanged - 0 fixed = 467 total (was 466)
+1 💚	compile	1m 27s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 55s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	1m 28s		the patch passed
+1 💚	javadoc	0m 59s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	4m 4s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html	hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	26m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	376m 52s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		492m 46s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs
	Primitive value is boxed and then immediately unboxed in org.apache.hadoop.hdfs.server.sps.metrics.ExternalSPSBeanMetrics.updateMovementFinishedBlocksCount() At ExternalSPSBeanMetrics.java:then immediately unboxed in org.apache.hadoop.hdfs.server.sps.metrics.ExternalSPSBeanMetrics.updateMovementFinishedBlocksCount() At ExternalSPSBeanMetrics.java:[line 87]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/1/artifact/out/Dockerfile
GITHUB PR	#4035
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 673b06c6b8d6 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 48f6ce4244d90348388cc7f14fbf5b0847321189
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/1/testReport/
Max. process+thread count	2129 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-28T10:07:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	37m 44s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 39s		trunk passed
+1 💚	shadedclient	26m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 22s		the patch passed
+1 💚	compile	1m 27s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 27s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 51s		the patch passed
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 31s		the patch passed
+1 💚	shadedclient	25m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	351m 28s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		463m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/2/artifact/out/Dockerfile
GITHUB PR	#4035
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 0a4e93122a78 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8addfa7ea1a95aa9289d6117042189233ad7a527
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/2/testReport/
Max. process+thread count	1882 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-03-04T05:22:25Z

Hi @tasanuma @jojochuang @ferhui , could you please take a look? Thanks.

hadoop-yetus · 2022-03-31T11:14:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 59s		trunk passed
+1 💚	compile	1m 34s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 32s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 24s		trunk passed
+1 💚	shadedclient	25m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	1m 23s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 23s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 22s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 28s		the patch passed
+1 💚	shadedclient	25m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	425m 57s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		540m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/3/artifact/out/Dockerfile
GITHUB PR	#4035
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux c3b03cc805d4 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2dd678f`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/3/testReport/
Max. process+thread count	2015 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4035/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-04-08T09:47:07Z

Hi @umamaheswararao @ayushtkn @tasanuma @ferhui , could you please take a look? Thank you very much.

tomscut · 2022-04-13T06:06:18Z

Hi @umamaheswararao @ayushtkn @tasanuma @ferhui
Currently, external SPS has no metrics. We do not know the migration progress of the corresponding block, adding metrics are helpful. Could you please take a look? Thank you very much.

ferhui

LGTM

tomscut · 2022-04-18T02:24:57Z

Thanks @ferhui for your review.

ayushtkn

LGTM

tomscut · 2022-04-22T01:36:39Z

Thanks @ayushtkn for your review.

ferhui · 2022-04-24T06:55:06Z

@tomscut Thanks for your contribution. @ayushtkn Thanks for your review! Merged

ferhui · 2022-04-24T06:59:21Z

@tomscut could you please update HDFS-16488 status. I couldn't log in apache site...

ferhui · 2022-04-24T07:46:46Z

Updated

tomscut · 2022-04-24T08:38:14Z

Thanks @ferhui and @ayushtkn .

yuanboliu · 2022-04-24T08:41:56Z

@tomscut @ferhui @ayushtkn
testExternalSPSMetrics is a duplicated function name which is already defined in TestExternalStoragePolicySatisfier.java
this patch is breaking trunk build, please fix it, thanks.

tomscut · 2022-04-24T08:54:20Z

@tomscut @ferhui @ayushtkn testExternalSPSMetrics is a duplicated function name which is already defined in TestExternalStoragePolicySatisfier.java this patch is breaking trunk build, please fix it, thanks.

Thanks @yuanboliu for pointing this out. I will fix this.

)" This reverts commit acc0e0a.

)" (#4232)

ferhui · 2022-04-24T09:20:38Z

Sorry, revert it.
Next time, will trigger CI if the PR is pending for a long time.

tomscut · 2022-04-24T09:22:14Z

I am sorry. I shouldn't use the same method name.

…ache#4035)" (apache#4232)

HDFS-16488. [SPS]: Expose metrics to JMX for external SPS

2dd678f

tomscut force-pushed the HDFS-16488 branch from 8addfa7 to 2dd678f Compare March 31, 2022 02:12

ferhui approved these changes Apr 18, 2022

View reviewed changes

ayushtkn approved these changes Apr 18, 2022

View reviewed changes

ferhui merged commit acc0e0a into apache:trunk Apr 24, 2022

tomscut mentioned this pull request Apr 24, 2022

HDFS-16560. [SPS]: Duplicate method names in TestExternalStoragePolic… #4231

Closed

ferhui added a commit that referenced this pull request Apr 24, 2022

Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035

cd13853

)" This reverts commit acc0e0a.

ferhui mentioned this pull request Apr 24, 2022

Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS" #4232

Merged

ferhui added a commit that referenced this pull request Apr 24, 2022

Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035

17d64ba

)" (#4232)

tomscut mentioned this pull request Apr 24, 2022

HDFS-16488. [SPS]: Expose metrics to JMX for external SPS #4234

Merged

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (apache#4035)

427ff9d

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (ap…

73b3e15

…ache#4035)" (apache#4232)

HDFS-16488. [SPS]: Expose metrics to JMX for external SPS #4035

HDFS-16488. [SPS]: Expose metrics to JMX for external SPS #4035

Uh oh!

Conversation

tomscut commented Feb 26, 2022

Uh oh!

hadoop-yetus commented Feb 26, 2022

Uh oh!

hadoop-yetus commented Feb 28, 2022

Uh oh!

tomscut commented Mar 4, 2022

Uh oh!

hadoop-yetus commented Mar 31, 2022

Uh oh!

tomscut commented Apr 8, 2022

Uh oh!

tomscut commented Apr 13, 2022

Uh oh!

ferhui left a comment

Choose a reason for hiding this comment

Uh oh!

tomscut commented Apr 18, 2022

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

tomscut commented Apr 22, 2022

Uh oh!

ferhui commented Apr 24, 2022

Uh oh!

ferhui commented Apr 24, 2022

Uh oh!

ferhui commented Apr 24, 2022

Uh oh!

tomscut commented Apr 24, 2022

Uh oh!

yuanboliu commented Apr 24, 2022

Uh oh!

tomscut commented Apr 24, 2022

Uh oh!

ferhui commented Apr 24, 2022

Uh oh!

tomscut commented Apr 24, 2022

Uh oh!

Uh oh!