YARN-11391 Add yarn RM DNS support #5196

lnbest0707 · 2022-12-07T22:55:28Z

Description of PR

Add DNS discovery support for YARN RM services. It would accept DNS multi-A record as input from yarn-site.xml. Upon configurations, it would resolve the DNS and do reverse lookup. Both server (RM, NM, app container) and client could utilize the same DNS functionality. Besides, it is backward compatible with plain DNS A record.

How was this patch tested?

local mvn test

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

slfan1989 · 2022-12-07T23:32:48Z

@lnbest0707 Thank you very much for your contribution, I think it would be better to make this part an optional configuration.

Why the code submission is 2 different users?

lnbest0707 · 2022-12-08T00:19:56Z

Hi @slfan1989 thanks for your reply.

I think it would be better to make this part an optional configuration.
None DNS resolving logics would be triggered if "yarn.resourcemanager.ha.resolve-needed" set as false (which is its default value).

Why the code submission is 2 different users?
May I know where there are two people in code commits? The other name is the key name for my github access token.

hadoop-yetus · 2022-12-08T03:02:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 45s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 11s		trunk passed
+1 💚	compile	23m 1s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	20m 26s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 15s		trunk passed
+1 💚	mvnsite	3m 49s		trunk passed
+1 💚	javadoc	3m 7s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 40s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 36s		trunk passed
+1 💚	shadedclient	21m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 16s		the patch passed
+1 💚	compile	22m 24s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
-1 ❌	javac	22m 24s	/results-compile-javac-root-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt	root-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 18 new + 2815 unchanged - 0 fixed = 2833 total (was 2815)
+1 💚	compile	20m 23s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 ❌	javac	20m 23s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt	root-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 generated 18 new + 2614 unchanged - 0 fixed = 2632 total (was 2614)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 39s	/results-checkstyle-root.txt	root: The patch generated 21 new + 184 unchanged - 0 fixed = 205 total (was 184)
+1 💚	mvnsite	3m 43s		the patch passed
+1 💚	javadoc	2m 58s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 40s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	2m 5s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	21m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 24s		hadoop-common in the patch passed.
-1 ❌	unit	1m 17s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 44s		hadoop-yarn-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		245m 3s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
	Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147]
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/1/artifact/out/Dockerfile
GITHUB PR	#5196
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 2592849dacae 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e897e65`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/1/testReport/
Max. process+thread count	2362 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2022-12-08T06:05:43Z

...rn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java

+          continue;
+        }
+        LOG.info("Resolved addresses for " + addr +
+            " is " + Arrays.toString(resolvedHostNames));


record logs, generally using placeholders in this way.

slfan1989 · 2022-12-08T06:05:52Z

...rn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/HAUtil.java

+          resolvedHostNames = dnr.getAllResolvedHostnameByDomainName(
+              address.getHostName(), requireFQDN);
+        } catch (UnknownHostException e) {
+          LOG.warn("Exception in resolving socket address "


slfan1989 · 2022-12-08T06:09:45Z

@lnbest0707 From my personal point of view, the quality of the code is good. We should add some instructions to tell users how to use these configurations.

hadoop-yetus · 2022-12-08T09:32:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 22s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 36s		trunk passed
+1 💚	compile	24m 35s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	23m 12s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 52s		trunk passed
+1 💚	mvnsite	3m 43s		trunk passed
+1 💚	javadoc	3m 9s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 35s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 39s		trunk passed
+1 💚	shadedclient	21m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 16s		the patch passed
+1 💚	compile	22m 17s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	22m 17s		the patch passed
+1 💚	compile	20m 22s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 35s	/results-checkstyle-root.txt	root: The patch generated 2 new + 184 unchanged - 0 fixed = 186 total (was 184)
+1 💚	mvnsite	3m 40s		the patch passed
+1 💚	javadoc	2m 49s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 34s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	2m 11s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	21m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 19s		hadoop-common in the patch passed.
+1 💚	unit	1m 17s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 44s		hadoop-yarn-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		247m 3s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
	Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/2/artifact/out/Dockerfile
GITHUB PR	#5196
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 692ce489485a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e2140c7`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/2/testReport/
Max. process+thread count	3159 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5196/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

lnbest0707 · 2022-12-08T17:37:08Z

@slfan1989 thanks a lot for your review. Will add the instructions, do you have any suggestions where to add it? Just in the commit message or in a separate file (or which file).

Besides, I feel the issue reported by Jenkins a false alert. The "Unsynchronized access" reported by it is a READ call from a log function. I don't think we should add any locks on logging call. Please share any suggestions to bypass it. Thanks.

slfan1989 · 2022-12-12T15:16:29Z

Besides, I feel the issue reported by Jenkins a false alert. The "Unsynchronized access" reported by it is a READ call from a log function. I don't think we should add any locks on logging call. Please share any suggestions to bypass it. Thanks.

I agree with you.
LGTM.

We better fix checkstyle.

lnbest0707 · 2022-12-13T03:02:02Z

Duplicate to #5213

@slfan1989 somehow my amending commit went to a new PR as above. Please check that one and I will close this. Sorry for inconvenience.

YARN-11391 Add yarn RM DNS support

e897e65

YARN-11391 Fix Jenkins reported issue

e2140c7

slfan1989 reviewed Dec 8, 2022

View reviewed changes

lnbest0707 closed this Dec 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11391 Add yarn RM DNS support #5196

YARN-11391 Add yarn RM DNS support #5196

Uh oh!

lnbest0707 commented Dec 7, 2022

Uh oh!

slfan1989 commented Dec 7, 2022

Uh oh!

lnbest0707 commented Dec 8, 2022

Uh oh!

hadoop-yetus commented Dec 8, 2022

Uh oh!

slfan1989 Dec 8, 2022

Uh oh!

slfan1989 Dec 8, 2022

Uh oh!

slfan1989 Dec 8, 2022

Uh oh!

slfan1989 commented Dec 8, 2022

Uh oh!

hadoop-yetus commented Dec 8, 2022

Uh oh!

lnbest0707 commented Dec 8, 2022

Uh oh!

slfan1989 commented Dec 12, 2022

Uh oh!

lnbest0707 commented Dec 13, 2022

Uh oh!

Uh oh!

YARN-11391 Add yarn RM DNS support #5196

YARN-11391 Add yarn RM DNS support #5196

Uh oh!

Conversation

lnbest0707 commented Dec 7, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

slfan1989 commented Dec 7, 2022

Uh oh!

lnbest0707 commented Dec 8, 2022

Uh oh!

hadoop-yetus commented Dec 8, 2022

Uh oh!

slfan1989 Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

slfan1989 Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

slfan1989 Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Dec 8, 2022

Uh oh!

hadoop-yetus commented Dec 8, 2022

Uh oh!

lnbest0707 commented Dec 8, 2022

Uh oh!

slfan1989 commented Dec 12, 2022

Uh oh!

lnbest0707 commented Dec 13, 2022

Uh oh!

Uh oh!