HDFS-16455. RBF: Add `zk-dt-secret-manager.jute.maxbuffer` property for Router's ZKDelegationTokenManager #3983

Neilxzn · 2022-02-11T11:51:24Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-16455

How was this patch tested?

no test

For code changes:

add zk-dt-secret-manager.jute.maxbufferproperty to default value 20000000, 20MB

jojochuang

Note: this is not just a RBF only change.
The ZKDT is used by YARN, KMS as well.

It's a good idea to specify jute buffer size. We've seen similar issues with KMS too.

jojochuang

The patch itself looks good. I think one of the problem today is there's no documentation around the zkdt usage unless you dig into the code. We should add a better doc as a follow-up jira.

hadoop-yetus · 2022-02-11T15:09:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 6s		trunk passed
+1 💚	compile	22m 22s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	19m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 6s		trunk passed
+1 💚	mvnsite	1m 41s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 28s		trunk passed
+1 💚	shadedclient	22m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 58s		the patch passed
+1 💚	compile	21m 37s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	21m 37s		the patch passed
+1 💚	compile	19m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	19m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 6s		the patch passed
+1 💚	mvnsite	1m 38s		the patch passed
+1 💚	javadoc	1m 10s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 44s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 37s		the patch passed
+1 💚	shadedclient	23m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 50s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		196m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/1/artifact/out/Dockerfile
GITHUB PR	#3983
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8e25a43a97f7 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `535e276`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/1/testReport/
Max. process+thread count	3158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Neilxzn · 2022-02-14T10:16:13Z

The patch itself looks good. I think one of the problem today is there's no documentation around the zkdt usage unless you dig into the code. We should add a better doc as a follow-up jira.

@jojochuang Thank you for your review. We should add a better doc around the zkdt usage. +1

...rc/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java

hadoop-yetus · 2022-02-15T05:29:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 1s		trunk passed
+1 💚	compile	22m 9s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	19m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 7s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 42s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 26s		trunk passed
+1 💚	shadedclient	21m 53s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 59s		the patch passed
+1 💚	compile	21m 31s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	21m 31s		the patch passed
+1 💚	compile	19m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	19m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 5s		the patch passed
+1 💚	mvnsite	1m 39s		the patch passed
+1 💚	javadoc	1m 9s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 44s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 36s		the patch passed
+1 💚	shadedclient	22m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 37s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		196m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/3/artifact/out/Dockerfile
GITHUB PR	#3983
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 5991516568fc 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e898cb2`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/3/testReport/
Max. process+thread count	1289 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-15T05:50:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 14s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 22s		trunk passed
+1 💚	compile	25m 38s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	21m 14s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 34s		trunk passed
+1 💚	shadedclient	25m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	24m 13s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	24m 13s		the patch passed
+1 💚	compile	21m 21s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 36s		the patch passed
+1 💚	javadoc	1m 5s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 41s		the patch passed
+1 💚	shadedclient	25m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 4s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		219m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/2/artifact/out/Dockerfile
GITHUB PR	#3983
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 378f1572afc2 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e898cb2`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/2/testReport/
Max. process+thread count	3059 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3983/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

add zk-dt-secret-manager.jute.maxbuffer config

535e276

Neilxzn changed the title ~~add zk-dt-secret-manager.jute.maxbuffer config~~ HDFS-16455. RBF: add zk-dt-secret-manager.jute.maxbuffer property for Router's ZKDelegationTokenManager Feb 11, 2022

jojochuang reviewed Feb 11, 2022

View reviewed changes

jojochuang approved these changes Feb 11, 2022

View reviewed changes

goiri changed the title ~~HDFS-16455. RBF: add zk-dt-secret-manager.jute.maxbuffer property for Router's ZKDelegationTokenManager~~ HDFS-16455. RBF: Add zk-dt-secret-manager.jute.maxbuffer property for Router's ZKDelegationTokenManager Feb 11, 2022

goiri reviewed Feb 14, 2022

View reviewed changes

xiezhineng added 2 commits February 15, 2022 10:09

fix some checkstyle error

a08a491

fix some checkstyle error 2

e898cb2

goiri approved these changes Feb 15, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16455. RBF: Add `zk-dt-secret-manager.jute.maxbuffer` property for Router's ZKDelegationTokenManager #3983

HDFS-16455. RBF: Add `zk-dt-secret-manager.jute.maxbuffer` property for Router's ZKDelegationTokenManager #3983

Neilxzn commented Feb 11, 2022

Uh oh!

jojochuang left a comment •

edited

Loading

Uh oh!

jojochuang left a comment

Uh oh!

hadoop-yetus commented Feb 11, 2022

Uh oh!

Neilxzn commented Feb 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Feb 15, 2022

Uh oh!

hadoop-yetus commented Feb 15, 2022

Uh oh!

Uh oh!

HDFS-16455. RBF: Add zk-dt-secret-manager.jute.maxbuffer property for Router's ZKDelegationTokenManager #3983

Are you sure you want to change the base?

HDFS-16455. RBF: Add zk-dt-secret-manager.jute.maxbuffer property for Router's ZKDelegationTokenManager #3983

Conversation

Neilxzn commented Feb 11, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

jojochuang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Feb 11, 2022

Uh oh!

Neilxzn commented Feb 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Feb 15, 2022

Uh oh!

hadoop-yetus commented Feb 15, 2022

Uh oh!

Uh oh!

HDFS-16455. RBF: Add `zk-dt-secret-manager.jute.maxbuffer` property for Router's ZKDelegationTokenManager #3983

HDFS-16455. RBF: Add `zk-dt-secret-manager.jute.maxbuffer` property for Router's ZKDelegationTokenManager #3983

jojochuang left a comment •

edited

Loading