HADOOP-18079. Upgrade Netty to 4.1.77. #3977

jojochuang · 2022-02-10T08:17:08Z

Output of mvn dependency:tree after the change:

[INFO] +- io.netty:netty-all:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-buffer:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-dns:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-haproxy:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-http:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-memcache:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-mqtt:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-redis:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-smtp:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-socks:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-stomp:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-codec-xml:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-common:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-tcnative-classes:jar:2.0.48.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-resolver:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-resolver-dns:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-rxtx:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-sctp:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-udt:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-epoll:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-native-unix-common:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-kqueue:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.74.Final:compile
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.74.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.74.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.74.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.74.Final:runtime
[INFO] |  +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.74.Final:runtime
[INFO] |  \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.74.Final:runtime

jojochuang · 2022-02-22T07:49:16Z

Failed tests don't look related; they don't fail in my local tree.

prasad-acit · 2022-04-07T16:32:08Z

Thanks @jojochuang for the PR.
Code changes are fine and i checked the test case. These are random and not related to the code changes.
@Hexiaoqiao / @hemanthboyina can you plz take a look?

jojochuang · 2022-04-08T02:59:02Z

I'll merge this PR later today, and then we can start rolling RCs for next 3.3.x and 3.2.x releases

prasad-acit · 2022-04-22T11:16:21Z

@jojochuang this PR can be merged?

dmmkr · 2022-04-25T18:06:32Z

@jojochuang , can we upgrade to 4.1.75.Final, so that the CVE's from netty-tcnative-classes:jar:2.0.48.Final can also be fixed.

jojochuang · 2022-04-25T18:22:13Z

@dmmkr would you share more details? Our use of netty shouldn't depend on tcnative.
Also if the CVE warning comes from OWASP, it is likely a false-positive: jeremylong/DependencyCheck#3865

brahmareddybattula · 2022-04-26T05:36:25Z

Looks to tc-native are false positive only.

I suggest, we can move to 4.1.75 as some of the downstream(1) already move to 4.1.75 like below and it contains good bug fixes (2)

Reference:
1)
https://issues.apache.org/jira/browse/SPARK-38541,
https://issues.apache.org/jira/browse/HIVE-26029,
https://issues.apache.org/jira/browse/ZOOKEEPER-4529

https://netty.io/news/2022/03/10/4-1-75-Final.html

jojochuang · 2022-04-28T00:49:53Z

Ok pushed a change to update to netty 4.1.75.
Basically other than version change, the only difference is that tcnative is no longer in the classpath.

jojochuang · 2022-04-28T22:09:36Z

The test failures do not reproduce in my local tree. I'm triggering a rebuild to double check.

brahmareddybattula · 2022-04-28T19:30:17Z

hadoop-project/pom.xml

Looks netty-transport given two times.. Here and #992

Very good catch. Thanks @brahmareddybattula . Updated.

brahmareddybattula · 2022-04-28T19:34:12Z

hadoop-project/pom.xml

Sorry to ask again, looks now 4.1.76 also available. May be we can raise another jira for this if not with this.

steveloughran · 2022-05-31T09:51:24Z

do these tests which fail go near netty code? that is: are they directly or indirectly related?

Change-Id: I70703c2fbd176a0d6b1c05a47ae67a83b185eb74

Change-Id: I4607d544d523b28fa80b311e66b0f9d5e5e55589

Change-Id: I3e3c77b3d0804fd05e6b319690a1e46ac569ed7b

Change-Id: Id5d6eef9e364e136b1f723b0876fb4234f8d96e1

Change-Id: I304914ae6dcf464be4662039d1ea683411a65e7b

jojochuang · 2022-07-12T19:54:31Z

A CVE was fixed in 4.1.77 so bump up the version again.

jojochuang · 2022-07-15T17:23:55Z

The javac warning is unrelated. It was there before this patch and exists even in trunk.

steveloughran

+1

steveloughran · 2022-07-18T09:41:46Z

@jojochuang how far back can we take this do you think?

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7) Change-Id: Idadb5c50329d74daf101829fc573edfa99b4eec9

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7) Change-Id: Idadb5c50329d74daf101829fc573edfa99b4eec9 (cherry picked from commit c545341)

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7)

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7) (cherry picked from commit 0c12873)

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang

…che#4592) Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7) (cherry picked from commit 0c12873) Change-Id: Id78e33554e8aa93bf7afd183fc533cfdd285b4ea

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7) Change-Id: I7cf7025e0f78816f6614531104ae584373cab7da

This list captures the current state of non-upstream changes in our branch that are not in the public repo. ---Changes cherry-picked to branch-3.3.6-dremio from branch-3.3.2-dremio--- The below changes were on branch-3.3.2-dremio and needed to be brought to branch-3.3.6-dremio to prevent regressing scenarios these changes addressed. HADOOP-18928: S3AFileSystem URL encodes twice where Path has trailing / (proposed) DX-69726: Bumping okie from 1.6.0 to 3.4.0 (CVE-2023-3635) DX-69726: Bumping okie from 1.6.0 to 3.4.0 (CVE-2023-3635) DX-66470: Allow for custom shared key signer for ABFS DX-66673: Backport HADOOP-18602. Remove netty3 dependency DX-66673: Backport MAPREDUCE-7434. Fix ShuffleHandler tests. Contributed by Tamas Domok DX-66673: Backport MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (apache#5311) DX-66673: Backport HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 apache#3259. Contributed by Szilard Nemeth. DX-66673: Backport HADOOP-17115. Replace Guava Sets usage by Hadoop's own Sets in hadoop-common and hadoop-tools (apache#2985) HADOOP-18676. jettison dependency override in hadoop-common lib DX-52816: Downgrade azure-data-lake-store-sdk to 2.3.3 to support dremio version. DX-52701: Remove node based module by Naveen Kumar DX-32012: Adding BatchList Iterator for ListFiles by “ajmeera.nagaraju” DX-18552: Make file status check optional in S3AFileSystem create() Add flag to skip native tests by Laurent Goujon DX-21904: Support S3 requester-pays headers by Brandon Huang DX-21471: Fix checking of use of OAuth credentials with AzureNativeFileSystem DX-19314: make new kms format configurable DX-17058 Add FileSystem to META-INF/services DX-17317 Fix incorrect parameter passed into AzureADAuthenticator-getTokenUsingClientCreds by TiffanyLam DX-17276 Azure AD support for StorageV1 by James Duong DX-17276 Add Azure AD support in Dremio's hadoop-azure library for Storage V1 support unwraps BindException in HttpServer2 ---Changes picked up by moving to 3.3.6--- The below changes were changes on branch-3.3.2-dremio that did not need to come to branch-3.3.6-dremio as the public 3.3.6 branch contained the fixes already. DX-67500: Backport HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. DX-66673: Backport HADOOP-18079. Upgrade Netty to 4.1.77. (apache#3977) DX-66673: Backport HADOOP-11245. Update NFS gateway to use Netty4 (apache#2832) (apache#4997) DX-64051: Bump jettison from 1.1 to 1.5.4 in hadoop/branch-3.3.2-dremio DX-64051: Bump jettison from 1.1 to 1.5.4 in hadoop/branch-3.3.2-dremio DX-63800 Bump commons-net from 3.6 to 3.9.0 to address CVE-2021-37533 DX-27168: removing org.codehaus.jackson Change-Id: I6cdb968e33826105caff96e1c3d2c6313a550689

jojochuang mentioned this pull request Apr 8, 2022

HDFS-16384. Upgrade Netty to 4.1.72.Final #4132

Closed

jojochuang changed the title ~~HADOOP-18079. Upgrade Netty to 4.1.74.~~ HADOOP-18079. Upgrade Netty to 4.1.75. Apr 28, 2022

brahmareddybattula reviewed Apr 29, 2022

View reviewed changes

jojochuang changed the title ~~HADOOP-18079. Upgrade Netty to 4.1.75.~~ HADOOP-18079. Upgrade Netty to 4.1.76. May 1, 2022

jojochuang changed the title ~~HADOOP-18079. Upgrade Netty to 4.1.76.~~ HADOOP-18079. Upgrade Netty to 4.1.77. Jul 12, 2022

jojochuang added 5 commits July 12, 2022 12:53

HADOOP-18079. Upgrade Netty to 4.1.74.

a0354c9

Change-Id: I70703c2fbd176a0d6b1c05a47ae67a83b185eb74

Harmonize netty version.

9a8f193

Change-Id: I4607d544d523b28fa80b311e66b0f9d5e5e55589

Update to netty 4.1.75

7c154ef

Change-Id: I3e3c77b3d0804fd05e6b319690a1e46ac569ed7b

Update to Netty 4.1.76

906981e

Change-Id: Id5d6eef9e364e136b1f723b0876fb4234f8d96e1

Bump version to 4.1.77.

8ee426f

Change-Id: I304914ae6dcf464be4662039d1ea683411a65e7b

jojochuang force-pushed the HADOOP-18079 branch from 74086b2 to 8ee426f Compare July 12, 2022 19:54

This comment was marked as outdated.

Sign in to view

apache deleted a comment from hadoop-yetus Jul 18, 2022

steveloughran approved these changes Jul 18, 2022

View reviewed changes

steveloughran merged commit a55ace7 into apache:trunk Jul 18, 2022

apache deleted a comment from hadoop-yetus Jul 18, 2022

jojochuang mentioned this pull request Jul 19, 2022

HADOOP-18079. Upgrade Netty to 4.1.77. (#3977) #4592

Merged

jojochuang mentioned this pull request Jul 19, 2022

HADOOP-18079. Upgrade Netty to 4.1.77. (#3977) #4593

Merged

4 tasks

jojochuang added a commit that referenced this pull request Jul 26, 2022

HADOOP-18079. Upgrade Netty to 4.1.77. (#3977) (#4592)

0c12873

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang (cherry picked from commit a55ace7)

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18079. Upgrade Netty to 4.1.77. (apache#3977)

4c32336

Upgrade netty to address CVE-2019-20444, CVE-2019-20445 CVE-2022-24823 Contributed by Wei-Chiu Chuang

HADOOP-18079. Upgrade Netty to 4.1.77. #3977

HADOOP-18079. Upgrade Netty to 4.1.77. #3977

Uh oh!

Conversation

jojochuang commented Feb 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jojochuang commented Feb 22, 2022

Uh oh!

prasad-acit commented Apr 7, 2022

Uh oh!

jojochuang commented Apr 8, 2022

Uh oh!

prasad-acit commented Apr 22, 2022

Uh oh!

dmmkr commented Apr 25, 2022

Uh oh!

jojochuang commented Apr 25, 2022

Uh oh!

brahmareddybattula commented Apr 26, 2022

Uh oh!

jojochuang commented Apr 28, 2022

Uh oh!

jojochuang commented Apr 28, 2022

Uh oh!

brahmareddybattula Apr 28, 2022

Choose a reason for hiding this comment

Uh oh!

jojochuang Apr 30, 2022

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula Apr 28, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran commented May 31, 2022

Uh oh!

jojochuang commented Jul 12, 2022

Uh oh!

This comment was marked as outdated.

jojochuang commented Jul 15, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Jul 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jojochuang commented Feb 10, 2022 •

edited

Loading