MINOR: log4j2 upgrade follow-up #18290

frankvicky · 2024-12-21T03:17:05Z

Please refer to #17373 (comment) for further details.
This PR does the following things:

Move deprecation warning to kafka-run-class
Remove unnecessary jacksonDatabindYaml dependency from modules
Add upgrade notes about log4j2.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

chia7712 · 2024-12-21T03:28:18Z

@frankvicky as @ijuma's comment in https://issues.apache.org/jira/browse/KAFKA-18247, please add the log4j2 transform ci to the upgrade note.

ijuma · 2024-12-21T04:40:52Z

docs/upgrade.html

                    See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223">KIP-750</a> for more details
                </li>
+                <li>
+                    Logging framework has been migrated from Log4j to Log4j2.


Shall we mention the behavior if you do not convert the configuration files? I believe there is a compatibility mode with some limitations?

Log4j provides a way to convert Log4j 1.x properties at runtime (Log4j Migration Guide).

To enable this feature, we can set both LOG4J_COMPATIBILITY=true and specify the configuration file using LOG4J_CONFIGURATION_FILE=file.

Do we want to automatically detect if log4j 1.x properties are used and, if so, automatically enable this mode? It seems like a huge problem if customers lose their logging while upgrading to 4.0+.

We can always enable that compatibility feature as the bridge jar is already included now.

Even better if it can be always enabled with no significant downside.

chia7712

@frankvicky thanks for your patch

chia7712 · 2024-12-22T04:03:40Z

build.gradle

    implementation project(':storage')
    implementation project(':server-common')
    implementation libs.slf4jApi
-    implementation libs.jacksonDatabindYaml


this is required as test-common needs to parse yaml in production code.

chia7712 · 2024-12-22T04:20:35Z

build.gradle

    implementation project(':raft')
    implementation project(':storage')
    implementation project(':server-common')
    implementation libs.slf4jApi


Could you please take a look at @ijuma's comment (#17373 (comment))

maybe we can create a variable to collect them and then add them to implementation for each module:

log4jRuntimeLibs = [ libs.log4j1Bridge2Api libs.jacksonDatabindYaml ] log4jLibs = [ libs.slf4jApi libs.slf4jLog4j2 libs.log4j2Api libs.log4j2Core ] ... implementation log4jLibs runtimeOnly log4jRuntimeLibs

I have tried it on my local machine but it led to tons of failure of tests.
I would keep debugging on my local machine.

chia7712 · 2024-12-22T04:25:53Z

docs/upgrade.html

                    See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223">KIP-750</a> for more details
                </li>
+                <li>
+                    Logging framework has been migrated from Log4j to Log4j2.


Log4j provides a way to convert Log4j 1.x properties at runtime (Log4j Migration Guide).

To enable this feature, we can set both LOG4J_COMPATIBILITY=true and specify the configuration file using LOG4J_CONFIGURATION_FILE=file.

ijuma · 2024-12-24T03:57:30Z

build.gradle

-    implementation libs.slf4jApi
    implementation libs.opentelemetryProto
    implementation libs.protobuf
+    implementation log4jLibs


Hmm, clients and streams should not have any log dependency besides slf4j. The idea is that client modules shouldn't include logging implementations while server modules should. Client modules run alongside user applications and the logging library should be aligned with the application logging library.

chia7712

@frankvicky could you please also remove log4j2CoreTest? it is unused.

chia7712 · 2024-12-24T08:28:31Z

config/tools-log4j2.yaml

 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+Configuration:


Could you please open a jira for this change? tools-log4j.properties is still used in code base

Sure.
https://issues.apache.org/jira/browse/KAFKA-18347

chia7712 · 2024-12-24T08:32:27Z

build.gradle

    implementation libs.junitPlatformLanucher
    implementation libs.junitJupiterApi
    implementation libs.junitJupiter
+    implementation log4jLibs


line#1656 implementation libs.slf4jApi is unnecessary now.

chia7712 · 2024-12-28T11:52:19Z

bin/kafka-run-class.sh

+      LOG4J1_CONFIG=$(echo "$KAFKA_LOG4J_OPTS" | grep -o 'log4j\.configuration=\S*' | cut -d'=' -f2)
+
+      # Enable Log4j 1.x configuration compatibility mode for Log4j 2
+      export LOG4J_COMPATIBILITY=true


@frankvicky could you please test the compatibility on your local?

From document, set one of confituration from log4j1.compatibility and log4j.configuration if needing to install log4j1 to log4j2 bridge. It looks like we can set LOG4J_COMPATIBILITY=true only if LOG4J1_CONFIG has properties file. Also, setting LOG4J_CONFIGURATION_FILE="$LOG4J1_CONFIG" only if LOG4J1_CONFIG doesn't have properties file.

https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#ConfigurationCompatibility

frankvicky · 2024-12-29T06:58:17Z

Hi @chia7712
I have tested log4j compatibility on my local machine.

Explicit set KAFKA_LOG4J_OPTS:

Leave KAFKA_LOG4J_OPTS as null:

FrankYang0529

LGTM. Thanks for the patch. With log4j 1 properties, I can see log on console.

chia7712 · 2024-12-29T15:28:53Z

build.gradle

          // ZooKeeper (potentially older and containing CVEs)
          libs.nettyHandler,
          libs.nettyTransportNativeEpoll,
+          libs.slf4jApi,


why we need this change?

spotbugsSlf4j - configuration for the SLF4J provider to run SpotBugs \--- org.slf4j:slf4j-simple:2.0.0 \--- org.slf4j:slf4j-api:2.0.0

spotbugs requires the 2.x slf4j, so forcing the version can cause log issue in running spotbugs

I previously encountered version inconsistency issues on my local machine. While those issues have been resolved, this change represents a modification that should have been removed but was accidentally missed during cleanup. 😬

Reviewers: PoAn Yang <[email protected]>, Chia-Ping Tsai <[email protected]>

github-actions bot added triage PRs from the community build Gradle build or GitHub Actions small Small PRs labels Dec 21, 2024

chia7712 added the ci-approved label Dec 21, 2024

frankvicky force-pushed the log4j2-follow-up branch from a5214a4 to 3b5a4c6 Compare December 21, 2024 04:10

ijuma reviewed Dec 21, 2024

View reviewed changes

github-actions bot removed the triage PRs from the community label Dec 22, 2024

chia7712 reviewed Dec 22, 2024

View reviewed changes

frankvicky force-pushed the log4j2-follow-up branch 2 times, most recently from c95cd3c to c69d421 Compare December 23, 2024 16:14

github-actions bot removed the small Small PRs label Dec 23, 2024

ijuma reviewed Dec 24, 2024

View reviewed changes

chia7712 reviewed Dec 24, 2024

View reviewed changes

frankvicky force-pushed the log4j2-follow-up branch 3 times, most recently from e6000ec to 5af4fb4 Compare December 24, 2024 11:09

chia7712 mentioned this pull request Dec 27, 2024

KAFKA-9366: Upgrade log4j to log4j2 #17373

Merged

3 tasks

chia7712 reviewed Dec 28, 2024

View reviewed changes

frankvicky force-pushed the log4j2-follow-up branch from 5af4fb4 to 279ede0 Compare December 28, 2024 16:19

FrankYang0529 approved these changes Dec 29, 2024

View reviewed changes

chia7712 reviewed Dec 29, 2024

View reviewed changes

MINOR: log4j2 upgrade follow-up

14fb9ed

frankvicky force-pushed the log4j2-follow-up branch from 279ede0 to 14fb9ed Compare December 29, 2024 15:44

chia7712 approved these changes Dec 30, 2024

View reviewed changes

chia7712 merged commit 585b7db into apache:trunk Dec 30, 2024
9 checks passed

chia7712 pushed a commit that referenced this pull request Dec 30, 2024

MINOR: log4j2 upgrade follow-up (#18290)

98d4c5b

Reviewers: PoAn Yang <[email protected]>, Chia-Ping Tsai <[email protected]>

tedyu pushed a commit to tedyu/kafka that referenced this pull request Jan 6, 2025

MINOR: log4j2 upgrade follow-up (apache#18290)

682af0c

Reviewers: PoAn Yang <[email protected]>, Chia-Ping Tsai <[email protected]>

MINOR: log4j2 upgrade follow-up #18290

MINOR: log4j2 upgrade follow-up #18290

Uh oh!

Conversation

frankvicky commented Dec 21, 2024

Committer Checklist (excluded from commit message)

Uh oh!

chia7712 commented Dec 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankvicky commented Dec 29, 2024

Uh oh!

FrankYang0529 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants