KAFKA-9366: Upgrade log4j to log4j2 #17373

frankvicky · 2024-10-04T15:43:35Z

JIRA: KAFKA-9366

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

milestone 1

frankvicky · 2024-10-04T16:43:26Z

This is the initial version. I'd like to run it on CI first.

mumrah

Thanks for tackling this @frankvicky.

Is it really necessary to rename the log4j config files? The KIP mentions a system property that allows us to load log4j.properties files https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158870552#KIP653:Upgradelog4jtolog4j2-Compatibility,Deprecation,andMigrationPlan

I could understand migrating our internal log4j configs for tests and such, but I think we should avoid forcing a change in our production configs.

I also see that this page mentions a log4j to log4j2 bridge https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html. Would that help us at all?

mumrah · 2024-10-04T17:36:06Z

build.gradle

+    testRuntimeOnly libs.slf4jLog4j2
+    testRuntimeOnly libs.junitPlatformLanucher


These (and other similar testRuntimeOnly) should be put into the runtimeTestLibs definition

mumrah · 2024-10-04T17:36:26Z

build.gradle

    implementation libs.slf4jApi
    implementation libs.jacksonAnnotations
    implementation libs.jacksonDatabind
+    implementation libs.bndlib


What's this dependency for?

Hi @mumrah
I add this to fix the warning during build:

/home/frankvicky/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.24.1/7ebeb12c20606373005af4232cd0ecca72613dda/log4j-api-2.24.1.jar(/org/apache/logging/log4j/Level.class): warning: Cannot find annotation method 'value()' in type 'BaselineIgnore': class file for aQute.bnd.annotation.baseline.BaselineIgnore not found

I'm not convinced we need that dependency. Also it seems to complain about an annotation so at least we should not need it at runtime, so we should not include it in our distribution package. Currently it's included in the artifact generated by releaseTarGz.

Fair enough, I will try to solve this one.

The BND annotations are intentionally in the provided Maven scope of all Log4j artifacts, so that these annotations with CLASS retention do not end up in the runtime classpath. You can do the same and add them as compileOnly in Gradle.

The compiler warnings should disappear once JDK-8342833 is fixed.
Untile then we will remove the outdated ones (see apache/logging-log4j2#3133) in the next Log4j release, which should remove the warning on Level.

Thanks for the information.
I have already changed its scope to compile time.
PTAL 😄

mumrah · 2024-10-04T17:47:01Z

bin/kafka-server-start.sh

 base_dir=$(dirname $0)

 if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
-    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"


This could break some existing Kafka installations. If users are extracting in place or copying previous config files to a new installation directory, they will be expecting the log4j.properties to still work.

The switch from the legacy to the new configuration format can be based on the presence of specific files:

if [ -f "$base_dir/../config/log4j.properties" ]; then echo DEPRECATED: Using Log4j 1.x configuration file \$KAFKA_HOME/config/log4j.properties >&2 echo To use a Log4j 2.x configuration, create a \$KAFKA_HOME/config/log4j2.xml file and remove the Log4j 1.x configration. >&2 echo See https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#Log4j2ConfigurationFormat for details about Log4j configuration file migration. >&2 export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties" elif [ -f "$base_dir/../config/log4j2.xml" ]; then export KAFKA_LOG4J_OPTS="-Dlog4j2.configurationFile=$base_dir/../config/log4j2.xml" fi

I’m not sure why we override the KAFKA_LOG4J_OPTS here. We typically allow users to define custom KAFKA_LOG4J_OPTS. Moreover, overriding KAFKA_LOG4J_OPTS can break many end-to-end tests, as they often create log4j configurations dynamically and pass them through KAFKA_LOG4J_OPTS

Noted that we do not require users to strictly use the path $base_dir/../config/log4j2.xml.

frankvicky · 2024-10-06T03:32:10Z

Hello @mumrah
Thanks for your feedback. Unfortunately, I barely missed the KIP for some reason, but I'll take a look and adjust the PR accordingly. 😺

revert unnecessary change

chia7712 · 2024-12-20T16:21:41Z

@frankvicky Could you please open minor to address following items?

move the deprecation warnings to kafka-run-class KAFKA-9366: Upgrade log4j to log4j2 #17373 (comment)
remove unnecessary reference of jacksonDatabindYaml KAFKA-9366: Upgrade log4j to log4j2 #17373 (comment)
add upgrade notes KAFKA-9366: Upgrade log4j to log4j2 #17373 (comment)

jolshan · 2024-12-20T16:37:36Z

@jolshan thanks for your report. the known issue caused by this PR is the "logger" of connect. And I run the one of test (transactions_upgrade_test.py) on my local:

Interesting. This one fails consistently on our infra (and succeeds without this change). I wonder if there is some different test configuration that causes the issue.

This reverts commit b37b89c.

chia7712 · 2024-12-20T18:03:46Z

There are many tests which are failed quickly.

[DEBUG - 2024-12-18 22:36:58,718 - remoteaccount - _log - lineno:180]: ubuntu@worker47: Running ssh command: tail -c +1 /mnt/kafka/server-start-stdout-stderr.log | grep 'Kafka\s*Server.*started'
[DEBUG - 2024-12-18 22:36:58,974 - jmx - start_jmx_tool - lineno:56]: ubuntu@worker47: Not starting jmx tool because no jmx objects are defined
[DEBUG - 2024-12-18 22:36:58,974 - remoteaccount - _log - lineno:180]: ubuntu@worker47: Running ssh command: ps ax | grep -i kafka.Kafka | grep -v grep | awk '{print $1}'
[INFO  - 2024-12-18 22:36:58,983 - runner_client - log - lineno:459]: RunnerClient: kafkatest.tests.core.transactions_upgrade_test.TransactionsUpgradeTest.test_transactions_upgrade.from_kafka_version=3.1.2.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=False.group_protocol=None: Tearing down...

It can pass the check of grep 'Kafka\s*Server.*started' (kraft controller) but the check of process id fails. I open a MINOR to add more log (#18286) @jolshan Could you please run the patch on your infra?

jolshan · 2024-12-20T19:10:54Z

@chia7712 sure. I will run it and post the results

jolshan · 2024-12-20T19:44:36Z

Here's the result. https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-12-20--001.85d9ae90-e619-4689-a395-6d4075645b53--1734723292--chia7712--add_log_pid--acd9853691/report.html

This pull request replaces Log4j with Log4j2 across the entire project, including dependencies, configurations, and code. The notable changes are listed below: 1. Introduce Log4j2 Instead of Log4j 2. Change Configuration File Format from Properties to YAML 3. Adds warnings to notify users if they are still using Log4j properties, encouraging them to transition to Log4j2 configurations Co-authored-by: Lee Dongjin <[email protected]> Reviewers: Luke Chen <[email protected]>, Mickael Maison <[email protected]>, Chia-Ping Tsai <[email protected]>

trnguyencflt · 2025-01-08T14:30:16Z

This PR breaks downstream project that depends on kafka_2.13 and reload4j because there is clashing in class LoggingEvent, which exists in reload4j and log4j-1.2-api jars. The application will crash with this exception

Encountered fatal fault: caught exception
java.lang.IncompatibleClassChangeError: class org.apache.log4j.bridge.LogEventAdapter overrides final method org.apache.log4j.spi.LoggingEvent.getProperty(Ljava/lang/String;)Ljava/lang/String;
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:59)
	at org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
	at org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:89)
	at org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
	at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
	at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
	at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)
	at org.apache.commons.logging.impl.Log4jApiLogFactory$LogAdapter.getContext(Log4jApiLogFactory.java:161)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
	at org.apache.commons.logging.impl.Log4jApiLogFactory.getInstance(Log4jApiLogFactory.java:210)
	at org.apache.commons.logging.impl.Log4jApiLogFactory.getInstance(Log4jApiLogFactory.java:205)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:987)

ppkarwasz · 2025-01-08T14:56:16Z

This PR breaks downstream project that depends on kafka_2.13 and reload4j because there is clashing in class LoggingEvent, which exists in reload4j and log4j-1.2-api jars.

You can not have both reload4j and log4j-1.2-api on your classpath, since they are both replacements of log4j:log4j.

ijuma · 2025-01-08T15:05:14Z

The point is that kafka was including reload4j before and hence many other projects aligned with that. It's one thing to cause logging to change, it's another to cause projects not to start anymore with a IncompatibleClassChangeError - that's a much bigger deal.

chia7712 · 2025-01-08T15:58:11Z

This PR breaks downstream project that depends on kafka_2.13 and reload4j

I assume the scenario you describe is to add "kafka_2.13" as dependency for the downstream project. The log4j2 are declared as "implementation" in core module, so downstream project should not "include" log4j2 in the dependencies automatically. not sure why log4j-1.2-api is included in the downstream project. @trnguyencflt Do you add log4j-1.2-api to your project manually?

chia7712 · 2025-01-08T15:59:22Z

@trnguyencflt thanks for your report. I'd like to reduce the gap of upgrading to kafka 4.0 as much as possible, so please share the details to me

trnguyencflt · 2025-01-09T17:35:30Z

@chia7712 we include kafka_2.13 as dependency via maven https://github.com/confluentinc/kafka-rest/blob/d6405f714e0e453d6eb5b9a496faac4202c89cab/kafka-rest/pom.xml#L42, and it brings log4j-1.2-api as a runtime dependency (below is from mvn dependency:tree)

[INFO] |  +- org.apache.logging.log4j:log4j-1.2-api:jar:2.24.1:runtime

chia7712 · 2025-01-09T19:03:24Z

@trnguyencflt thanks for your response. log4j-1.2-api is used to convert log4j.properties at runtime, and hence maybe we can remove it from gradle runtime scope and then add it into distribution directly. With that change, log4j-1.2-api gets removed from published pom file but it remains in the distribution.

this is the pom file with above approach and there is not log4j-1.2-api

chia7712@chia7712-ubuntu:~/.m2/repository/org/apache/kafka/kafka_2.13/4.1.0-SNAPSHOT$ cat kafka_2.13-4.1.0-SNAPSHOT.pom | grep log4j
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-slf4j-impl</artifactId>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-api</artifactId>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-core</artifactId>

By contrast, log4j-1.2-api is still included by distribution.

chia7712@chia7712-ubuntu:~/project/kafka/core/build/distributions/kafka_2.13-4.1.0-SNAPSHOT$ ls libs | grep log4j
log4j-1.2-api-2.24.1.jar
log4j-api-2.24.1.jar
log4j-core-2.24.1.jar
log4j-slf4j-impl-2.24.1.jar

frankvicky · 2025-01-10T02:21:07Z

Hi @trnguyencflt
Thanks for report.
I have filed KAFKA-18466 to track this issue.
I will filed a PR for this issue soon.

chia7712 · 2025-01-13T07:07:40Z

@trnguyencflt we have resolved the issue in #18472. If you have some free time, could you please test it on your project?
Please don't hesitate to provide any feedback if you think something has been overlooked. Thank you!

…ing kafka JIRA: KAFKA-18752 In KAFKA-18466 we remove the slf4j impl from runtime scope to fix the compatible issue (see apache#17373 (comment)). However, that also causes a side effect that in running some tasks slf4k can't find any provider as they are included to "release-only" We could fix this by adding releasOnly to class path

frankvicky and others added 2 commits October 4, 2024 23:31

WIP

4cc09d9

milestone 1

Merge branch 'trunk' into KAFKA-9366

d71bbdf

github-actions bot added streams core Kafka Broker tools connect kraft mirror-maker-2 storage Pull requests that target the storage module build Gradle build or GitHub Actions docker Official Docker image clients labels Oct 4, 2024

frankvicky force-pushed the KAFKA-9366 branch from 093c8d6 to 0ecc326 Compare October 4, 2024 16:37

frankvicky marked this pull request as ready for review October 4, 2024 16:42

frankvicky added the do-not-merge PRs that are only open temporarily and should not be merged label Oct 4, 2024

mumrah reviewed Oct 4, 2024

View reviewed changes

frankvicky marked this pull request as draft October 6, 2024 03:19

frankvicky added 2 commits October 10, 2024 21:53

KAFKA-9366: Upgrade log4j to log4j2

347388a

Merge branch 'trunk' into KAFKA-9366

9fc1203

frankvicky force-pushed the KAFKA-9366 branch from 0ecc326 to 9fc1203 Compare October 10, 2024 13:54

frankvicky marked this pull request as ready for review October 10, 2024 13:55

frankvicky force-pushed the KAFKA-9366 branch from 0c1e6bf to 4b722a5 Compare October 10, 2024 14:27

frankvicky marked this pull request as draft October 10, 2024 16:23

frankvicky added 2 commits October 11, 2024 19:53

KAFKA-9366: Cleanup

d12f311

Merge branch 'trunk' into KAFKA-9366

7200b43

frankvicky force-pushed the KAFKA-9366 branch from 4b722a5 to 7200b43 Compare October 11, 2024 11:53

frankvicky marked this pull request as ready for review October 11, 2024 11:54

revert unnecessary change

7419f5f

revert unnecessary change

jolshan added a commit to jolshan/kafka that referenced this pull request Dec 20, 2024

Revert "KAFKA-9366 Upgrade log4j to log4j2 (apache#17373)"

2f7729d

This reverts commit b37b89c.

chia7712 mentioned this pull request Dec 20, 2024

MINOR: upgrade ubuntu from trusty to bionic #18286

Closed

3 tasks

frankvicky mentioned this pull request Dec 21, 2024

MINOR: log4j2 upgrade follow-up #18290

Merged

3 tasks

frankvicky mentioned this pull request Jan 10, 2025

KAFKA-18466: Remove log4j-1.2-api from runtime scope while keeping it in distribution package #18472

Merged

3 tasks

GunalKupta mentioned this pull request Mar 28, 2025

DGS-20118: Upgrade log4j to log4j2 confluentinc/schema-registry#3642

Merged

4 tasks

This was referenced Apr 1, 2025

KNET-18079: Migrate reload4j (log4j1) to log4j2 confluentinc/kafka-rest#1368

Merged

KNET-18079: Upgrade log4j1 to log4j2 confluentinc/rest-utils#561

Merged

GunalKupta mentioned this pull request Apr 9, 2025

Convert XML log4j2 properties files to YAML confluentinc/schema-registry#3663

Merged

4 tasks

frankvicky mentioned this pull request Apr 17, 2025

KAFKA-18752: Fix the warning of no-operation caused by slf4j in building kafka #19501

Open

		testRuntimeOnly libs.slf4jLog4j2
		testRuntimeOnly libs.junitPlatformLanucher

KAFKA-9366: Upgrade log4j to log4j2 #17373

KAFKA-9366: Upgrade log4j to log4j2 #17373

Uh oh!

Conversation

frankvicky commented Oct 4, 2024

Committer Checklist (excluded from commit message)

Uh oh!

frankvicky commented Oct 4, 2024

Uh oh!

mumrah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frankvicky commented Oct 6, 2024

Uh oh!

chia7712 commented Dec 20, 2024

Uh oh!

jolshan commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chia7712 commented Dec 20, 2024

Uh oh!

jolshan commented Dec 20, 2024

Uh oh!

jolshan commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trnguyencflt commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ppkarwasz commented Jan 8, 2025

Uh oh!

ijuma commented Jan 8, 2025

Uh oh!

chia7712 commented Jan 8, 2025

Uh oh!

chia7712 commented Jan 8, 2025

Uh oh!

trnguyencflt commented Jan 9, 2025

Uh oh!

chia7712 commented Jan 9, 2025

Uh oh!

frankvicky commented Jan 10, 2025

Uh oh!

chia7712 commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

jolshan commented Dec 20, 2024 •

edited

Loading

jolshan commented Dec 20, 2024 •

edited

Loading

trnguyencflt commented Jan 8, 2025 •

edited

Loading