KafkaItemReader reset offset to zero for new job instance

[gmadalon]

OBSERVED BEHAVIOR

KafkaItemReader always reset offset to zero even when the last job execution was finished OK.
So, if my batch runs every night to process a Kafka topic it doesn't work because it will read all messages every night not just the messages not processed.

ROOT CAUSE

KafkaItemReader line 248:

	for (TopicPartition topicPartition : this.topicPartitions) {
		this.partitionOffsets.put(topicPartition, 0L);
	}

PROPOSED SOLUTIONS

Remove these lines and don't set offset to 0 anymore.

[alexisgra]

Looks like someone fixed the problem in this PR : #737 , the fix should be embedded in version 4.3.0.

[gmadalon]

Hi @alexisgra,
You are right, thank you for your comment.
I'll close this issue.

[Vickd811]

Hi, I am still observing this behaviour. I have 4.3.3 version of Spring-Batch, I am running it with Spring boot parent 2.5.0.
With auto.offset.reset set as "earliest" when i restart my server, it goes back to offset 0. if i keep value to "latest", it does not read anything until messages are published while batch is active. I am using MapJobRepository with spring-batch, no database for storing spring execution context. Sorry, to comment on a closed thread, i am totally stuck on it.

[fmbenhassine]

@Vickd811 please check this answer on StackOverflow: https://stackoverflow.com/a/65882465/5019386. If you still need support, please add a comment on that SO thread or ask a new question on SO. Thank you.

[Vickd811]

Thanks @benas , it resolved the issue. The other behavior i am observing is re-process of last processed offset for a fixed delay scheduled job. So, i am using kafka reader for one of my job where i need to submit the job after every 10 seconds of previous job completion. It is scheduled with a fixed delay, so once first instance of the job is completed, scheduler submits another instance after 10 seconds, the second instance is still picking up the last offset of previous job run. Would you know how to circumvent this? It doesn’t happen when I completely kill the process and restart.

[fmbenhassine]

@Vickd811 Glad it helped. For the other question, please ask it on StackOverflow. We would like to keep the issue tracker exclusively for bug reports and feature requests, see https://github.com/spring-projects/spring-batch/blob/main/SUPPORT.md. Thank you.