Kafka Consumer Event Loop #58
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To be able to properly stream Kafka you need to have an event loop that properly facilitates streaming Kafka.
There are several reasons why this is needed:
poll
eventspoll
continuously, or pause partitions to guarantee proper back-pressure without expensive rebalancing/repartitioningAlso see https://tuleism.github.io/blog/2021/parallel-backpressured-kafka-consumer/.
This PR adds a custom event loop for facilitating this based on reactor-kafka.
Some work still needs to be done to support
EXACTLY_ONE
andAT_MOST_ONCE
delivery.Given this work makes everything in the
Consumer.kt
file obsolete I decided to move it to a.receiver
package, following similar naming asreactor-kafka
but this is prone to change/improvement towards 1.0.I tried to add as much documentation inside of the code as possible. All feedback, and suggestions, code reviews or questions are welcome! 🙏