blocking consumer

[pmackles]

Should SimpleConsumer block when there are no messages to be read? Based on the examples, I thought that would be the case but when I run them, they seem to exit as soon as all of the messages have been read.

[mumrah]

@pmackles indeed, the consumer will terminate when there are no more messages to read. The current consumer implementation is a very simplistic, single-threaded approach (hence the name SimpleConsumer). There are plans to create a queue-based consumer which would give you blocking behavior.

Is there an particular example that's unclear in the README?

[mahendra]

@pmackles @mumrah - as part of some ongoing multiprocess consumer, I have implemented a blocking API for SimpleConsumer without making use of Queues or Events. This has been done by using FetchRequest parameters.

https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html#AGuideToTheKafkaProtocol-FetchRequest

The code is available on https://github.com/mahendra/kafka-python/compare/partition

[pmackles]

Perfect. Thanks for the heads up. I think blocking is a new feature in FetchRequest for 0.8 so the way you implemented makes perfect sense to me.

[mahendra]

@pmackles cool

[mahendra]

Closing this, since this has been implemented in #35 and merged.