BrokerConnection slow initialization in version >= 1.4.0 when using broker-version auto-detection

[romulorosa]

Hello everyone,

I've updated my kafka-python from 1.3.5 to 1.4.0 and could notice that a producer initialization in 1.4.0 is slower than in 1.3.5.

Using 1.3.5

In [1]: from kafka.producer.kafka import KafkaProducer
   ...: import time
   ...: 
   ...: start = time.time()
   ...: p = KafkaProducer(bootstrap_servers='kafka:9092')
   ...: end = time.time()
   ...: print(end - start)
   ...: 
0.12534785270690918

Using 1.4.0

In [1]: from kafka.producer.kafka import KafkaProducer
   ...: import time
   ...: 
   ...: start = time.time()
   ...: p = KafkaProducer(bootstrap_servers='kafka:9092')
   ...: end = time.time()
   ...: print(end - start)
   ...: 
1.1414685249328613

Also I've performed some tests using python profile and the result shows that poll method is taking more time to run.

Using 1.3.5

Using 1.4.0

Does anyone know the reason of this overhead?

Thanks.

[romulorosa]

Hi guys,

I've found the reason of the ovehead. Basically it was introduced by calling selector.select(1) in kafka.conn.BrokerConnection#check_version (https://github.com/dpkp/kafka-python/blob/master/kafka/conn.py#L954).
I don't have enough context to know preciselly why we have this call always, eventhough the first attempt to get the responses of api requests was sucessfully completed. So i've promoted some changes and i would like to know if it makes sense. Basically i check if the previous attempt it's OK before call selector.select(1).

https://github.com/dpkp/kafka-python/compare/master...romulorosa:chore-improve-producer-initialization?expand=1

@dpkp Please, can you take a look into these changes?

[dpkp]

Thanks for investigating. Would a simpler change be to call selector.select(1) first, before calling self.recv() ? If there are bytes available to read from the socket, then select(1) should return immediately and recv() will then read those bytes and handle processing. I think this was just an error when I added the selectors to BrokerConnection in 1.4.0. If you'd like to submit a PR to fix, I'm happy to merge.

[romulorosa]

@dpkp Nice catch! I've opened the PR #1475

[dpkp]

Thanks -- merged and fixed.