KAFKA-3949: Fix race condition between group rebalance and metadata update

[jeffwidman]

Details in https://issues.apache.org/jira/browse/KAFKA-3949

Note that the fix refactored a fair bit of code for managing subscription state: apache/kafka#1762

And then KIP-70 (tracked in #1242) further modified this code.

Related: #1237 / #1240 / #1242

[jeffwidman]

@dpkp, I saw you self-assigned this... are you actively working on it?

I'm once again seeing the symptoms of #1237 from time to time in the test suite for our internal wrapper around kafka-python at my day job. I assume this is the root cause, although I haven't had time to verify yet.

At my day job, it's a priority to get this fixed, as a particular service that uses pattern subscription has multiple consumer groups across many hosts so noticing when they're stalled due to zombie subscriptions is very difficult.

I've spent some time reading through the Java fix (which appears more straightforward than I initially thought), but haven't started porting it yet. Mostly I've been trying to figure out how to write a test that forces this race condition (#1251).

@dpkp I am certain you will be much faster than I will at porting the Java fix, so if you're interested in this, by all means please tackle it. But if you don't think you'll be able to get to it for several weeks, let me know as I'll be able to spend more time on it next week.

[dpkp]

Yes, I have a PR ready but it depends on #1266 so I have not pushed it yet.

[jeffwidman]

@dpkp now that #1266 is merged, do you mind pushing up your PR for this?

I've got another team that is hitting consumers silently failing to consume every once in a while, and I suspect it's due to this...

[dpkp]

Yes -- unfortunately it has got some complex merge conflicts that need working out. Hoping to get these resolved + tested locally this week.