Processes report coordinator role erroneously

[ajbeamon]

I've seen cases where a process logs trace events with the coordinator role ("CD") even though it's not a coordinator. I don't yet know why this is happening.

[ajbeamon]

I'm not sure if this is representative of the cases I've seen, but I've been able to reproduce this by trying to connect to a cluster using a cluster file that specifies the wrong processes as coordinators. It doesn't seem to happen, though, if I use an old cluster file that forwards to the correct coordinators.

In the original examples it seemed this was happening to a process as it was being added to an existing cluster.

In my reproduction the sequence of events appears to be that we get an OpenDatabaseCoordRequest that runs monitorLeaderForProxies, which subsequently issues periodic GetLeaderRequests . These requests always fall back to the block that pushes the reply into the notify queue.

When nextInterval elapses, all of the conditions are met to terminate the leader nomination except for the fact that a request exists in the notify queue. We then send back an empty nominee to the original request and receive a new one to start the process over again.

This goes on indefinitely, even after the connected client disconnects.

[ajbeamon]

I think I figured out the common reason this is probably happening -- as part of the fdbcli kill command, we now send a GetLeaderRequest to every worker to establish a connection (#2512). This request goes to the coordinator's interface, and any message to that interface results in a process thinking it is a coordinator.

[jzhou77]

We saw frequent CD role flipping in a recent upgrade to 7.1.4, where a process can flip hundreds of times per minute.

[jzhou77]

When sending a kill command to reboot a worker, the client will try to establish a connection to the worker process using

foundationdb/fdbclient/NativeAPI.actor.cpp

Line 8277 in cb750b1

addInterfs.push_back(addInterfaceActor(&address_interface, connectLock, it));

, which calls

foundationdb/fdbclient/NativeAPI.actor.cpp

Line 8244 in cb750b1

wait(brokenPromiseToNever(leaderInterf.getLeader.getReply(GetLeaderRequest())))) {

to send a GetLeaderRequest to the worker. We saw over 1.5M coordinator being recruited events in a period of 22 minutes, due to the problem described in #2837 (comment)

Since GetLeaderRequest is quite spurious, how about we replace with a PingRequest that does minimal thing of reporting live?

[jzhou77]

There is a debugPing RequestStream in WorkerInterface. However, it was not in either ClientWorkerInterface or ClientLeaderRegInterface, and is not a well known endpoint. We could move debugPing to ClientWorkerInterface, but this breaks compatibility. Another idea is to overload reboot or profiler in ClientWorkerInterface to function as ping.

[sfc-gh-anoyes]

WLTOKEN_PROTOCOL_INFO might be a good candidate if we just want to make a connection. There's also a separate issue of fdbcli exiting before having flushed its outgoing buffer in flow transport when you do fdbcli --exec 'kill; kill all'. I think I'd prefer to see the reboot request just actually respond and then fdbcli can wait on those futures. That's a more reasonable to make sure we actually send requests IMO. We probably also want to wait on a timeout in case of unavailabity.

[sfc-gh-abeamon]

Yeah, this is the same request that caused most problems in the original report. We could use a different interface in kill to solve this particular case, but I'm not sure it universally solves the issue. Really what I think we want is a better way to identify when a process is serving as a coordinator and use that to manage the role lifetime. It shouldn't be based on the fact that we receive various request types.