Advice to make a custom worker Tree-capable

[mattaezell]

I have a custom Worker class that inherits from DistantWorker. It works great with topology disabled, but when topology is enabled, it no longer works. It seems that the distant_worker is being called on each gateway with the command set to ClusterShell.Gateway -Bu. My worker doesn't exec the command, but it seems like that's what is expected here? It also appears that my worker is seeing some of the XML handshake for tree mode, and any output sent upstream isn't being properly wrapped in XML.

I know this is likely very cryptic without seeing my custom code, and I expect this is a use case that hasn't seen much interest. Any insights or pointers you could provide would be very much appreciated.

[degremont]

Indeed, this is a bit cryptic but I will do my best :)

Indeed, each Gateway is managed by a ClusterShell process called with python -mClusterShell.Gateway -Bu as you probably already figured out. https://github.com/cea-hpc/clustershell/blob/master/lib/ClusterShell/Worker/Tree.py#L172

This process will read data and events from stdin and send requests on stdout. Each Worker should be able to execute any remote commands and read stdin and write to stdout. That's the requirements to support your own worker for Gateways.

What capacity is missing from your worker?

The Tree code is not the cleanest possible and it could be a bit tangled with other worker code. However its main design relies on a specific EventHandler and a fake DistantWorker which internally launches and manages real distant workers.

So far, I don't see a specific reason you could not achieve what you want except if your distant worker has specific limitations. Maybe @thiell could have a better knowledge than me.

Out of curiosity, are you working on a worker for an ORNL internal project or a public software like Slurm, Flux, ... ?

[mattaezell]

Between this response and your comment in #421, I think I see the issue now. I incorrectly assumed that the TreeWorker was handling the communication between the root and gateway, and then the distant worker was invoked only on the gateway. In actuality (I believe, please correct me if I'm wrong), the TreeWorker spawns the distant worker on the root, which then spawns the gateway code on the gateway, which then spawns the distant worker again with the user-provided command.

My custom worker does not handle arbitrary shell commands.

It seems like what I actually want is to be able to specify a distant worker for tree communication, and a different worker for the "last hop". That doesn't seem like a common use case, so it looks like I need to add support for at least the gateway command.

Right now, this is an internal project at ORNL. If it matures sufficient and proves to be useful to us, we will go through the process to open-source it.

[mattaezell]

It seems like what I actually want is to be able to specify a distant worker for tree communication, and a different worker for the "last hop". That doesn't seem like a common use case, so it looks like I need to add support for at least the gateway command.

This seems like exactly what distant_worker and local_worker were designed for. Currently, Tree.py hardcodes the ExecWorker if not remote. The "right way" to set a distant or local worker is through defaults, but defaults are not sent over the gateway propagation channel (as far as I can tell).

I have hardcoded Tree.py to use my custom worker instead of ExecWorker and things are working again. If you have any guidance on the "right way" to send the desired local_worker, I can open a PR.

Thanks for all your help!

[degremont]

I must admit that's the first time I'm seeing this requirements when you need SSH to connect from the root node to the gateways, but on the gateways, you use another tool to actually run the specific command.

• Local_worker aims to run commands locally, on the root node if there are no tree mode, or on the last gateways if there is a tree. This is useful to launch commands that do network communication itself, like ping or ipmipower by example (https://clustershell.readthedocs.io/en/v1.8.3/tools/clush.html#tree-mode-specific-options)

• distant_worker aims to run command on another node using remote execution command tool like ssh.

Task info is propagated to gateways automatically. task defaults is local to each one, and could be set in /etc/clustershell/defaults.conf locally on your gateways.

[degremont]

Is that what you are trying to achieve?
clush --worker=tree --remote=false --local_worker=Foo

[mattaezell]

The first set of commands I'm implementing is actually making REST calls to Redfish-enabled BMCs from either the root or a gateway, so local_worker actually matches what I'm trying to do.

Is that what you are trying to achieve?
clush --worker=tree --remote=false --local_worker=Foo

Yes, I think so, though not using clush (using a custom frontend). But I still want to be able to use clush as normal with either a SSH distant worker or Exec as a local worker (so changing the defaults isn't necessarily the best option).