Clarification on headless and centralized mode

[clintkitson]

I believe there is room for clarification in the spec to ensure CO implementations align regarding the operational modes.

In the case of a centralized deployment, it is clear that a single controller would run satisfying the controller publish calls.

In a headless operation there are multiple ways of implementing that may cause incompatibility for plugins/COs. At what place should a headless plugin implement the logic for attaching volumes? Would this be as the Controller whereas the controller is deployed to every node? Or is this Controller a noop with the logic residing in the Node where a publish both attaches and mounts?

For consistency I think no matter whether its headless/centralized, plugins should be implemented consistently where methods should not be overloaded based on operational mode. For headless this would mean every node would include Controllers+Node+Identity and attach operations would be invoked from the Controller.

[saad-ali]

For headless this would mean every node would include Controllers+Node+Identity and attach operations would be invoked from the Controller.

The spec leaves it purposely unspecified to allow vendors to decide. In Kubernetes we suggest running headless -- one option, like you said would be controller+node+identity to co-exist and deployed on all nodes, but for that to happen you need to manage service discovery between control plan and the multiple controller services or a leader election protocol. In Kubernetes we instead suggested deploying controller as a single instance independently of node service to avoid all that.

[clintkitson]

@saad-ali Why would leader election be needed? In the case of a single controller, what is its purpose? Maybe we aren't aligned on the use case.

I would envision a headless deployment being used based on the storage platform. For example, Ceph does not allow centralized attaches. In this case the attach logic always needs to happen on the node.

For vSphere a node attach request for a node on the node to which you are trying to fulfill the request. In this case logic for attaching would need to be placed on every host.

[akutz]

Hi @saad-ali,

@clintkitson, @codenrhoden, and I discussed this subject yesterday afternoon and arrived at a new understanding of the suggested CSI deployment architectures.

Up until this point, when myself, @clintkitson, or @codenrhoden would consider the CSI deployment models (linked above), our discussions were SP-centric -- which makes sense. That's when we realized that you, @jdef, @jieyu, @cpuguy83, et. al have a, not unsurprisingly, CO-centric view of CSI's supported architectures.

So why does this matter? Well, it matters because it greatly changes the way a person may perceive the roles of a CO and SP when considering the example models. For example, when I look at the deployment models here's what I see:

SP-Centric View

A specification that indicates the SP deployment models that a CO SHALL support.

However, yesterday, during my discussion with my colleagues, it hit me like a bag of rocks that what you and others with a CO-centric view see is in fact:

CO-Centric View

A specification that indicates the CO deployment models that an SP SHALL support.

This disjointed view of such a central piece of CSI has massive ramifications. For example, using the SP-Centric View, an SP that does not support a centralized controller would result in a CSI support matrix like so:

Deployment Model	Kubernetes	Docker	Mesosphere
Centralized Controller	✓	?	?
Headless Split		?	?
Headless Unified		?	?

Essentially, if an SP does not support a centralized controller then K8s only supports the first deployment model. I do not think this is the intention. I think the intention is in line with the CO-centric view that an SP should support all deployment models, but the specification's language is ambiguous at best:

It SHOULD be possible to ship cross-CO compatible Plugins for a variety of deployment architectures. A CO should be equipped to handle both centralized and headless plugins, as well as split-component and unified plugins.

Please note that the SHOULD refers to shipping SPs, not whether an SP supports a model. That's an odd phrase, right? Also, the line about what a CO should support does not capitalize should, leaving it up to the reader to decide if should is SHOULD (and not SHALL).

It's my opinion that the specification needs immediate improvements to the documentation before a v0.1 can be tagged. Otherwise the current state is simply too ambiguous. This is also why discussions about CSI should try to involve both people with CO and SP points-of-view, as it's clear that it's far too easy for a homogenous group to either misunderstand something or not understand how something could be misunderstood.

Addendum

By the way, this is the very subject that @codenrhoden tried to raise in #86. The issue was closed without the suggested change to include a SupportedModes flag that an SP could advertise, and I'd like to reiterate the value of such a flag. Without it K8s could not reject an SP that does not support K8s model -- which as it stands today is only the centralized model. K8s does not support the CSI headless models since there are multiple controllers.

[jdef]

Ala the spec:

The primary focus of this specification is on the protocol between a CO and a Plugin. It SHOULD be possible to ship cross-CO compatible Plugins for a variety of deployment architectures. A CO should be equipped to handle both centralized and headless plugins, as well as split-component and unified plugins. Several of these possibilities are illustrated in the following figures.

There is no SHALL anywhere in that statement. When we wrote this, the intent was to recommend a standard set of deployment options for plugins. Not all plugins would be required to support all deployment options. Not all COs would be required to support all deployment options.

IMO the view is "A specification that recommends SP deployment models, some of which a CO MAY support. A SP MAY support any number of deployment options."

There were several attempts to address deployment discovery through packaging and/or configuration. It was decided that the spec should primarily focus on protocol and I think the only breach we've made there is the requirement for CSI_ENDPOINT. Everything else that's related to packaging is a recommendation.

[jdef]

A CO should be equipped to handle both centralized and headless plugins...

should probably read

A CO SHOULD be equipped to handle both centralized and headless plugins

[akutz]

Hi @jdef,

I suppose the large issue at the moment is that our team is the only one producing SPs and K8s 1.9 is the only CO with CSI support. At least one of our SPs does not (natively) support centralized mode, but that's all K8s does support.

Everything else that's related to packaging is a recommendation

That is a very broad definition of packaging, but I understand the intent.

I concur with your proposed documentation change.

[saad-ali]

That's a good point @akutz. It sounds like spec should be clearer that "centralized" or "headless" is from the CO perspective.

That said, regarding @clintkitson's original suggestion:

For consistency I think no matter whether its headless/centralized, plugins should be implemented consistently where methods should not be overloaded based on operational mode. For headless this would mean every node would include Controllers+Node+Identity and attach operations would be invoked from the Controller.

We must retain the flexibility in the CSI spec to allow compatible plugins to deploy either controller only or node only. The plugin should be able to deploy however it wants. Imagine the case where controller operations require special credentials, in that case an SP may want to limit the deployment of the controller (with appropriate credentials) to a single instance of the controller on a cluster, but deploy the node component on every machine. This doesn't imply that the SP has to ship two binaries, they could achieve this with a single binary that accepts a flag or env var that sets the expected mode. Other volume plugins may only want to ship a single binary that can support both node and controller and deploy that to every machine in the cluster.

Based on my reading of https://github.com/container-storage-interface/spec/blob/master/spec.md#architecture and the current Kubernetes implementation, I believe this is possible.

[saad-ali]

In addition to:

* It sounds like spec should be clearer that "centralized" or "headless" is from the CO perspective.

We may want to modify CSI spec to allow for colocated controller/node plugins for Local Volume provisioning cases.