"node affinity conflict" following Deployment instructions in README

[michael-db]

With a microk8s v1.16.2 cluster on Fedora, I followed the instructions in the README, replacing 1.13 in the README with 1.16, i.e.,

./deploy/kubernetes-1.16/deploy-hostpath.sh

One problem encountered was that csi-hostpathplugin-0 stuck in ContainerCreating

    Warning  FailedMount  108s (x9 over 3m56s)  kubelet, localhost.localdomain  MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/lib/kubelet/plugins_registry is not a directory

which I got past by

sudo mkdir /var/lib/kubelet/plugins_registry

Maybe csi-hostpath-plugin.yaml should use type DirectoryOrCreate for that path?

However, the problem that I haven't worked around is that the consumer pod doesn't schedule:

kubectl describe pod my-csi-app
 ...
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) had volume node affinity conflict.

kubectl describe pv <ID>
...
Node Affinity:
  Required Terms:
    Term 0:        topology.hostpath.csi/node in [localhost.localdomain]
...

I haven't yet learned enough to resolve this (if it's my error) or suggest a correction (if the documentation or something else is wrong). I see that https://kubernetes-csi.github.io/docs/topology.html says that a CSINode object should be created, but

kubectl get csinode
No resources found.

According to the same source, https://kubernetes-csi.github.io/docs/csi-node-object.html node-driver-registrar is responsible for creating that object, but evidently it doesn't, even though there are no errors in the log. I've found no evidence in grepping the node-driver-registrar source code that it does or should.

[msau42]

The 1.16 deployment is not meant to work with 1.13 because it uses newer Kubernetes features. Try using a 1.13 deployment spec from an older release

[michael-db]

@msau42 Why do you think I'm using 1.13, Michelle? It's 1.16

: csi-driver-host-path 227;  kcf version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
: csi-driver-host-path 228;  alias kcf
alias kcf='KUBECONFIG=~/fedora1.kubeconfig kubectl'

[msau42]

Oh sorry, I misread your original comment about replacing 1.13. Can you check your kubelet logs for hints to driver registration? It is kubelet that is supposed to create the CSINode object. And make sure that the CSINodeInfo feature gate is enabled on your cluster (all components, apiserver, scheduler, kubelet, kube-controller-manager, etc)

[michael-db]

Thank you for clarifying the answer to "What creates the CSINode object?".
At https://kubernetes-csi.github.io/docs/csi-node-object.html I read
"In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default."
so I understood that the --feature-gate was unnecessary for 1.16.
However, I added it anyway to the files

/var/snap/microk8s/current/args/kube-apiserver
/var/snap/microk8s/current/args/kube-controller-manager
/var/snap/microk8s/current/args/kubelet
/var/snap/microk8s/current/args/kube-proxy
/var/snap/microk8s/current/args/kube-scheduler

and restarted microk8s.
I verified with 'ps aux' that it had been passed to the kube* commands, but the problem was unchanged.

I did

journalctl |grep kubelet|tail

and saw nothing added to it when I repeated the

kubectl apply -f ./examples/csi-app.yaml

(The lines at the end are "Image garbage collection failed" messages - the disk is at 92%.)

I also found the gocsi tool to see what that would say

: args 117;  sudo ~/go/src/gocsi/csc/csc node get-info --endpoint /var/lib/kubelet/plugins/csi-hostpath/csi.sock
localhost.localdomain	0	&csi.Topology{Segments:map[string]string{"topology.hostpath.csi/node":"localhost.localdomain"}, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}

but kubectl get csinode still outputs No resources found.

Thanks for responding. I'm still assuming the absence of the CSINode is the cause of the problem. I will get back to this tomorrow and try again to see if there is something I've missed.

[msau42]

The driver registration should happen as part of ./deploy-hostpath.sh.

Can you check that "/var/lib/kubelet" is configured as the kubelet's root in microk8s? If not, then the hostpath deployment spec should replace all instances of "/var/lib/kubelet" with whatever your kubelet root is.

[msau42]

The CSINodeInfo feature gate should be enabled by default starting in 1.14.

[michael-db]

That was inspired, Michelle.

pgrep -a kubelet|sed -n 's@.*\(--root-dir=[^ ]*\).*@\1@p' # output:
--root-dir=/var/snap/microk8s/common/var/lib/kubelet

I decided the simplest way to resolve that was to do

cd /var/lib
sudo mv kubelet/ kubelet.org
sudo ln -s /var/snap/microk8s/common/var/lib/kubelet/

And all was well.

I've closed a pull request because I don't have time for the "new contibutor"/corporate process now. You can amend the documentation or not as you wish.

Update: If might renew the pull request next week if I get a chance (and this hasn't been closed already). Thanks again.

[michael-db]

Sorry, but it's already apparent that I won't get contributor approval through my current employer. I suggest you add a brief note to the README.md and this can be closed.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[michael-db]

Formally, this can be fixed by adding a short note to the README.

However, what would be much better would be for the kubelet root dir to be made available symbolically so that any yaml that needs to refer to it could do so indirectly and remain independent of the actual value. If that were done, there would be no dependency on the kubernetes config options or distribution - it would even be possible to deploy drivers like this where some nodes had kubelet running with a different root-dir to other nodes.

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale