Kubeadm should preflight check on init and join if /etc/resolv.conf is pointed at 127.0.0.x

[mauilion]

What keywords did you search in kubeadm issues before filing this one?

127.0.0.1 preflight

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Versions

kubeadm version (use kubeadm version): all

Environment:

• Kubernetes version (use kubectl version): all
• Cloud provider or hardware configuration: all
• OS (e.g. from /etc/os-release): Ubuntu and many other distros that use systemd-resolver

What happened?

When the underlying node is configured to use systemd stub resolver or really any local dns caching solution the nodes resolv.conf is usually pointed at 127.0.0.X

When this happens the kubelet by default still uses that resolv.conf as the basis for the resolv.conf that is generated for pods with a dnsPolicy: Default. This has consequences for kube-dns as the kube-dns pods are configured to use this method to "derive" the default upstream servers and search paths.
If coredns is configured to use 127.0.0.X as it's upstream resolver nothing will be resolved as we now have a resolution loop.

What you expected to happen?

I think that kubeadm preflight checks should check for this and fail if the local /etc/resolv.conf includes 127.0.0.X in the servers line. 127.0.0.X because depending on the implementation it can be 127.0.0.53 (systemd stub resolver) or 127.0.0.1 (dnsmasq by default)
Should the preflight check fail the user can provide a different resolv.conf to the kubelet with the --resolv.conf flag.

How to reproduce it (as minimally and precisely as possible)?

Bring up an ubuntu system with systemd stub resolver configured.

Anything else we need to know?

This is a problem that comes up fairly often and seems like something that kubeadm can check for and help avoid.

[timothysc]

/assign @detiber

[chrisohaver]

Largely redundant with kubernetes/kubernetes#64665, which automatically detects the presence of systemd-resolved and adjusts kubelet flags. Note that the PR also removed similar redundant pre-flight checks that were added earlier.

If we do want this check (for general case when systemd-resolved is not involved), then it needs to be aware of systemd-resolved so it doesn't falsely trigger.

[chrisohaver]

That said, if kubernetes/kubernetes#64665 is failing to detect the systemd-resolved case, then that is a bug.

[neolit123]

@mauilion
what are the contents of /var/lib/kubelet/kubeadm-flags.env does it point to?:
/run/systemd/resolve/resolv.conf
if yes, then kubeadm detects systemd-resolved correctly.

I think that kubeadm preflight checks should check for this and fail if the local /etc/resolv.conf includes 127.0.0.X in the servers line. 127.0.0.X because depending on the implementation it can be 127.0.0.53 (systemd stub resolver) or 127.0.0.1 (dnsmasq by default)
Should the preflight check fail the user can provide a different resolv.conf to the kubelet with the --resolv.conf flag.

a couple of things:

• like @chrisohaver mentioned kubeadm automatically falls to a different .conf file and not /etc/resolv.conf in the case of systemd-resolved
• the user cannot provide a custom file because we override the flag that kubeadm passes to the kubelet.

so we can technically find the correct .conf file and verify the contents, but it's a question if we want to do it.

Largely redundant with kubernetes/kubernetes#64665, which automatically detects the presence of systemd-resolved and adjusts kubelet flags. Note that the PR also removed similar redundant pre-flight checks that were added earlier.

@chrisohaver
can you please explain the redundancy of the check in terms of keeping this issue here tracked?

[chrisohaver]

explain the redundancy

Well - this is a feature request - and the feature already essentially exists. Sort of...

This may be new issue. I was looking into systemd-resolved, and according to a (slightly old) discussion, when you stop it (via systemctl), it can leave /etc/resolv.conf in a bad state. https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

kubernetes/kubernetes#64665 only detects when systemd-resolved has a pid (i.e. running)... so, if a user disables systemd-resolved without following up and fixing /etc/resolv.conf, kubeadm wont detect it.

A check directly of /etc/resolv.conf for local addresses, as @mauilion suggests, may catch more cases.
But if it added, the check needs to be aware of a running systemd-resolved, so it doesnt throw warnings/errors in that case where kubeadm will automatically correct kubelet.

[neolit123]

thanks for the explanation.
yes, if the 127.0.0.x check is added it has to be done over the file that the kubelet uses and not always over the /etc/resolv.conf file.

still waiting on @mauilion to indicate if systemd-resolved is detected as per the /var/lib/kubelet/kubeadm-flags.env file.

cc @kad for comments.

[chrisohaver]

Another thought about kubernetes/kubernetes#64665...

I think it only checks at init time. It also needs to check at node join time, in the event that some nodes have systemd-resolved enabled, but the master (the one where kubeadm init was executed), was not running systemd-resolved. I believe that kubelet passes resolv.conf from the local node it's running on to pods in "default" dns policy.

[neolit123]

@chrisohaver the "kubelet phase" in kubeadm applies to both init and join time.
an ENV file that holds the resolv.conf param for the kubelet should be written for both cases.