Enable the ability to set readOnlyRootFilesystem: false

[calumreesmmt]

As part of a security policy audit, we've noticed that you are unable to set the root file system of the ingress nginx containers to ReadOnly. Here's some links for further context:
#6256
https://github.com/mspnp/aks-baseline-regulated/issues/10

One user has mentioned .ASP Net containers needing access to the file system and to overcome this it's possible to create a separate volume mount for the application to use, instead of it using the root file system.

Is it possible to implement the ability to have a read only file system?

[k8s-ci-robot]

@calumreesmmt: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

Please look for the chrooted deployment of the controller in the docs and the values file

[calumreesmmt]

Hi @longwuyuan I've deployed the chrooted version but it failed due to not having the permissions to do so.

When setting the read-only file system to true, this is the error we get:

unexpected error storing fake SSL Cert: could not create PEM certificate file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem: read-only file system

What is this cert? Is it the --default-ssl-certificate?

Is there a way to stop Nginx from generating this cert?

[longwuyuan]

Yes, its the default-ssl-certificate.
The use-case of chrooting and the use-case of ReadOnly are not the same.
Chrooted is a optional feature.
ReadOnly is not even tested or supported.
At least the /tmp gets used during config changes. I am almost certain that ReadOnly root filesystem is not possible to be supported or available as a feature.

[daniel-almeida]

@calumreesmmt , I've been trying to do the same thing. You can get around that particular error by mounting some emptyDir volumes. You'd need to map out all writable volumes (that I know of, /tmp, /etc/nginx, /nginx/ingress-controller, /var/cache/nginx, and /var/lib/nginx).

Even then, I'm seeing an issue with the UDS used for exporting prometheus metrics from this line:

Error creating prometheus collector:  listen unix /tmp/nginx/prometheus-nginx.socket: bind: no such file or directory

[calumreesmmt]

Thanks @daniel-almeida, this was my concern with implementing a workaround, you could be there a while trying to figure out an alternative method, but end up with something fragile and hard to maintain.

@longwuyuan Is it possible to raise this as a feature request?

[longwuyuan]

ReadOnly filesystem can not be supported. This is a Kubernetes Controller and by definition there is write needed for state change.

The chrooted controller was added after much thought.

But I am not making decisions. I am not even a developer. You can join the biweekly community meeting and have a discussion on the concerns around having a writable filesystem. It will likely produce more info. Link to meeting details here https://github.com/kubernetes/community/tree/master/sig-network

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[strongjz]

All the files that nginx manages would need to be moved to a volume mount
it writes the PID and other data once it starts up. The nginx config, generated by the controller, would need to be mounted in a volume config. So that requires coordination from the controller and nginx, especially after the DP/CP split. Ricardo was looking to send it via GRPC between them. There's also the log files https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/Dockerfile#L82 and
all of the other dirs nginx writes too https://github.com/kubernetes/ingress-nginx/blob/main/images/nginx/rootfs/Dockerfile#L58