Kubernetes Ingress Controller Wishlist

[dpbevin]

What should we add or change to make your life better?

I've captured some features that would be good for improving the Kubernetes Ingress controller code. I'm more than happy to contribute heavily in this space, as I had a previous Ingress Controller project based on Yarp with some of these items.

• Same ingress or service name in multiple Kubernetes namespaces #1389 Support ingresses of same name in multiple namespaces. Reasonably simple - cluster and route IDs don't include namespace, so clashes are possible.
• Single deployable. Combine the "sample ingress" project and the "controller" project into a single deployable, making it easy to set up and manage.
• Path rewrite via ingress annotations
• Default (fallback) TLS support for all ingresses, utilizing a Kube secret.
• SNI-based TLS support on a per-ingress basis. Again, based on Kube approach. (work in progress May 2022)
• Kubernetes Ingress Class Support #1397 Follow newer IngressClass specification. Allows multiple ingress controllers to be present in the same cluster. Not just other ingress technologies but potentially multiple yarp-based controllers for different usages (e.g. public vs internal ingresses).
• Report "scheduled for sync", etc. events to ingress resources to identify that Yarp controller is managing specific ingresses. Not really well documented from a Kube point of view but something that other ingress controller do.
• Docker image and Helm Chart
• Session affinity via ingress annotations
• OAuth enforcement for backend services that don't have their own auth. Similar to https://github.com/oauth2-proxy/oauth2-proxy.
• GRPC-routing
• OpenTracing support. Great for diagnosing issues
• Request size limiting via ingress annotations
• Header-based canary routing via ingress annotations
• Rate limiting via ingress annotations

Why?

To make this project a great option for Kubernetes ingress!

[samsp-msft]

We welcome contributions in this space as k8s is not our area of expertise, and we are heads down on trying to finish 1.0 for YARP.

I have questions on a couple of features:

• Single deployable. Combine the "sample ingress" project and the "controller" project into a single deployable, making it easy to set up and manage

We have found with the Service Fabric equivalent, that when you have a number of proxy instances querying the backend for changes, that can create quite a bit of load, which is why we are thinking that a separate instance for the controller makes more sense.

• Docker Image

This is on the roadmap, but as we have found that YARP's differentiator is its customizability, this has taken a back seat as that implies it will be a static binary, rather than something built for each scenario. Are you not wanting or needing any customization of the proxy logic and everything can be driven by config/annotations?

• GRPC-routing

What are you looking for specifically here? YARP can proxy GRPC requests already.

• OpenTracing support

I think this is already available when using .NET 6. @shirhatti can you confirm?

• Header-based canary routing via ingress annotations

What exactly are you looking for here?

• Rate limiting

We don't have rate limiting support directly in ASP.NET core, but you can use middleware such as AspNetCoreRateLimit, which will apply to inbound requests through YARP. This is an example of the kind of customization that is easy to implement because of the modular nature of YARP.

[shirhatti]

I think this is already available when using .NET 6. @shirhatti can you confirm?

We are already instrumented for OpenTelemetry (not OpenTracing*) support. The OpenTelemetry project already has support for Jaeger, Zipkin, and OTLP exporters (among others).

* OpenTracing is being retired in favor of OpenTelemetry

We don't have rate limiting support directly in ASP.NET core, but you can use middleware such as AspNetCoreRateLimit, which will apply to inbound requests through YARP. This is an example of the kind of customization that is easy to implement because of the modular nature of YARP.

This is something that's being worked on in ASP.NET in .NET 7. We should be shipping preview packages for rate limiting soon. See aspnet/AspLabs#384.
👀 @BrennanConroy

[macsux]

Few things that I would like to see added to the list

• Custom CRDs as an alternative to using standard Ingress resource. Ingress resource is very limiting and extending it view metadata annotations will get messy very quick for more advanced routing setups.
• Ability to combine multiple configuration sources instead of taking everything from the dispatcher. For example, I want to have routes managed via spring config server backed by git repository, but I want clusters to be dynamically built up by observing Kubernetes infrastructure. Potentially refactor the Receiver as an IConfigurationProvider so it can be overlayed with other config sources. This would have the added benefit for monitoring current state of YARP configuration via something like /env endpoint in Steeltoe Actuators

[dpbevin]

Hi @samsp-msft. Lots of great follow up questions there. Most can be answered in a similar fashion....

I rely heavily on a pre-built nginx Ingress controller (single deployable) in my K8S setup and I suspect I'm not alone 😜.

So, step 1. I'd like a drop-in replacement for nginx. I'm lazy, I don't want to have to compile something for every situation.

Nginx isn't without its flaws. Ive run into a problem with its support for GRPC streaming just this week (related to client body size limits).

Nginx isn't without its limitations too. Recently, I needed to do auth conversion - client certs on the front side, OIDC client credentials on the backend. Let's just say, plenty of frustration with Lua, Perl and JavaScript. I'm a C# nut at heart, so for those occasional "I need extensibility" situations, Yarp is a good fit. But these are the exceptions, not the rule.

Back to the single deployable...

Loading certs (from secrets) becomes harder with separate deployables because of the cluster role bindings. The actual ingress needs sni plus a way of reading the secrets, so the logic to do this would be split between the two deployables.

I've not seen high load with nginx personally, though I know there are some lurking, potential problems. K8S has a new feature called "EndpointSlices" that would likely improve things though. I don't see any of these issues being a deal-breaker for the majority of situations though.

The rich middleware of ASP.NET Core, combined with its blistering performance, means that a yarp-based ingress controller can easily be superior to many of the ingress controllers available today for K8S.

I'm sure I've missed a couple of items but I think this comment is long enough already 🤣.

Happy to talk it through more.

[GreenShadeZhang]

Hi @samsp-msft. Lots of great follow up questions there. Most can be answered in a similar fashion....

I rely heavily on a pre-built nginx Ingress controller (single deployable) in my K8S setup and I suspect I'm not alone 😜.

So, step 1. I'd like a drop-in replacement for nginx. I'm lazy, I don't want to have to compile something for every situation.

Nginx isn't without its flaws. Ive run into a problem with its support for GRPC streaming just this week (related to client body size limits).

Nginx isn't without its limitations too. Recently, I needed to do auth conversion - client certs on the front side, OIDC client credentials on the backend. Let's just say, plenty of frustration with Lua, Perl and JavaScript. I'm a C# nut at heart, so for those occasional "I need extensibility" situations, Yarp is a good fit. But these are the exceptions, not the rule.

Back to the single deployable...

Loading certs (from secrets) becomes harder with separate deployables because of the cluster role bindings. The actual ingress needs sni plus a way of reading the secrets, so the logic to do this would be split between the two deployables.

I've not seen high load with nginx personally, though I know there are some lurking, potential problems. K8S has a new feature called "EndpointSlices" that would likely improve things though. I don't see any of these issues being a deal-breaker for the majority of situations though.

The rich middleware of ASP.NET Core, combined with its blistering performance, means that a yarp-based ingress controller can easily be superior to many of the ingress controllers available today for K8S.

I'm sure I've missed a couple of items but I think this comment is long enough already 🤣.

Happy to talk it through more.

Recently, I am also looking for projects such as api gateways. There are functions provided by the mainstream istio envoy service mesh and nginx. Their extensions are basically based on lua or c++, and of course they also use webassembly. Many of them integrate well with k8s, such as ingress nginx or apisix. I think we really need a k8s ingress based on .net. This project just returned is also used as a reverse proxy, so I think your idea is really good, and I look forward to your continued sharing of development ideas for this feature.

[adityamandaleeka]

Should this also go with this list? #1145

We'll need to understand the scope of that work as well.