Node local routing to upstream pods

[brianehlert]

Adding an option to first route to node local upstream pods as a way to decrease cross zone routing costs when operating within a highly available cluster environment within a public cloud.
While it is possible to stretch a cluster across availability zones in multiple public clouds, this increases availability of the cluster and services it does raise networking costs as traffic crossing back and forth across Availability Zones can come with an increased price or premium.

As a way to control cross AZ costs, keeping traffic local to a Node or local to an AZ is one possible solution.
Both of these would be implemented in different ways.

This would override NGINX native load balancing across upstreams. For example, "least time last byte" which is commonly used as a way to optimize the response time would no longer be an option, as the priority for distributing load is shifted to minimize networking cost rather than optimizing application performance.

Node Local could be achieved in combination with a DaemonSet. With the load balancer layer providing the disaggregation.
It is believed that Node Local with a Deployment would be of little value since the NIC pods themselves are not balanced across the cluster and thus will overload some nodes and underload others.

Within a single AZ would begin to become more complex, asking the question if we then have different upstream configurations across the NIC pods themselves. Establishing a preferred and neighbor nodes, based on local or foreign Availability Zone.