KEP-5471 Extended Toleration Operators for Threshold-Based Placement

[helayoty]

• One-line PR description: Add numeric comparison operators (Lt, Gt) to Tolerations for SLA-based scheduling with threshold-based taint matching.

• Issue link: Extended Toleration Operators for Threshold-Based Placement #5471

• Other comments: cc @kubernetes/sig-scheduling-misc @kubernetes/sig-apps-misc

[SergeyKanzhelev]

can we include the scenario of adding tolerations on the semantic version comparison? It will likely require either a new operator or some way to express that a taint's string needs to be parsed as a semver.

[helayoty]

The current scope for this KEP is integer-only support for SLA/failure-probability because it’s the simplest and safest to implement (avoids floating-point parsing and complex type semantics).

I agree that semver comparisons are a valid and important future use case (e.g., kubelet version taints, device firmware versions). To keep this KEP narrowly scoped and implementable, I propose to document the semantic version comparison in a new Future Work section. And also consider if we would add this to the node affinity as well. wdyt?

[SergeyKanzhelev]

One consideration here is what is the delta here. It is a lot of red tape to run thru scenarios and API reviews. Making two changes together may be much easier than splitting into separate KEPs.

I am not objecting to this scope though. This is not blocking proposal, but I believe it will be very useful.

BTW, we may also want to support semver in CEL as well =). So potentially another scope increase

[helayoty]

Updated the scope to include both semver and CEL.

[helayoty]

@sanposhiho fyi for semver addition.

[helayoty]

fyi, after discussing it with @sanposhiho @macsko, we will have a separate KEP for how the semver can be implemented onto nodeaffinity, taint/toleration, and CEL, along with evaluating use case for each.

Created issue to track this feature ->#5500

[jackfrancis]

Is it fair to add a goal to ensure that the addition of new operators has zero operational effect on existing pod scheduling outcomes using the Equality or Exists operators? I'm thinking of the various switch toleration.Operator bits in source and assuming that adding more operators should have zero cost to existing scenarios.

[helayoty]

Yes, totally fair. Added a goal in addition to performance tests.

[jackfrancis]

Something to consider: for every scenario that is (strictly speaking) possible using existing semantics (e.g., NodeAffinity) it might be helpful to include that scenario here. A before/after type picture. Not wanting to add too much information to the KEP, but that would help folks to assess the ergonomic improvements.

[helayoty]

Differed to the separate KEP we will create for semver/CEL

[jackfrancis]

Maybe we can also inclue an example pod spec that will explicitly not get scheduled onto a spot node, e.g.,

---
# Critical workload will not be scheduled until a suitable high reliability node has capacity
apiVersion: v1
kind: Pod
metadata:
  name: critical-workload
spec:
  tolerations:
  - key: node.kubernetes.io/sla
    operator: Gt
    value: "950"
    effect: NoSchedule

[helayoty]

Added the example.

[jackfrancis]

Nit: Let's call this a "Short-lived batch training workload" to distinguish from other (very normal) batch workloads that would not want to be interrupted in an hour.

[helayoty]

Addressed.

[jackfrancis]

Nit: to make this more current, let's add 1.33/1.34 as our two example versions here

[helayoty]

Differed to the future KEP for semver/CEL

[jackfrancis]

SemverGe for this and below in this example?

[helayoty]

Differed to the future KEP for semver/CEL

[jackfrancis]

Nit: the pre-release and build metadata points are part of semver, so we could drop them. Or if being super explicit is helpful, let's add the "per semver specification" suffix to the Pre-release versions line item as well.

[helayoty]

Differed to the future KEP for semver/CEL

[jackfrancis]

This implies I could have two nodes on my cluster like this:

apiVersion: v1
kind: Node
metadata:
  name: node-1
spec:
  taints:
  - key: node.kubernetes.io/foo
    value: "100"
    effect: NoExecute
---
apiVersion: v1
kind: Node
metadata:
  name: node-2
spec:
  taints:
  - key: node.kubernetes.io/foo
    value: "v1.34.0"
    effect: NoExecute

And that the scheduler would simply schedule workloads with proper tolerations using either of the above semantics (int comparison or semver comparison) to the right node.

Also, apiserver has no preference between either of the below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  template:
    spec:
      tolerations:
      - key: node.kubernetes.io/sla
        operator: Gt
        value: "950"
        effect: NoExecute
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  template:
    spec:
      tolerations:
      - key: node.kubernetes.io/sla
        operator: Gt
        value: "v1.33.0"
        effect: NoExecute

During scheduling, if we encounter a matching node taint whose value is of a different type than the workload toleration (int vs semver) then we simply continue and rule out that node as a candidate for scheduling.

Just thinking out loud here, did I get that right?

[helayoty]

Differed to the future KEP for semver/CEL

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: helayoty
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric, sanposhiho for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[macsko]

Isn't returning just false here too simple? How would the user know that the node was rejected because of non-numeric taint?

[helayoty]

We shouldn't add logging or return errors here. I'd keep it simple return false for performance reason (this code path is hot and will be called for every pod-node combination during scheduling) and to be consistenent with the current ToleratesTaint method (returns false silently).

Invalid numeric values should be caught by API validation when pods/nodes are created, not during runtime scheduling. Users will still get feedback through API validation errors and scheduler events.

[macsko]

But we can't catch the invalid numeric values for the taints earlier, because node.kubernetes.io/sla=95.5 is a correct taint - it can be matched by the Equal operator. If the parsing during scheduling fails, we have 2 options:

• Node taint is misconfigured, because it uses non-numeric taint, when expected to use numeric one (e.g. 95.5 value instead of 955).
• Pod toleration is misconfigured, because it wants to use numeric operator on non-numeric taint.

I think for both cases we would like to inform user what's wrong, not only that the toleration is not matching for any node.

[macsko]

How would you implement those metrics? Pod could fail due to multiple reasons, depending on the node.

[helayoty]

Thanks for pointing this issue. After some thoughts the scheduler_numeric_taint_mismatches_total metric is problematic because pods can fail scheduling due to multiple plugins simultaneously. Also, the same pod might be rejected by different nodes for different reasons. Metrics updated.

[macsko]

This would need to be well documented in the API to make sure the users won't misunderstand