gcp: Add nodepool for tests #7943

ameukam · 2025-03-26T19:01:12Z

Related to:

Change instance type for GKE build clusters #2438

Setup a new nodepool with taints so we can schedule specific tests on it for evaluation before we move all the test to a new nodepool. This nodepool will also use COS and cgroups v2.

ameukam · 2025-03-26T19:01:28Z

/hold
cc @BenTheElder @xmudrii @upodroid

k8s-infra-ci-robot · 2025-03-26T19:01:35Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be created
+ resource "google_container_node_pool" "node_pool" {
      + cluster                     = (sensitive value)
      + id                          = (known after apply)
      + initial_node_count          = 1
      + instance_group_urls         = (known after apply)
      + location                    = (sensitive value)
      + managed_instance_group_urls = (known after apply)
      + max_pods_per_node           = (known after apply)
      + name                        = (known after apply)
      + name_prefix                 = "pool6-"
      + node_count                  = (known after apply)
      + node_locations              = [
          + "us-central1-b",
          + "us-central1-c",
          + "us-central1-f",
        ]
      + operation                   = (known after apply)
      + project                     = "k8s-infra-prow-build"
      + version                     = (known after apply)

      + autoscaling {
          + location_policy = (known after apply)
          + max_node_count  = 80
          + min_node_count  = 1
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = true
        }

      + network_config (known after apply)

      + node_config {
          + disk_size_gb      = 100
          + disk_type         = "hyperdisk-balanced"
          + effective_taints  = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = "COS_CONTAINERD"
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + logging_variant   = (known after apply)
          + machine_type      = "c4-highmem-8"
          + metadata          = {
              + "disable-legacy-endpoints" = "true"
            }
          + min_cpu_platform  = (known after apply)
          + oauth_scopes      = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
          + preemptible       = false
          + service_account   = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com"
          + spot              = false

          + confidential_nodes (known after apply)

          + gcfs_config (known after apply)

          + kubelet_config (known after apply)

          + shielded_instance_config (known after apply)

          + taint {
              + effect = "NO_SCHEDULE"
              + key    = "dedicated"
              + value  = "sig-testing"
            }

          + workload_metadata_config {
              + mode = "GKE_METADATA"
            }
        }

      + upgrade_settings (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 0 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

infra/gcp/terraform/k8s-infra-prow-build/main.tf

k8s-infra-ci-robot · 2025-03-26T21:55:17Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be created
+ resource "google_container_node_pool" "node_pool" {
      + cluster                     = (sensitive value)
      + id                          = (known after apply)
      + initial_node_count          = 1
      + instance_group_urls         = (known after apply)
      + location                    = (sensitive value)
      + managed_instance_group_urls = (known after apply)
      + max_pods_per_node           = (known after apply)
      + name                        = (known after apply)
      + name_prefix                 = "pool6-"
      + node_count                  = (known after apply)
      + node_locations              = [
          + "us-central1-b",
          + "us-central1-c",
          + "us-central1-f",
        ]
      + operation                   = (known after apply)
      + project                     = "k8s-infra-prow-build"
      + version                     = (known after apply)

      + autoscaling {
          + location_policy = (known after apply)
          + max_node_count  = 80
          + min_node_count  = 1
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = true
        }

      + network_config (known after apply)

      + node_config {
          + disk_size_gb      = 500
          + disk_type         = "hyperdisk-balanced"
          + effective_taints  = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = "COS_CONTAINERD"
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + logging_variant   = (known after apply)
          + machine_type      = "c4-highmem-8"
          + metadata          = {
              + "disable-legacy-endpoints" = "true"
            }
          + min_cpu_platform  = (known after apply)
          + oauth_scopes      = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
          + preemptible       = false
          + service_account   = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com"
          + spot              = false

          + confidential_nodes (known after apply)

          + gcfs_config (known after apply)

          + kubelet_config (known after apply)

          + shielded_instance_config (known after apply)

          + taint {
              + effect = "NO_SCHEDULE"
              + key    = "dedicated"
              + value  = "sig-testing"
            }

          + workload_metadata_config {
              + mode = "GKE_METADATA"
            }
        }

      + upgrade_settings (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 0 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

k8s-infra-ci-robot · 2025-03-27T13:30:22Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be created
+ resource "google_container_node_pool" "node_pool" {
      + cluster                     = (sensitive value)
      + id                          = (known after apply)
      + initial_node_count          = 1
      + instance_group_urls         = (known after apply)
      + location                    = (sensitive value)
      + managed_instance_group_urls = (known after apply)
      + max_pods_per_node           = (known after apply)
      + name                        = (known after apply)
      + name_prefix                 = "sig-testing-pool6-"
      + node_count                  = (known after apply)
      + node_locations              = [
          + "us-central1-b",
          + "us-central1-c",
          + "us-central1-f",
        ]
      + operation                   = (known after apply)
      + project                     = "k8s-infra-prow-build"
      + version                     = (known after apply)

      + autoscaling {
          + location_policy = (known after apply)
          + max_node_count  = 80
          + min_node_count  = 1
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = true
        }

      + network_config (known after apply)

      + node_config {
          + disk_size_gb      = 500
          + disk_type         = "hyperdisk-balanced"
          + effective_taints  = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = "COS_CONTAINERD"
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + logging_variant   = (known after apply)
          + machine_type      = "c4-highmem-8"
          + metadata          = {
              + "disable-legacy-endpoints" = "true"
            }
          + min_cpu_platform  = (known after apply)
          + oauth_scopes      = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
          + preemptible       = false
          + service_account   = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com"
          + spot              = false

          + confidential_nodes (known after apply)

          + gcfs_config (known after apply)

          + kubelet_config (known after apply)

          + shielded_instance_config (known after apply)

          + taint {
              + effect = "NO_SCHEDULE"
              + key    = "dedicated"
              + value  = "sig-testing"
            }

          + workload_metadata_config {
              + mode = "GKE_METADATA"
            }
        }

      + upgrade_settings (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 0 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

ameukam · 2025-03-27T22:07:35Z

atlantis apply

k8s-infra-ci-robot · 2025-03-27T22:07:42Z

Ran Apply for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Apply Error

Show Output

running "/atlantis/bin/terraform1.11.3 apply -input=false \"/atlantis/repos/kubernetes/k8s.io/7943/default/infra/gcp/terraform/k8s-infra-prow-build/default.tfplan\"" in "/atlantis/repos/kubernetes/k8s.io/7943/default/infra/gcp/terraform/k8s-infra-prow-build": exit status 1
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Creating...
╷
│ Error: error creating NodePool: googleapi: Error 400: Node_pool.name must be less than 40 characters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.RequestInfo",
│     "requestId": "0x670e6335c60a1705"
│   }
│ ]
│ , badRequest
│ 
│   with module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool,
│   on ../modules/gke-nodepool/main.tf line 17, in resource "google_container_node_pool" "node_pool":
│   17: resource "google_container_node_pool" "node_pool" {
│ 
╵

Related to: - kubernetes#2438 Setup a new nodepool with taints so we can schedule specific tests on it for evaluation before we move all the test to a new nodepool. This nodepool will also use COS and cgroups v2. Signed-off-by: Arnaud Meukam <[email protected]>

k8s-infra-ci-robot · 2025-03-27T23:16:24Z

Ran Plan for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be created
+ resource "google_container_node_pool" "node_pool" {
      + cluster                     = (sensitive value)
      + id                          = (known after apply)
      + initial_node_count          = 1
      + instance_group_urls         = (known after apply)
      + location                    = (sensitive value)
      + managed_instance_group_urls = (known after apply)
      + max_pods_per_node           = (known after apply)
      + name                        = (known after apply)
      + name_prefix                 = "pool6-"
      + node_count                  = (known after apply)
      + node_locations              = [
          + "us-central1-b",
          + "us-central1-c",
          + "us-central1-f",
        ]
      + operation                   = (known after apply)
      + project                     = "k8s-infra-prow-build"
      + version                     = (known after apply)

      + autoscaling {
          + location_policy = (known after apply)
          + max_node_count  = 80
          + min_node_count  = 1
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = true
        }

      + network_config (known after apply)

      + node_config {
          + disk_size_gb      = 500
          + disk_type         = "hyperdisk-balanced"
          + effective_taints  = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = "COS_CONTAINERD"
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + logging_variant   = (known after apply)
          + machine_type      = "c4-highmem-8"
          + metadata          = {
              + "disable-legacy-endpoints" = "true"
            }
          + min_cpu_platform  = (known after apply)
          + oauth_scopes      = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
          + preemptible       = false
          + service_account   = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com"
          + spot              = false

          + confidential_nodes (known after apply)

          + gcfs_config (known after apply)

          + kubelet_config (known after apply)

          + shielded_instance_config (known after apply)

          + taint {
              + effect = "NO_SCHEDULE"
              + key    = "dedicated"
              + value  = "sig-testing"
            }

          + workload_metadata_config {
              + mode = "GKE_METADATA"
            }
        }

      + upgrade_settings (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

▶️ To apply this plan, comment:

atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

🚮 To delete this plan and lock, click here

🔁 To plan this project again, comment:

atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 0 to change, 0 to destroy.

⏩ To apply all unapplied plans from this Pull Request, comment:
```
atlantis apply
```
🚮 To delete all plans and locks from this Pull Request, comment:
```
atlantis unlock
```

ameukam · 2025-03-27T23:20:31Z

atlantis apply

k8s-infra-ci-robot · 2025-03-27T23:24:23Z

Ran Apply for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Creating...
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [10s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [20s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [30s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [40s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [50s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m1s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m11s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m21s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m31s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m41s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [1m51s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m1s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m11s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m21s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m31s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m41s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [2m51s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [3m1s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [3m11s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [3m21s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [3m31s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Still creating... [3m41s elapsed]
module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool: Creation complete after 3m45s [id=projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

upodroid

LGTM

k8s-ci-robot · 2025-03-30T12:48:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~infra/gcp/terraform/OWNERS~~ [ameukam,upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

upodroid · 2025-03-30T15:39:47Z

Canceling the hold as the infra change has been applied

/hold cancel

k8s-infra-ci-robot · 2025-03-30T15:44:37Z

Locks and plans deleted for the projects and workspaces modified in this pull request:

dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Follow-up: - kubernetes/k8s.io#7943 move some prowjobs to a dedicated nodepool in order to evaluate new instances. Signed-off-by: Arnaud Meukam <[email protected]>

Follow-up of: - kubernetes/k8s.io#7943 Move ci-test-infra-continuous-test to a dedicated nodepool Signed-off-by: Arnaud Meukam <[email protected]>

BenTheElder · 2025-05-17T00:28:54Z

infra/gcp/terraform/k8s-infra-prow-build/main.tf

+  disk_size_gb    = 500
+  disk_type       = "hyperdisk-balanced"
+  service_account = module.prow_build_cluster.cluster_node_sa.email
+  taints          = [{ key = "dedicated", value = "sig-testing", effect = "NO_SCHEDULE" }]


weirdly I don't see this taint on the actual cluster nodes?

$gcloud container node-pools describe pool6-20250327232037500200000001 --project k8s-infra-prow-build --format='value(config.taints)' --location us-central1 --cluster prow-build {'effect': 'NO_SCHEDULE', 'key': 'dedicated', 'value': 'sig-testing'}

k8s-ci-robot requested review from upodroid and Verolop March 26, 2025 19:01

k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Mar 26, 2025

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 26, 2025

ameukam commented Mar 26, 2025

View reviewed changes

infra/gcp/terraform/k8s-infra-prow-build/main.tf Show resolved Hide resolved

ameukam force-pushed the prow-build-nodepool-c4 branch from 4c22d88 to 993f75a Compare March 26, 2025 21:54

ameukam force-pushed the prow-build-nodepool-c4 branch from 993f75a to a29a7c6 Compare March 27, 2025 13:30

ameukam force-pushed the prow-build-nodepool-c4 branch from a29a7c6 to e135ed4 Compare March 27, 2025 23:16

upodroid approved these changes Mar 30, 2025

View reviewed changes

k8s-ci-robot assigned upodroid Mar 30, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 30, 2025

k8s-ci-robot merged commit 610792f into kubernetes:main Mar 30, 2025
13 checks passed

k8s-ci-robot added this to the v1.33 milestone Mar 30, 2025

This was referenced Mar 31, 2025

sig-testing: run some tests on dedicated nodepool kubernetes/test-infra#34630

Closed

sig-testing: move canaries tests to dedicated nodepool kubernetes/test-infra#34642

Merged

ameukam added a commit to ameukam/test-infra that referenced this pull request Apr 13, 2025

sig-testing: move test to dedicated nodepool

de1aa83

Follow-up of: - kubernetes/k8s.io#7943 Move ci-test-infra-continuous-test to a dedicated nodepool Signed-off-by: Arnaud Meukam <[email protected]>

ameukam mentioned this pull request Apr 13, 2025

sig-testing: move test to dedicated nodepool kubernetes/test-infra#34679

Merged

ameukam added a commit to ameukam/test-infra that referenced this pull request Apr 24, 2025

sig-testing: move test to dedicated nodepool

5ede168

Follow-up of: - kubernetes/k8s.io#7943 Move ci-test-infra-continuous-test to a dedicated nodepool Signed-off-by: Arnaud Meukam <[email protected]>

upodroid mentioned this pull request May 6, 2025

test the new GKE nodes with sig-testing canary jobs kubernetes/test-infra#34761

Merged

ameukam mentioned this pull request May 7, 2025

GCP: add GKE nodepool for SIG Testing #8078

Open

BenTheElder reviewed May 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gcp: Add nodepool for tests #7943

gcp: Add nodepool for tests #7943

ameukam commented Mar 26, 2025

ameukam commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 27, 2025

ameukam commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

ameukam commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

upodroid left a comment

k8s-ci-robot commented Mar 30, 2025

upodroid commented Mar 30, 2025

k8s-infra-ci-robot commented Mar 30, 2025

BenTheElder May 17, 2025

ameukam May 17, 2025

gcp: Add nodepool for tests #7943

gcp: Add nodepool for tests #7943

Conversation

ameukam commented Mar 26, 2025

ameukam commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 26, 2025

k8s-infra-ci-robot commented Mar 27, 2025

ameukam commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

ameukam commented Mar 27, 2025

k8s-infra-ci-robot commented Mar 27, 2025

upodroid left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 30, 2025

upodroid commented Mar 30, 2025

k8s-infra-ci-robot commented Mar 30, 2025

BenTheElder May 17, 2025

Choose a reason for hiding this comment

ameukam May 17, 2025

Choose a reason for hiding this comment