v3.12.2 breaks with "ensure CRDs are installed first" with `--disable-validation`

[z0rc]

Helmfile version 1.1.2.

After upgrading helm-diff from v3.12.1 to v3.12.2, I started to see errors running helmfile apply on empty cluster. My helmfile.yaml.gotmpl has number of releases where one release installs CRD (as example using https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds) and second release deployes CR. Second release needs first release and also has disableValidationOnInstall: true.

My env vars:

% env | grep HELM
HELM_DIFF_COLOR=true
HELM_DIFF_NORMALIZE_MANIFESTS=true
HELM_DIFF_OUTPUT_CONTEXT=5
HELM_DIFF_THREE_WAY_MERGE=true
HELM_DIFF_USE_UPGRADE_DRY_RUN=true

I tried to change HELM_DIFF_USE_UPGRADE_DRY_RUN=false, but it didn't have an effect on the error. #795 might be related.

Error:

PATH:
  /opt/homebrew/bin/helm

ARGS:
  0: helm (4 bytes)
  1: --kube-context (14 bytes)
  2: poc-01 (6 bytes)
  3: diff (4 bytes)
  4: upgrade (7 bytes)
  5: --allow-unreleased (18 bytes)
  6: node-local-dns (14 bytes)
  7: deliveryhero/node-local-dns (27 bytes)
  8: --version (9 bytes)
  9: 2.1.8 (5 bytes)
  10: --disable-validation (20 bytes)
  11: --kube-context (14 bytes)
  12: poc-01 (6 bytes)
  13: --namespace (11 bytes)
  14: kube-system (11 bytes)
  15: --values (8 bytes)
  16: /var/folders/r0/96rf0nss61v_4twdyp_k4zf40000gn/T/helmfile248792395/kube-system-node-local-dns-values-699c4c4fb7 (111 bytes)
  17: --reset-values (14 bytes)
  18: --detailed-exitcode (19 bytes)
  19: --color (7 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Enabled three way merge via the envvar
  Enabled normalize manifests via the envvar
  Error: unable to generate manifests: unable to build kubernetes objects from new release manifest: resource mapping not found for name: "node-local-dns" namespace: "kube-system" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
  ensure CRDs are installed first
  Error: plugin "diff" exited with error

COMBINED OUTPUT:
  Enabled three way merge via the envvar
  Enabled normalize manifests via the envvar
  Error: unable to generate manifests: unable to build kubernetes objects from new release manifest: resource mapping not found for name: "node-local-dns" namespace: "kube-system" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
  ensure CRDs are installed first
  Error: plugin "diff" exited with error
err 11: command "/opt/homebrew/bin/helm" exited with non-zero status:

[z0rc]

Here is sample helmfile.yaml.gotmpl for reproduction:

---
repositories:
- name: prometheus-community
  url: https://prometheus-community.github.io/helm-charts
- name: deliveryhero
  url: https://github.com/raw/deliveryhero/helm-charts/refs/heads/master/

releases:
- name: prometheus-operator-crds
  namespace: monitoring
  chart: prometheus-community/prometheus-operator-crds
  version: 21.0.0
  suppressDiff: true
- name: node-local-dns
  namespace: kube-system  # it must be in kube-system as it depends on kube-dns service in that namespace
  chart: deliveryhero/node-local-dns
  version: 2.1.8
  disableValidationOnInstall: true
  needs:
  - monitoring/prometheus-operator-crds
  values:
  - config:
      dnsServer: 8.8.8.8  # put a real ip kube-dns service here
      bindIp: true
      commProtocol: prefer_udp
      prefetch:
        enabled: true
    serviceMonitor:
      enabled: true

With helm-diff v.3.12.1 the helmfile apply is successful:

% helmfile apply
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Adding repo deliveryhero https://github.com/raw/deliveryhero/helm-charts/refs/heads/master/
"deliveryhero" has been added to your repositories

Listing releases matching ^node-local-dns$
Comparing release=prometheus-operator-crds, chart=prometheus-community/prometheus-operator-crds, namespace=monitoring

Comparing release=node-local-dns, chart=deliveryhero/node-local-dns, namespace=kube-system
Upgrading release=prometheus-operator-crds, chart=prometheus-community/prometheus-operator-crds, namespace=monitoring
Release "prometheus-operator-crds" does not exist. Installing it now.
NAME: prometheus-operator-crds
LAST DEPLOYED: Tue Jun 24 12:52:25 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^prometheus-operator-crds$
prometheus-operator-crds        monitoring      1               2025-06-24 12:52:25.345657 +0300 EEST   deployed        prometheus-operator-crds-21.0.0 v0.83.0

Upgrading release=node-local-dns, chart=deliveryhero/node-local-dns, namespace=kube-system
Release "node-local-dns" does not exist. Installing it now.
NAME: node-local-dns
LAST DEPLOYED: Tue Jun 24 12:52:37 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^node-local-dns$
node-local-dns  kube-system     1               2025-06-24 12:52:37.5851 +0300 EEST     deployed        node-local-dns-2.1.8    1.26.4


UPDATED RELEASES:
NAME                       NAMESPACE     CHART                                           VERSION   DURATION
prometheus-operator-crds   monitoring    prometheus-community/prometheus-operator-crds   21.0.0         12s
node-local-dns             kube-system   deliveryhero/node-local-dns                     2.1.8           8s

With helm-diff v.3.12.2 the helmfile apply fails:

% helmfile apply
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Adding repo deliveryhero https://github.com/raw/deliveryhero/helm-charts/refs/heads/master/
"deliveryhero" has been added to your repositories

Listing releases matching ^node-local-dns$
Comparing release=prometheus-operator-crds, chart=prometheus-community/prometheus-operator-crds, namespace=monitoring

Comparing release=node-local-dns, chart=deliveryhero/node-local-dns, namespace=kube-system
in ./helmfile.yaml.gotmpl: command "/opt/homebrew/bin/helm" exited with non-zero status:

PATH:
  /opt/homebrew/bin/helm

ARGS:
  0: helm (4 bytes)
  1: diff (4 bytes)
  2: upgrade (7 bytes)
  3: --allow-unreleased (18 bytes)
  4: node-local-dns (14 bytes)
  5: deliveryhero/node-local-dns (27 bytes)
  6: --version (9 bytes)
  7: 2.1.8 (5 bytes)
  8: --disable-validation (20 bytes)
  9: --namespace (11 bytes)
  10: kube-system (11 bytes)
  11: --values (8 bytes)
  12: /var/folders/r0/96rf0nss61v_4twdyp_k4zf40000gn/T/helmfile1347727151/kube-system-node-local-dns-values-666c444b6b (112 bytes)
  13: --reset-values (14 bytes)
  14: --detailed-exitcode (19 bytes)
  15: --color (7 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Enabled three way merge via the envvar
  Enabled normalize manifests via the envvar
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "node-local-dns" namespace: "kube-system" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
  ensure CRDs are installed first
  Error: plugin "diff" exited with error

COMBINED OUTPUT:
  Enabled three way merge via the envvar
  Enabled normalize manifests via the envvar
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "node-local-dns" namespace: "kube-system" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
  ensure CRDs are installed first
  Error: plugin "diff" exited with error

[yxxhero]

@z0rc got it.

[yxxhero]

@z0rc could you post more logs by adding --debug flag?

[z0rc]

@yxxhero I'm attaching file with output of helmfile --debug apply for helm-diff v3.12.2 (one with the bug). Keep in mind it's 70K lines, because debug output contains full prometheus-operator crd definition.

helmfile_debug.txt

[yxxhero]

@z0rc could you post the v3.12.1 logs?

[z0rc]

@yxxhero here are logs of helmfile --debug apply with helm-diff v3.12.1.

helmfile_debug_3.12.1.txt