Starting profile using podman driver fail with: "volume already exists" using random profile name

[nirs]

What Happened?

Running minikube in github actions started to fail with:

Error: volume with name test-af672ccebb7d698c8cd745eb7d14911b-cluster already exists:
volume already exists

We use a random profile name, so it cannot exists:

minikube start --profile test-af672ccebb7d698c8cd745eb7d14911b-cluster \
    --driver podman \
    --container-runtime cri-o \
    --disk-size 20g \
    --nodes 1 \
    --cni auto \
    --cpus 2 \
    --memory 2g \
    --alsologtostderr

The failures started this week, this code was running without errors for several month
as part of ramen github actions.

Example failing run:
https://github.com/RamenDR/ramen/actions/runs/5348065862/jobs/9697442318

Attach the log file

Attached log from our tests - this is a log from the drenv tool, showing the
minikube command, exitcode, output, and error captured during the failed run.

logs.txt

Operating System

Ubuntu

Driver

Podman

[nirs]

For reference, switching to docker driver fixes this issue, but we are already using
podman for building and managing container images and would like to avoid the docker
dependency.

[mascenzi80]

@nirs were you able to make podman work with minikube?

[afbjorklund]

You can seee sudo -n podman volume create test-af672ccebb7d698c8cd745eb7d14911b-cluster --label name.minikube.sigs.k8s.io=test-af672ccebb7d698c8cd745eb7d14911b-cluster --label created_by.minikube.sigs.k8s.io=true in the log, so the volume is created by minikube.

It seems to be some log spam from a previous error, the real issue is Error validating CNI config file

[nirs]

@nirs were you able to make podman work with minikube?

We switched to docker for our github actions tests, did not try again yet.

[nirs]

It seems to be some log spam from a previous error, the real issue is Error validating CNI config file

@afbjorklund Any clue on debugging this issue?

[afbjorklund]

All I saw was the log lines:
"Error validating CNI config file /etc/cni/net.d/test-af672ccebb7d698c8cd745eb7d14911b-cluster.conflist: [plugin bridge does not support config version \"1.0.0\" plugin portmap does not support config version \"1.0.0\" plugin firewall does not support config version \"1.0.0\" plugin tuning does not support config version \"1.0.0\"]"

Wonder what "auto" picks up...

      I0622 16:35:41.107655    1891 cni.go:84] Creating CNI manager for "auto"
      I0622 16:35:41.107672    1891 cni.go:142] "podman" driver + "crio" runtime found, recommending kindnet

EDIT: But maybe I am misreading this, and it is the podman network that is broken ?

The crio network should have a different name. Maybe the host just needs new CNI...

The error is weird though, since podman3 should be using cniVersion 0.4.0 for net

// Current reports the version of the CNI spec implemented by this library
func Current() string {
	return "0.4.0"
}

[nirs]

I created a repo for debugging this:
https://github.com/nirs/test-minikube-podman

Turns out that you get pretty old podman (3.4) in Ubuntu 22.04 while current version
is 4.6. You can install the latest version via the Kubic project:
https://build.opensuse.org/package/show/devel:kubic:libcontainers:unstable/podman

With podman 4.6 minikube works fine:
https://github.com/nirs/test-minikube-podman/actions/runs/5898563462/job/15999784769

So I think we can close this issue and add a known issue for using podman on Ubuntu here:
https://minikube.sigs.k8s.io/docs/drivers/podman/#known-issues

[rmsilva1973]

@afbjorklund take a look at the details I added to issue #17154 regarding podman CNI plugin.

[mascenzi80]

I created a repo for debugging this: https://github.com/nirs/test-minikube-podman

Turns out that you get pretty old podman (3.4) in Ubuntu 22.04 while current version is 4.6. You can install the latest version via the Kubic project: https://build.opensuse.org/package/show/devel:kubic:libcontainers:unstable/podman

With podman 4.6 minikube works fine: https://github.com/nirs/test-minikube-podman/actions/runs/5898563462/job/15999784769

So I think we can close this issue and add a known issue for using podman on Ubuntu here: https://minikube.sigs.k8s.io/docs/drivers/podman/#known-issues

@nirs when I attempt to install via kubic I only get version 4.4. I'm running Ubuntu 20, so I'm assuming that because you are running 22.04, you have access to 4.6. Any thoughts?

[nirs]

@nirs when I attempt to install via kubic I only get version 4.4. I'm running Ubuntu 20, so I'm assuming that because you are running 22.04, you have access to 4.6. Any thoughts?

I tried to add a ubuntu-20.04 build here:
https://github.com/nirs/test-minikube-podman/actions/runs/6088573062/job/16519507180

It seems there is no podman build for ubuntu 20.04 in kubic repo.

[mascenzi80]

@nirs not sure what I'm doing wrong, but I set up ubuntu22.04 in a VM to verify we can make this work before updating our servers. But I'm still only getting access to Podman 3.4. I'm following your instructions from github test

  sudo mkdir -p /etc/apt/keyrings
  
  curl -fsSL "https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/Release.key" \
    | gpg --dearmor \
    | sudo tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null
	
  echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\
      https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/ /" \
    | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null
	
  sudo apt-get update -qq
  
  sudo apt-get -qq -y install podman