[Bug]: Driver Pod Restart Causes Init:CrashLoopBackOff When GPU Workloads Are Running

[moditanisha22]

Describe the bug
We have observed an issue while upgrading the GPU Operator from version 25.3.4 to 25.10.0 in a multi-node GPU cluster. During the upgrade, if a node has active GPU workloads, the GPU driver pod can get stuck in the Init:CrashLoopBackOff state.

According to NVIDIA’s documentation on the GPU Operator upgrade workflow:
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html the expected behavior is that the upgrade controller first terminates all GPU workloads and then proceeds with the driver upgrade.

However, during our testing, we observed different behavior.
Before the upgrade started, the upgrade-done label was already present on the node from the older version. When the upgrade began, the upgrade controller did not trigger. Instead, the driver DaemonSet rollout started immediately, causing the driver pod to restart with the new version.

This behavior was unexpected because the driver DaemonSet upgrade strategy is set to OnDelete, and deletion of driver pods should be controlled by the upgrade controller. In this case, the upgrade controller did not start as expected.

During the restart, the driver pod’s init container (k8s-driver-manager) attempted to delete the GPU workloads but failed because the autoUpgrade policy was enabled, as indicated in the logs.

In a ubuntu os based clusters, when a node had active GPU workloads, the upgrade controller did not trigger as expected. Instead, the driver DaemonSet rollout started immediately, leading to a driver pod restart without proper workload eviction. This resulted in the deadlock scenario and caused the driver pod to get stuck.

k8s-driver-manager logs

W0225 08:26:00.486943 1515429 client_config.go:667] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time=2026-02-25T08:26:00Z level=info msg=Starting driver uninstallation process
time=2026-02-25T08:26:00Z level=info msg=Fetching current component labels
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.operator-validator" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.operator-validator"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.container-toolkit" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.container-toolkit"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.device-plugin" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.device-plugin"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.gpu-feature-discovery" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.gpu-feature-discovery"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.dcgm-exporter" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.dcgm-exporter"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.dcgm" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.dcgm"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.mig-manager" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.mig-manager"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.nvsm" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.nvsm"=true
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.sandbox-validator" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.sandbox-validator"=
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.sandbox-device-plugin" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.sandbox-device-plugin"=
time=2026-02-25T08:26:00Z level=info msg=Getting current value of the "nvidia.com/gpu.deploy.vgpu-device-manager" node label
time=2026-02-25T08:26:00Z level=info msg=Current value of "nvidia.com/gpu.deploy.vgpu-device-manager"=
time=2026-02-25T08:26:00Z level=info msg=Current value of AUTO_UPGRADE_POLICY_ENABLED=true
time=2026-02-25T08:26:00Z level=info msg=Shutting down all GPU clients on the current node by disabling their component-specific nodeSelector labels
time=2026-02-25T08:26:01Z level=info msg=Waiting for nvidia-operator-validator to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for nvidia-container-toolkit-daemonset to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for nvidia-device-plugin-daemonset to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for gpu-feature-discovery to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for nvidia-dcgm-exporter to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for nvidia-dcgm to shutdown
time=2026-02-25T08:26:06Z level=info msg=Waiting for mig-manager to shutdown
time=2026-02-25T08:26:06Z level=info msg=Auto eviction of GPU pods on node <node_name> is disabled by the upgrade policy
time=2026-02-25T08:26:06Z level=info msg=Cleaning up NVIDIA driver
time=2026-02-25T08:26:06Z level=info msg=Unloading NVIDIA driver kernel modules
time=2026-02-25T08:26:06Z level=warning msg=Failed to unload kernel module nvidia_uvm: resource temporarily unavailable
time=2026-02-25T08:26:06Z level=warning msg=Failed to unload kernel module nvidia: resource temporarily unavailable
time=2026-02-25T08:26:06Z level=info msg=Could not unload NVIDIA driver kernel modules, driver is in use
time=2026-02-25T08:26:06Z level=info msg=Module               Size       Ref Count       Used by

time=2026-02-25T08:26:06Z level=info msg=Auto drain of the node is disabled by the upgrade policy
time=2026-02-25T08:26:06Z level=error msg=Failed to uninstall nvidia driver components
time=2026-02-25T08:26:06Z level=info msg=Performing cleanup on failure
time=2026-02-25T08:26:06Z level=info msg=Auto eviction of GPU pods on node <node_name> is disabled by the upgrade policy
time=2026-02-25T08:26:06Z level=info msg=Auto drain of the node is disabled by the upgrade policy
time=2026-02-25T08:26:06Z level=info msg=Rescheduling all GPU clients on the current node by enabling their component-specific nodeSelector labels
nvidia_uvm           2179072    48              -
nvidia               11554816   206             nvidia_uvm,
ecc                  45056      1               nvidia,
time=2026-02-25T08:26:06Z level=fatal msg=failed to uninstall nvidia driver components: failed to unload driver: resource temporarily unavailable

This resulted in a deadlock situation:

• The driver pod restart requires GPU workloads to be terminated in order to unload the driver module.
• The GPU workloads were not terminated because the upgrade controller workflow was not triggered.
  As a result, neither the driver pod nor the workloads could proceed, and the driver pod remained stuck in the Init:CrashLoopBackOff state.

To Reproduce

• Install v25.3.4 GPU Operator with driver.usePrecompiled was enabled.
• Run any GPU Workload.
• Upgrade to v25.10.0 with running workloads with driver.usePrecompiled was disabled.
  or
  Manually restart the driver DaemonSet pod running on a node that has active GPU workloads.

Expected behavior
As per documentation https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html, the expected behavior during the upgrade process is:

1. Disable all clients to the GPU driver.
2. Unload the current GPU driver kernel modules.
3. Start the updated GPU driver pod.
4. Install the updated GPU driver and load the updated kernel modules.
5. Enable the clients of the GPU driver.

Environment (please provide the following information):

• GPU Operator Version: v25.3.4 upgrade to v25.10.0
• OS: Ubuntu24.04
• Kernel Version: 6.8.0-90-generic
• Container Runtime Version: 2.1.5
• Kubernetes Distro and Version: k8s version (v1.34.2)

Gpu_operator pod logs
gpu-operator.log

Cluster-policy configurations

 driver:
      enabled: true
      manager:
        env:
        - name: ENABLE_GPU_POD_EVICTION
          value: "true"
        - name: ENABLE_AUTO_DRAIN
          value: "false"
        - name: DRAIN_USE_FORCE
          value: "false"
        - name: DRAIN_POD_SELECTOR_LABEL
          value: ""
        - name: DRAIN_TIMEOUT_SECONDS
          value: 0s
        - name: DRAIN_DELETE_EMPTYDIR_DATA
          value: "true"
        image: k8s-driver-manager
      upgradePolicy:
        autoUpgrade: true
        drain:
          deleteEmptyDir: false
          enable: false
          force: false
          timeoutSeconds: 300
        maxParallelUpgrades: 1
        maxUnavailable: 25%
        podDeletion:
          deleteEmptyDir: true
          force: true
          timeoutSeconds: 900

[rajathagasthya]

@moditanisha22 Are you able to provide us more logs by running the must-gather.sh script? You can do:

curl -sL https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh | bash

[tariq1890]

It is expected that all active GPU workload pods be brought down so that the driver upgrade can happen successfully. Although the k8s-driver-manager init containers lists and deletes all workload pods that requests GPUs via resource requests and limits, It is possible that you may have other gpu workload pods that don't requests GPUs this but instead, use the NVIDIA_VISIBLE_DEVICES env variable. The k8s-driver-manager has no way to reliably retrieve these pods, so you will have to tear them down yourself in order to unblock the driver upgrade controller.

[moditanisha22]

@moditanisha22 Are you able to provide us more logs by running the must-gather.sh script? You can do:

curl -sL https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh | bash

Attached logs-

nvidia-gpu-operator_20260226_0011.zip

During further investigation, this issue was found to be related to the behavior difference when switching from precompiled to compiled drivers during the upgrade.

In gpu-operator v25.3.x, driver.usePrecompiled was enabled .
In gpu-operator v25.10.0, it was changed to false (precompiled drivers disabled), which switches the setup from precompiled to compiled drivers.

This change modified the driver DaemonSet by removing the nodeSelector added for precompiled drivers. As a result, Kubernetes detected a DaemonSet change and triggered an automatic rollout of the driver pods. This rollout bypassed the OnDelete strategy, and the upgrade controller did not get a chance to manage GPU workload eviction on multi-node clusters. When GPU workloads were still running, this led to the Init:CrashLoopBackOff issue.

To validate this behavior, the following test was performed:

• Installed gpu-operator v25.3.x with driver.usePrecompiled=true
• Patched ClusterPolicy to disable precompiled drivers before upgrade
• Ensured all pods were stable
• Upgraded to gpu-operator v25.10.0

In this scenario, there was no change in the driver DaemonSet during the upgrade, no automatic rollout was triggered, the upgrade controller executed correctly, GPU workloads were terminated, and the issue did not occur.

This confirms that the DaemonSet change caused by switching from precompiled to compiled drivers is a key trigger for this problem.

[rahulait]

Hi @moditanisha22, thank you for the detailed report. This looks like a feature we don't support as of today.

We currently support following driver installations:

1. Compiled drivers managed by clusterpolicy
2. Precompiled drivers managed by clusterpolicy
3. Compiled drivers managed by nvidiadriver
4. Precompiled drivers managed by nvidiadriver

We don't support seamless upgrades from one type to another as of today. In your case, upgrade or switch was done from precompiled to compiled drivers.

In case of precompiled driver with clusterpolicy, the driver daemonset name would be something like:

k get daemonset -n gpu-operator | grep driver
nvidia-driver-daemonset-5.15.0-171-generic-ubuntu22.04   1         1         1       1            1           feature.node.kubernetes.io/kernel-version.full=5.15.0-171-generic,nvidia.com/gpu.deploy.driver=true   3d20h

In case of compiled driver with clusterpolicy, the driver daemonset name would be something like:

k get daemonset -n gpu-operator | grep driver
nvidia-driver-daemonset                      1         1         1       1            1           nvidia.com/gpu.deploy.driver=true                                      2d19h

Since you changed from pre-compiled to compiled, entire daemonset would be deleted and new daemonset created, and there is no seamless migration of driver pods from one daemonset to another. If this switch (from one type to another) is done before upgrading to newer release/driver version, then only driver daemonset's content will change during upgrade and upgrade-controller will take care of rolling out all driver pods in a controlled manner.

I am marking this issue as an enhancement we can look to support in future.

[moditanisha22]

Tried to manually restart the driver pod on a setup without pre-compiled drivers where active GPU workloads are running. When restarting the driver pod with an active GPU workload, the pod gets stuck in Init:CrashLoopBackOff because the k8s-driver-manager fails to unload NVIDIA kernel modules that are still in use.

Reproduction Steps

• Deploy GPU Operator (compiled drivers, driver.usePrecompiled=false)
• Run active GPU workloads on the node
• Delete the driver pod:
• The driver pod enters Init:CrashLoopBackOff

OS: Ubuntu24.04, Rhel8.10

Observed Behavior

• The k8s-driver-manager init container attempts to unload NVIDIA kernel modules (nvidia, nvidia_uvm)
• Unload fails with "resource temporarily unavailable" because GPU workloads still hold references to the kernel modules
• The workload pod was not using the NVIDIA_VISIBLE_DEVICES environment variable and was requesting GPUs via standard resource limits, yet the driver cleanup still failed.

Cluster-policy configurations

   driver:
      certConfig:
        name: ""
      enabled: true
      image: driver
      imagePullPolicy: IfNotPresent
      kernelModuleConfig:
        name: ""
      kernelModuleType: auto
      licensingConfig:
        nlsEnabled: true
        secretName: ""
      manager:
        env:
        - name: ENABLE_GPU_POD_EVICTION
          value: "true"
        - name: ENABLE_AUTO_DRAIN
          value: "false"
        - name: DRAIN_USE_FORCE
          value: "false"
        - name: DRAIN_POD_SELECTOR_LABEL
          value: ""
        - name: DRAIN_TIMEOUT_SECONDS
          value: 0s
        - name: DRAIN_DELETE_EMPTYDIR_DATA
          value: "true"
        image: k8s-driver-manager
        imagePullPolicy: IfNotPresent
        repository: nvcr.io/nvidia/cloud-native
        version: v0.9.1
      rdma:
        enabled: false
        useHostMofed: false
      repoConfig:
        configMapName: custom-repo
      repository: nvcr.io/nvidia
      resources:
        requests:
          cpu: 100m
          memory: 4096M
      upgradePolicy:
        autoUpgrade: true
        drain:
          deleteEmptyDir: false
          enable: false
          force: false
          timeoutSeconds: 300
        maxParallelUpgrades: 1
        maxUnavailable: 25%
        podDeletion:
          deleteEmptyDir: true
          force: true
          timeoutSeconds: 900
        waitForCompletion:
          timeoutSeconds: 0
      useNvidiaDriverCRD: false
      usePrecompiled: false
      version: 580.126.16

[rahulait]

What if you disable upgrade controller? Basically same as above, except changing autoUpgrade to false:

      upgradePolicy:
        autoUpgrade: false