Fixing the 'could not select device driver' GPU Error in Docker

Why This Error Happens

You hit enter on docker run --gpus all, and instead of a running container, you get a blunt error message. It basically means Docker is looking for a GPU-capable driver but can't find one. Out of the box, Docker uses the standard runc runtime. While runc is great for isolating CPU and RAM, it has no idea how to talk to NVIDIA hardware.

To fix this, you need a "translator" called the NVIDIA Container Toolkit. Without it, Docker doesn't know what the --gpus flag even means.

Step 1: Check Your Host Drivers

Everything starts with the base drivers. If your host OS can't see the GPU, Docker definitely won't. Open your terminal and run:

nvidia-smi

You should see a status table showing your driver version (e.g., 535.129.03) and your GPU model, like an RTX 3080 or A100. If you see "command not found" or a communication error, stop here. You must install the NVIDIA proprietary drivers before touching Docker.

Step 2: Install the NVIDIA Container Toolkit

Most modern Linux setups (Ubuntu 22.04, Debian, or CentOS) follow a similar path. We need to add NVIDIA's official package repository so your system knows where to download the toolkit.

First, set up the GPG key and the repository list:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Now, refresh your package list and install the toolkit. It's a small download, usually under 20MB.

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Step 3: Register the NVIDIA Runtime

Simply having the toolkit on your hard drive isn't enough. You have to tell Docker to actually use it. NVIDIA provides a handy utility that automates the configuration of your daemon.json file.

sudo nvidia-ctk runtime configure --runtime=docker

This command updates /etc/docker/daemon.json. It adds a new runtime named "nvidia" and points it to the toolkit binary. If you're curious, you can peek at the file with cat /etc/docker/daemon.json to see the new configuration block.

Step 4: Restart and Verify

Docker only reads its configuration when it starts up. Restart the service to apply your changes:

sudo systemctl restart docker

Now for the moment of truth. Run a tiny CUDA container to see if it can access the GPU. We'll use the official NVIDIA image to run nvidia-smi from inside the container.

docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu22.04 nvidia-smi

Success looks like a familiar GPU table appearing in your terminal. If you see it, the bridge is built.

Troubleshooting Docker Compose

If you're using Docker Compose and still seeing the error, your YAML file is likely the culprit. Compose requires a specific deploy block to reserve hardware. Ensure your docker-compose.yml uses version 3.8 or higher.

services:
  gpu-worker:
    image: nvidia/cuda:12.0-base-ubuntu22.04
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Pro-Tips for Stability

• WSL2 Users: Don't install NVIDIA drivers inside your Ubuntu WSL distro. Install the latest Game Ready or Studio drivers on Windows. Docker Desktop handles the rest automatically.
• Kernel Updates: Sometimes a Linux kernel update breaks the NVIDIA driver. If nvidia-smi stops working on the host, Docker will fail too. A simple sudo apt install --reinstall nvidia-driver-<version> usually fixes it.
• Persistence: On production rigs, use sudo nvidia-smi -pm 1 to enable persistence mode. This keeps the driver loaded and reduces container startup latency by about 100-200ms.