背景

起初是笔者想要将本地的docker-compose 启动的Milvus切换到GPU版本, 按照官方guide没法正常启动,于是研究了以下Milvus的GPU容器版本如何正常启动以及进一步讲如何在容器中正常使用宿主机的GPU.

启动失败分析

按照官方的guide容器启动GPU版本Milvus出现如下的情况

a5b9bf712bc7   milvusdb/milvus:v2.4.17-gpu                "/tini -- milvus run…"   3 minutes ago   Created

milvus的container处于created状态而不是正常运行的状态,使用指令 docker inspect milvus-standalone找到错误原因
"Error": "could not select device driver \"nvidia\" with capabilities: [[gpu]]"

"State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "could not select device driver \"nvidia\" with capabilities: [[gpu]]",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },

这个问题chatgpt说法是没有安装nvidia 容器运行时 nvidia-container-toolkit.
再从docker官方文档入手看如何解决,如下官方doc提到docker-compose 怎么使用GPU资源:
https://docs.docker.com/compose/how-tos/gpu-support/
在此文当的prerequirements里面提到需要额外安装的驱动或者工具:
https://docs.docker.com/engine/containers/resource_constraints/#gpu
提到额外需要安装的工具, 其中nvidia driver笔者早已经安装了, 从文档看下来笔者缺少的就是nvidia-container-toolkit
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

安装配置以及验证nvidia-container-toolkit

安装过程

参考nvida-container-toolkit的官方安装guide, 笔者系统是Ubuntu 22.04 所以采用apt安装

1. Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Optionally, configure the repository to use experimental packages:

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

2. Update the packages list from the repository:

sudo apt-get update

3. Install the NVIDIA Container Toolkit packages:

sudo apt-get install -y nvidia-container-toolkit

正常情况下nvidia-container-toolkit就安装好了,可以通过命令行check

nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.17.2

配置过程

笔者本机是docker环境,所以这里只罗列出docker相关配置

1. Configure the container runtime by using the nvidia-ctk command:

sudo nvidia-ctk runtime configure --runtime=docker

此操作会修改docker配置文件/etc/docker/daemon.json 查看是否生效

sudo cat /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

2. Restart the Docker daemon:

sudo systemctl restart docker

至此,nviida-container-toolkit的安装以及配置完成.

验证

使用nvidia-smi查看GPU device

0  NVIDIA GeForce RTX 4090

笔者只有一张显卡且device id 是 0 所以使用如下docker指令验证:

sudo docker run -it --rm --gpus device=0 ubuntu nvidia-smi

输出内容为容器内运行nvidia-smi指令的结果,不包错则nvidia-container-toolkit正常安装且正常配置.

重新启动milvus gpu容器版本

重新运行官方的milvus gpu docker-compose.yml 则可以正常启动, 附带笔者在官方的docker-compose.yml文件上添加的attu的完整版本docker-compose 文件, 通过localhost:3000则可以直接访问attu操作milvus.

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/data/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/data/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  
  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.17-gpu
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/data/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: ["gpu"]
              device_ids: ["0"]
    depends_on:
      - "etcd"
      - "minio"
 
  ui:
    image: zilliz/attu:v2.4.12
    container_name: attu
    restart: always
    ports:
      - "3000:3000"
    environment:
      - MILVUS_URL=standalone:19530
    depends_on:
      - "standalone"

networks:
  default:
    name: milvus_standalone_net

回到正题

回到正体,docker容器内如何正确使用宿主机的GPU资源.
其实只需要正确安装配置好nvidia-smi以及nvidia-container-toolkit参考docker关于正确使用宿主机器GPU的文档
https://docs.docker.com/compose/how-tos/gpu-support/
正确配置好device属性即可,参考官方给的一个example:

services:
  test:
    image: nvidia/cuda:12.3.1-base-ubuntu20.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]