Kubernetes机器学习平台搭建：构建企业级ML训练环境

一、机器学习平台概述

Kubernetes机器学习平台是基于K8s构建的ML训练和部署基础设施，支持数据科学家进行模型训练、验证和部署。

1.1 ML平台架构

                      ┌─────────────────────────┐
                      │        用户界面          │
                      │  (Jupyter/TensorBoard) │
                      └───────────┬─────────────┘
                                  │
        ┌─────────────────────────┼─────────────────────────┐
        │                         │                         │
        ▼                         ▼                         ▼
┌───────────────┐         ┌───────────────┐         ┌───────────────┐
│   训练调度器   │         │   模型仓库     │         │   数据存储     │
│   (Kubeflow)  │         │  (MLflow)     │         │  (MinIO)      │
└───────────────┘         └───────────────┘         └───────────────┘
        │                         │                         │
        ▼                         ▼                         ▼
┌───────────────┐         ┌───────────────┐         ┌───────────────┐
│   GPU节点池    │         │   CPU节点池    │         │   存储集群     │
│  (训练任务)   │         │  (预处理)     │         │  (数据/模型)  │
└───────────────┘         └───────────────┘         └───────────────┘

1.2 核心组件

组件 功能 工具
训练调度 管理训练任务 Kubeflow、Argo Workflows
模型管理 模型版本控制 MLflow、DVC
数据存储 数据集管理 MinIO、PV/PVC
资源管理 GPU/CPU调度 Kubernetes调度器
可视化 实验追踪 TensorBoard、Weights & Biases

二、Kubeflow部署

2.1 Kubeflow安装

# 安装Kubeflow
export KUBEFLOW_RELEASE_VERSION=v1.8.0
export KUSTOMIZE_VERSION=v5.0.1

git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout ${KUBEFLOW_RELEASE_VERSION}

# 部署Kubeflow
while ! kustomize build example | kubectl apply -f -; do echo "Retrying..."; sleep 10; done

2.2 Kubeflow Pipeline配置

apiVersion: kubeflow.org/v1
kind: Pipeline
metadata:
  name: ml-pipeline
spec:
  pipelineSpec:
    tasks:
      - name: preprocess
        taskSpec:
          podSpec:
            containers:
              - name: preprocess
                image: preprocess:latest
                command: ["python", "preprocess.py"]
      - name: train
        taskSpec:
          podSpec:
            containers:
              - name: train
                image: train:latest
                command: ["python", "train.py"]
                resources:
                  limits:
                    nvidia.com/gpu: 1
        dependencies:
          - preprocess

三、MLflow配置

3.1 MLflow部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow
  namespace: mlflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mlflow
  template:
    metadata:
      labels:
        app: mlflow
    spec:
      containers:
        - name: mlflow
          image: mlflow:latest
          ports:
            - containerPort: 5000
          env:
            - name: MLFLOW_S3_ENDPOINT_URL
              value: http://minio:9000
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: accesskey
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: secretkey
          command:
            - mlflow
            - server
            - --host=0.0.0.0
            - --port=5000
            - --backend-store-uri=postgresql://mlflow:password@postgres/mlflow
            - --default-artifact-root=s3://mlflow/

3.2 MLflow模型注册

import mlflow
import mlflow.sklearn

mlflow.set_tracking_uri("http://mlflow:5000")

with mlflow.start_run():
    # 训练模型
    model = train_model()
    
    # 记录参数
    mlflow.log_param("learning_rate", 0.01)
    
    # 记录指标
    mlflow.log_metric("accuracy", 0.95)
    
    # 保存模型
    mlflow.sklearn.log_model(model, "model")
    
    # 注册模型
    mlflow.register_model(
        "runs:/{}/model".format(mlflow.active_run().info.run_id),
        "my-model"
    )

四、GPU资源管理

4.1 GPU节点配置

apiVersion: v1
kind: Node
metadata:
  name: gpu-node-01
  labels:
    nvidia.com/gpu.present: "true"
    node-role.kubernetes.io/gpu: ""
spec:
  taints:
    - key: nvidia.com/gpu
      value: "true"
      effect: NoSchedule

4.2 GPU资源请求

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-pod
spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Equal
      value: "true"
      effect: NoSchedule
  containers:
    - name: training
      image: tensorflow/tensorflow:latest-gpu
      command: ["python", "train.py"]
      resources:
        limits:
          nvidia.com/gpu: 2
          memory: 32Gi
          cpu: "8"
        requests:
          nvidia.com/gpu: 2
          memory: 16Gi
          cpu: "4"

五、数据存储配置

5.1 MinIO部署

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
  namespace: minio
spec:
  serviceName: minio
  replicas: 4
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
        - name: minio
          image: minio/minio:latest
          ports:
            - containerPort: 9000
          command:
            - minio
            - server
            - /data
            - --console-address
            - ":9001"
          volumeMounts:
            - name: data
              mountPath: /data
          env:
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: accesskey
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: secretkey
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

5.2 PVC配置

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data
  namespace: ml
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi
  storageClassName: nfs-storage

六、JupyterHub部署

6.1 JupyterHub配置

apiVersion: hub.jupyter.org/v1
kind: Hub
metadata:
  name: jupyterhub
  namespace: jupyterhub
spec:
  image:
    name: jupyterhub/k8s-hub
    tag: 2.0.0
  proxy:
    secretToken: <secret-token>
  auth:
    type: github
    github:
      clientId: <client-id>
      clientSecret: <client-secret>
      callbackUrl: https://jupyter.example.com/hub/oauth_callback
  singleuser:
    image:
      name: jupyter/scipy-notebook
      tag: latest
    storage:
      type: persistent-claim
      capacity: 10Gi

6.2 用户配置

apiVersion: hub.jupyter.org/v1
kind: User
metadata:
  name: datascientist
  namespace: jupyterhub
spec:
  profile:
    displayName: Data Scientist
    admin: false
  server:
    resources:
      limits:
        cpu: "4"
        memory: 16Gi
      requests:
        cpu: "2"
        memory: 8Gi

七、TensorBoard配置

7.1 TensorBoard部署

apiVersion: v1
kind: Service
metadata:
  name: tensorboard
  namespace: ml
spec:
  type: ClusterIP
  selector:
    app: tensorboard
  ports:
    - port: 6006
      targetPort: 6006

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tensorboard
  namespace: ml
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tensorboard
  template:
    metadata:
      labels:
        app: tensorboard
    spec:
      containers:
        - name: tensorboard
          image: tensorflow/tensorflow:latest
          command:
            - tensorboard
            - --logdir=/logs
            - --host=0.0.0.0
          ports:
            - containerPort: 6006
          volumeMounts:
            - name: logs
              mountPath: /logs
      volumes:
        - name: logs
          persistentVolumeClaim:
            claimName: tensorboard-logs

八、模型部署

8.1 TensorFlow Serving

apiVersion: v1
kind: Service
metadata:
  name: tf-serving
  namespace: ml
spec:
  type: ClusterIP
  selector:
    app: tf-serving
  ports:
    - port: 8501
      targetPort: 8501

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tf-serving
  namespace: ml
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tf-serving
  template:
    metadata:
      labels:
        app: tf-serving
    spec:
      containers:
        - name: tf-serving
          image: tensorflow/serving:latest
          ports:
            - containerPort: 8500
            - containerPort: 8501
          args:
            - "--model_name=my-model"
            - "--model_base_path=s3://models/my-model"
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: accesskey
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-creds
                  key: secretkey
            - name: S3_ENDPOINT
              value: http://minio:9000

8.2 gRPC推理服务

apiVersion: v1
kind: Service
metadata:
  name: model-service
  namespace: ml
spec:
  type: ClusterIP
  selector:
    app: model-service
  ports:
    - port: 9000
      targetPort: 9000
      name: grpc
    - port: 8080
      targetPort: 8080
      name: http

九、监控与日志

9.1 训练指标监控

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ml-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: ml-exporter
  endpoints:
    - port: metrics
      interval: 15s

9.2 资源使用监控

apiVersion: v1
kind: ConfigMap
metadata:
  name: ml-metrics-config
  namespace: monitoring
data:
  prometheus.rules: |
    groups:
      - name: ml.rules
        rules:
          - record: ml_training_duration_seconds
            expr: sum(rate(kube_pod_running_duration_seconds{app="training"}[5m]))
          - record: ml_gpu_utilization
            expr: sum(nvidia_gpu_utilization{job="nvidia-dcgm-exporter"})

十、总结

Kubernetes机器学习平台搭建需要考虑：

  1. 训练调度：使用Kubeflow管理ML工作流
  2. 模型管理：使用MLflow进行模型版本控制
  3. GPU资源：配置GPU节点池和资源调度
  4. 数据存储：部署MinIO管理数据集
  5. 开发环境：使用JupyterHub提供交互式开发
  6. 可视化：配置TensorBoard进行实验追踪
  7. 模型部署：使用TensorFlow Serving部署模型
  8. 监控告警：建立训练指标和资源使用监控

建议根据团队规模和业务需求选择合适的组件，构建高效的ML平台。

参考资料：

# 云原生 # kubernetes
Logo
https://edu.csdn.net/learn/39067/627173?utm_source=2019755004

汇聚全球AI编程工具，助力开发者即刻编程。

更多推荐

cover

一个 API 入口调用多个大模型：AiiOnly客户端 + CC Switch 打通 Codex AI编程流程

avatar AI编程社区
cover

VSCode Claude Code + MiMo 中转免登录完整配置教程（解决原生登录弹窗 + 模型报错）

avatar AI编程社区
cover

从零开始做一个属于自己的 Skill（保姆级教程）

avatar AI编程社区