高可用淘客机器人后端服务的容器化部署与 Kubernetes 自动扩缩容策略

大家好，我是 微赚淘客系统3.0 的研发者省赚客！

微赚淘客机器人需7×24小时响应用户消息（微信/Telegram），流量具有强突发性（如大促期间QPS飙升10倍）。传统虚拟机部署难以弹性应对，我们基于 Kubernetes + Quarkus 原生镜像构建容器化体系，结合 HPA 与 KEDA 实现秒级自动扩缩容，保障SLA ≥ 99.95%，资源成本降低40%。

一、轻量级容器镜像构建

使用 Quarkus + GraalVM 编译原生可执行文件，Dockerfile 极简：

# Dockerfile
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.7
WORKDIR /app
COPY target/*-runner /app/bot-service
RUN chmod +x /app/bot-service
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/q/health || exit 1
CMD ["./bot-service", "-Dquarkus.http.host=0.0.0.0"]

构建命令：

./mvnw package -Pnative -Dquarkus.native.container-build=true
docker build -t juwatech.cn/rebate-bot:v1.2.0 .

生成镜像仅 42MB，启动时间 < 30ms。

二、Kubernetes Deployment 配置

定义无状态服务：

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rebate-bot
  labels:
    app: rebate-bot
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rebate-bot
  template:
    metadata:
      labels:
        app: rebate-bot
    spec:
      containers:
      - name: bot
        image: juwatech.cn/rebate-bot:v1.2.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /q/health/live
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /q/health/ready
            port: 8080
          initialDelaySeconds: 2
          periodSeconds: 5

三、基于 CPU/Memory 的 HPA 扩缩容

基础指标自动扩缩：

# hpa-cpu.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rebate-bot-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rebate-bot
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

当平均 CPU 使用率 > 60%，HPA 自动增加 Pod 数量。

四、基于业务指标的 KEDA 扩缩容

HPA 无法感知消息队列积压，我们引入 KEDA 监听 Kafka Lag：

# keda-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rebate-bot-keda
spec:
  scaleTargetRef:
    name: rebate-bot
  pollingInterval: 15   # 每15秒检查一次
  cooldownPeriod: 120   # 缩容冷却时间
  minReplicaCount: 2
  maxReplicaCount: 100
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.juwatech.cn:9092
      consumerGroup: rebate-bot-group
      topic: wechat.message.incoming
      lagThreshold: "10"  # 每个分区积压>10条即扩容

同时监听 Redis List 长度（用于 Telegram Webhook 备份队列）：

  - type: redis
    metadata:
      address: redis.juwatech.cn:6379
      listName: telegram:pending:messages
      listLength: "50"  # 队列长度>50触发扩容

五、Java 应用适配优雅终止

确保 Pod 终止前处理完当前请求：

// juwatech.cn.bot.lifecycle.GracefulShutdownHandler
@Component
public class GracefulShutdownHandler implements HealthIndicator {

    private volatile boolean shuttingDown = false;

    @PreDestroy
    public void onShutdown() {
        shuttingDown = true;
        // 停止接受新消息
        messageConsumer.pause();

        // 等待最多30秒处理完积压任务
        long start = System.currentTimeMillis();
        while (messageConsumer.hasPendingTasks() && (System.currentTimeMillis() - start) < 30_000) {
            Thread.sleep(100);
        }
    }

    @Override
    public Health health() {
        if (shuttingDown) {
            return Health.down().withDetail("reason", "graceful_shutdown").build();
        }
        return Health.up().build();
    }
}

配合 Kubernetes preStop 钩子：

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 30"]

六、多可用区高可用部署

Deployment 分布在三个 AZ：

spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - rebate-bot
              topologyKey: topology.kubernetes.io/zone

Service 配置会话保持（针对长轮询场景）：

apiVersion: v1
kind: Service
metadata:
  name: rebate-bot-svc
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 300
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: rebate-bot

七、监控与告警

采集关键指标至 Prometheus：

// juwatech.cn.monitor.MetricsExporter
@GET
@Path("/metrics")
@Produces(MediaType.TEXT_PLAIN)
public String exportMetrics() {
    return """
        bot_active_pods %d
        bot_queue_lag %d
        bot_processing_time_ms{method="wechat"} %.2f
        """.formatted(
            getActivePodCount(),
            getKafkaLag("wechat.message.incoming"),
            getAvgProcessingTime("wechat")
        );
}

配置告警规则：

• kube_deployment_status_replicas_unavailable > 0 → 服务不可用
• keda_scaledObject_metric_lag > 1000 持续5分钟 → 扩容失败

本文著作权归 微赚淘客系统3.0 研发团队，转载请注明出处！