Kubernetes 极速安装与排障实战手册 (v1.35.2)

2018_like菜

390人浏览 · 2026-03-17 09:59:53

2018_like菜 · 2026-03-17 09:59:53 发布

本次ubuntu22系统

# **Kubernetes 高可用集群极速安装与排障实战手册 (v1.35.2)**

## 第一章：基础安装篇

### 1.1 所有节点基础环境
```bash
# 关闭 swap
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

# 加载内核模块
cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
nf_conntrack
EOF
modprobe overlay br_netfilter nf_conntrack

# 配置系统参数
cat > /etc/sysctl.d/99-kubernetes.conf << 'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
kernel.panic = 10
kernel.panic_on_oops = 1
vm.overcommit_memory = 1
vm.swappiness = 0
net.netfilter.nf_conntrack_max = 1048576
net.core.somaxconn = 32768
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 60999
fs.inotify.max_user_watches = 524288
fs.file-max = 1048576
EOF
sysctl -p /etc/sysctl.d/99-kubernetes.conf
```

### 1.2 配置 containerd（关键：镜像加速）
```bash
# 安装 containerd
apt-get update && apt-get install -y containerd

# 生成配置并修改
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.8"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10"#' /etc/containerd/config.toml

# 能下载就不替换，添加镜像加速器【注意下载不下镜像就配置，下面的加速都是这样，换源加速】
cat >> /etc/containerd/config.toml << EOF

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://docker.m.daocloud.io", "https://docker.xuanyuan.me"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
endpoint = ["https://quay.m.daocloud.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
endpoint = ["https://k8s.m.daocloud.io"]
EOF

# 重启并启用
systemctl restart containerd && systemctl enable containerd
```

### 1.3 安装 kubeadm/kubelet/kubectl
```bash
# 配置源
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update

# 安装并锁定版本
apt-get install -y kubelet=1.35.2-1.1 kubeadm=1.35.2-1.1 kubectl=1.35.2-1.1
apt-mark hold kubelet kubeadm kubectl
```

### 1.4 Master 节点初始化
```bash
kubeadm init \
--apiserver-advertise-address=192.168.1.38 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16 \
--kubernetes-version v1.35.2

# 配置 kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 保存 join 命令
kubeadm token create --print-join-command > /root/kubeadm-join-command.txt
```

---

## 第二章：网络插件（Calico）安装与排障

### 2.1 下载并修改 Calico 配置
```bash
# 下载
wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.0/manifests/calico.yaml

# 修改 Pod CIDR（必须和初始化一致）
sed -i '/CALICO_IPV4POOL_CIDR/,+1 s/# //' calico.yaml
sed -i 's/192.168.0.0\/16/10.244.0.0\/16/' calico.yaml

# 修改镜像地址为国内源
sed -i 's?docker.io/calico?quay.io/calico?g' calico.yaml
# 注意：quay.io 也需要加速，已在 containerd 中配置

# 应用
kubectl apply -f calico.yaml
```

### 2.2 Calico 常见问题修复
```bash
# 问题1：install-cni 容器一直失败
# 解决：在 Worker 节点清理 CNI 目录
rm -rf /etc/cni/net.d/*
rm -rf /opt/cni/bin/*
systemctl restart containerd

# 问题2：token 缺失导致 Unauthorized
kubectl delete secret -n kube-system calico-cni-plugin-token 2>/dev/null
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: calico-cni-plugin-token
namespace: kube-system
annotations:
kubernetes.io/service-account.name: calico-cni-plugin
type: kubernetes.io/service-account-token
EOF

# 问题3：BGP not established（对端节点 Calico 没起来）
kubectl delete pod -n kube-system -l k8s-app=calico-node
```

---

## 第三章：Worker 节点加入

### 3.1 Worker 节点初始化脚本
```bash
# 在 Worker 节点执行
cat > worker-init.sh << 'EOF'
#!/bin/bash
# 基础环境（同 Master）
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
modprobe overlay br_netfilter nf_conntrack
cat > /etc/sysctl.d/99-kubernetes.conf << 'EOL'
# ... 复制上面的 sysctl 配置 ...
EOL
sysctl -p /etc/sysctl.d/99-kubernetes.conf

# 安装 containerd 并配置加速器（同 Master）
apt-get update && apt-get install -y containerd
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.8"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10"#' /etc/containerd/config.toml
# 添加 mirrors 配置（同上）
systemctl restart containerd && systemctl enable containerd

# 安装 kubelet/kubeadm
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet=1.35.2-1.1 kubeadm=1.35.2-1.1
apt-mark hold kubelet kubeadm
EOF

chmod +x worker-init.sh
./worker-init.sh

# 加入集群
kubeadm join 192.168.1.38:6443 --token <your-token> --discovery-token-ca-cert-hash sha256:<your-hash>
```

---

## 第四章：控制平面组件排障（重启后的噩梦解决）

### 4.1 核心问题：`probe-port` 变量未替换
这是重启后 etcd、apiserver、controller-manager、scheduler 卡在 `0/1` 的根本原因。

```bash
# 检查是否有 probe-port 变量
grep -r "probe-port" /etc/kubernetes/manifests/

# 一键替换所有 probe-port 为实际端口
sed -i 's/port: probe-port/port: 2381/g' /etc/kubernetes/manifests/etcd.yaml # etcd 端口 2381
sed -i 's/port: probe-port/port: 10257/g' /etc/kubernetes/manifests/kube-controller-manager.yaml # controller-manager 端口 10257
sed -i 's/port: probe-port/port: 10259/g' /etc/kubernetes/manifests/kube-scheduler.yaml # scheduler 端口 10259
# apiserver 通常没有 probe-port 问题，如有则改 10259

# 修改后无需重启，kubelet 自动生效
```

### 4.2 etcd 健康检查
```bash
# 手动测试 etcd 健康端口
curl http://127.0.0.1:2381/readyz

# 测试 etcd 业务端口
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key \
endpoint health
```

### 4.3 容器名称冲突（CreateContainerError）
```bash
# 清理残留容器
crictl ps -a | grep -E "kube-controller-manager|kube-scheduler" | grep Exited | awk '{print $1}' | xargs -r crictl rm

# 重启 kubelet
systemctl restart kubelet
```

### 4.4 一键重启所有控制平面组件
```bash
# 移动清单文件触发重建
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
mv /etc/kubernetes/manifests/${component}.yaml /tmp/
sleep 5
mv /tmp/${component}.yaml /etc/kubernetes/manifests/
done
systemctl restart kubelet
```

---

## 第五章：重启后一键恢复脚本

重启后，先重启worker后master 只有节点一个pod运行不起来

kubectl delete pod -n kube-system calico-node-thfbj 例如这个是节点pod 直接删除

kubectl get pods -n kube-system -w | grep calico-node 再观察

创建 `/root/k8s-recover.sh`，包含所有经验：

```bash
#!/bin/bash
echo "========================================="
echo "Kubernetes 重启后一键恢复脚本"
echo "========================================="

echo "1. 等待 60 秒让系统启动..."
sleep 60

echo "2. 检查 probe-port 变量"
if grep -r "probe-port" /etc/kubernetes/manifests/; then
echo "发现 probe-port 变量，正在修复..."
sed -i 's/port: probe-port/port: 2381/g' /etc/kubernetes/manifests/etcd.yaml
sed -i 's/port: probe-port/port: 10257/g' /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -i 's/port: probe-port/port: 10259/g' /etc/kubernetes/manifests/kube-scheduler.yaml
fi

echo "3. 清理残留容器"
crictl ps -a | grep -E "kube-controller-manager|kube-scheduler" | grep Exited | awk '{print $1}' | xargs -r crictl rm

echo "4. 重启 kubelet"
systemctl restart kubelet
sleep 30

echo "5. 检查 etcd 健康"
curl -s http://127.0.0.1:2381/readyz && echo " etcd 健康" || echo " etcd 异常"

echo "6. 重启 Calico 和 kube-proxy"
kubectl delete pod -n kube-system -l k8s-app=calico-node 2>/dev/null
kubectl delete pod -n kube-system -l k8s-app=kube-proxy 2>/dev/null
kubectl delete pod -n kube-system -l k8s-app=kube-dns 2>/dev/null

echo "7. 等待 60 秒..."
sleep 60

echo "8. 最终状态："
kubectl get pods -A
kubectl get nodes

echo "========================================="
echo "恢复脚本执行完毕！"
```

```bash
chmod +x /root/k8s-recover.sh
```

---

## 第六章：日常运维与验证

### 6.1 快速验证集群
```bash
# 测试 DNS
kubectl run test-dns \
--image=docker.m.daocloud.io/busybox:latest \
--rm -it \
--restart=Never \
-- nslookup kubernetes.default.svc.cluster.local

# 测试调度
kubectl run test-nginx --image=nginx --restart=Never --rm -it -- echo "ok"

# 查看所有 Pod
kubectl get pods -A -o wide
```

### 6.2 节点维护
```bash
# 正确重启顺序：先 Worker，后 Master
# 在 Worker 节点
reboot
# 等待 3 分钟后，在 Master 节点
reboot

# 重启后执行恢复脚本
/root/k8s-recover.sh
```

### 6.3 备份重要数据
```bash
# 备份证书
cp -r /etc/kubernetes/pki /root/k8s-backup-$(date +%Y%m%d)

# 备份静态 Pod 清单
cp -r /etc/kubernetes/manifests /root/manifests-backup-$(date +%Y%m%d)
```

---

## 第七章：问题速查表

| 现象 | 错误日志关键词 | 快速解决 |
|:---|:---|:---|
| etcd 0/1 | `probe-port` | `sed -i 's/port: probe-port/port: 2381/g' /etc/kubernetes/manifests/etcd.yaml` |
| controller-manager 0/1 | `probe-port` | `sed -i 's/port: probe-port/port: 10257/g' /etc/kubernetes/manifests/kube-controller-manager.yaml` |
| scheduler 0/1 | `probe-port` | `sed -i 's/port: probe-port/port: 10259/g' /etc/kubernetes/manifests/kube-scheduler.yaml` |
| Calico Unauthorized | `Unauthorized` | 重新创建 token（见 2.2） |
| Calico BGP not established | `BGP not established with` | 删除对端节点 Calico Pod |
| CreateContainerError | `name is reserved for` | `crictl rm` 清理残留容器 |
| 镜像拉取超时 | `i/o timeout` | 检查 containerd 加速器配置 |

---

这份手册包含了我们这次所有实战经验，你只需保存为 `.md` 文件，以后无论是新装集群还是处理重启故障，都能快速解决。