k8s安装，rocky9.7
·
常用命令：
#搜索yum源中wget的包 yum list wget --showduplicates | sort -r # 使用如下下载方式则会连同依赖一起下载 yum -y install --downloadonly --downloaddir /data/repo wget.x86_64 # 参数解析 # --downloadonly 指定仅下载 # --downloaddir /tmp/ 指定下载路径 # net-snmp.x86_64 下载包名 #安装 rpm -ivh wget-1.21.1-7.el9.x86_64.rpm
一、服务器准备工作
1、关闭防火墙、SELinux
systemctl stop firewalld && systemctl disable firewalld setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
2、机器间免密
2.1 生成秘钥
建议PaaS节点（一直点击回车键即可）：ssh-keygen
2.2 分发密钥
本机也要分发：ssh-copy-id root@192.168.x.x
注：第一次分发秘钥，系统可能需要确定是否分发，需要输入yes，然后需要输入密码
3、关闭swap交换分区
sed -ri 's/.*swap.*/#&/' /etc/fstab swapoff -a && sysctl -w vm.swappiness=0 free -m
4、chrony时间同步
略
5、配置ulimit
# 文件句柄 ulimit -SHn 65535 cat > /etc/security/limits.conf <<EOF * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF
6、设置主机名和hosts本地
#vi /etc/hosts cat >> /etc/hosts << EOF 192.168.0.170 master EOF
7、yum源安装
准备yum源，建议放在仓库机器，拷贝yum源文件夹repo、httpd到目标机器的/var/www/html/rocky目录（若无则需要创建该目录）下。
7.1 安装httpd、createrepo服务
#进入/var/www/html/httpd目录下 cd /var/www/html/rocky #安装httpd和createrepo cd httpd #执行安装命令 yum localinstall *.rpm -y #查看httpd启动情况 systemctl status httpd #若没有启动的话执行以下命令启动 systemctl start httpd && systemctl enable httpd
若出现如下图表示启动成功。
8、创建repodata文件
createrepo -pdo /var/www/html/rocky/repo/ /var/www/html/rocky/repo/ #添加权限 cd /var/www/html/rocky/repo/ chmod -Rf 777 *
9、使用yum源
cd /etc/yum.repos.d/ #备份原有yum源 mkdir bak mv *.repo bak cat > /etc/yum.repos.d/rocky.repo << EOF [localrepo] name=centos baseurl=http://192.168.0.170/rocky/repo/ enabled=1 gpgcheck=0 EOF yum clean all && yum makecache yum list
10、安装docker
#进入docker的yum源文件夹 rpm -ivh *.rpm #若搭建yum源后，可直接执行 yum -y install docker-ce
10.1 修改docker.service containerd.service
vi /usr/lib/systemd/system/containerd.service #将LimitNOFILE=infinity修改为LimitNOFILE=1048576
10.2 设置仓库地址，修改cgroup
#设置仓库地址，修改cgroup
cat > /etc/docker/daemon.json << EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"insecure-registries":["0.0.0.0/0"],
"max-concurrent-downloads": 10,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"data-root": "/var/lib/docker"
}
EOF
#重载docker配置，重启docker
systemctl enable docker.socket && systemctl daemon-reload && systemctl enable docker && systemctl restart docker
10.3 建立镜像挂载路径并启动仓库
#复制仓库文件并加载 docker load < registry.tar mkdir -p /home/registry-data docker run -d -p 5000:5000 -v /home/registry-data:/var/lib/registry --restart=always --name registry registry:latest
11、安装cri-docker：
为了继续使用Docker作为容器运行时，需要安装cri-docker将Docker作为容器运行时使用
11.1 解压cri-docker安装包后拷贝到/usr/bin/目录下，并赋权
tar xf cri-dockerd-0.3.0.amd64.tgz cp cri-dockerd/cri-dockerd /usr/bin/ chmod +x /usr/bin/cri-dockerd
11.2 修改service配置文件
cp cri-docker.service /usr/lib/systemd/system/cri-docker.service vi /usr/lib/systemd/system/cri-docker.service 修改“ExecStart=/usr/bin/cri-dockerd...里的镜像修改为自己镜像仓库对应的镜像”
[Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=192.168.0.170:5000/pause:3.7 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target
11.3 创建cri-docker的socket文件
cp cri-docker.socket /usr/lib/systemd/system/
11.4 启动服务并检查状态
systemctl daemon-reload systemctl enable cri-docker && systemctl start cri-docker systemctl is-active cri-docker cri-dockerd --version
二、部署工作（高可用集群部署-可选）
1. CA证书（任意节点）
1.1 安装cfssl
cp cfssl /usr/bin/cfssl cp cfssljson /usr/bin/cfssljson chmod +x /usr/bin/cfssl /usr/bin/cfssljson # 验证，会显示版本等信息 cfssl version #Version: 1.6.3 #Runtime: go1.18
1.2 生成根证书
mkdir -p /data/pki/
cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "seven"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
#生成ca-config.json
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
1.3 查看生成的证书
我们最终想要的就是ca-key.pem和ca.pem，一个秘钥，一个证书
ls #ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
1.4 分发ca证书
mkdir -p /etc/kubernetes/pki/ #分发至当前节点 cp ca*.pem /etc/kubernetes/pki/ #分发至其他节点 scp ca*.pem <user>@<node-ip>:/etc/kubernetes/pki/
2. 部署ETCD集群
etcd集群部署的时候包含主节点，强烈建议部署etcd的个数为奇数如：3,5,7等；如无特殊要求，3个即可满足
2.1 解压并分发
tar xf etcd-v3.5.6-linux-amd64.tar.gz cd etcd-v3.5.6-linux-amd64 cp etcd etcdctl /usr/bin/
2.2 生成证书与私钥
（需要修改etcd-csr.json的ip地址，改为对应的etcd节点IP）
mkdir -p /data/pki/etcd
cd /data/pki/etcd
cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.0.170"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
#查看生成的证书
ls
#etcd.csr etcd-csr.json etcd-key.pem etcd.pem
2.3 分发证书
#分发至当前节点 cp etcd*.pem /etc/kubernetes/pki/ #分发至其他节点 scp etcd*.pem <user>@<node-ip>:/etc/kubernetes/pki/
2.4 配置service文件、etcd.config.yml
cat > /etc/systemd/system/etcd.service << EOF [Unit] Description=Etcd Service Documentation=https://coreos.com/etcd/docs/latest/ After=network.target [Service] Type=notify ExecStart=/usr/bin/etcd --config-file=/etc/kubernetes/etcd.config.yml Restart=on-failure RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target Alias=etcd3.service EOF cat > /etc/kubernetes/etcd.config.yml << EOF name: 'etcd1' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://192.168.0.170:2380' listen-client-urls: 'https://192.168.0.170:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://192.168.0.170:2380' advertise-client-urls: 'https://192.168.0.170:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'etcd1=https://192.168.0.170:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd.pem' key-file: '/etc/kubernetes/pki/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd.pem' key-file: '/etc/kubernetes/pki/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF
2.5 创建数据和工作目录
mkdir -p /var/lib/etcd
2.6 启动服务并设置开机自启
systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd journalctl -f -u etcd
2.7 验证启动状态
#命令一（老版本etcd）： etcdctl --endpoint=https://192.168.0.170:2379 --ca-file=/etc/kubernetes/pki/ca.pem --cert-file=/etc/kubernetes/pki/etcd.pem --key-file=/etc/kubernetes/pki/etcd-key.pem member list #命令二（新版本etcd）： etcdctl --endpoints=https://192.168.0.170:2379 --cacert=/etc/kubernetes/pki/ca.pem --cert=/etc/kubernetes/pki/etcd.pem --key=/etc/kubernetes/pki/etcd-key.pem member list
3、部署kube-apiserver服务
3.1 解压并分发
tar xf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
chmod +x kube*
cp kube{let,ctl,-apiserver,-scheduler,-controller-manager,-proxy} /usr/bin/
kube-apiserver --version
3.2 生成证书和私钥
mkdir -p /data/pki/apiserver
cd /data/pki/apiserver
cat > /data/pki/apiserver/kubernetes-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.0.170",
"10.112.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
#查看生成的证书
ls
#kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
3.3 分发证书
#分发至当前节点 cp kubernetes*.pem /etc/kubernetes/pki/ #分发至其他节点 scp kubernetes*.pem <user>@<node-ip>:/etc/kubernetes/pki/
3.4 创建SA账号
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048 openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
3.5 修改service文件
vi etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/bin/kube-apiserver \
--v=2 \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=192.168.0.170 \
--service-cluster-ip-range=10.112.0.0/12 \
--service-node-port-range=30000-52767 \
--etcd-servers=https://192.168.0.170:2379 \
--etcd-cafile=/etc/kubernetes/pki/ca.pem \
--etcd-certfile=/etc/kubernetes/pki/etcd.pem \
--etcd-keyfile=/etc/kubernetes/pki/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/kubernetes.pem \
--tls-private-key-file=/etc/kubernetes/pki/kubernetes-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/kubernetes.pem \
--kubelet-client-key=/etc/kubernetes/pki/kubernetes-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/kubernetes.pem \
--proxy-client-key-file=/etc/kubernetes/pki/kubernetes-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--token-auth-file=/etc/kubernetes/token.csv
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
（修改证书及etcd、token文件等配置信息） 【token生成方式：（token.csv文件中的token是在主节点上生成的，本部署文档提供token.csv模板，只需替换token.csv文件第一行第一个token串即可。在主节点执行 head -c 16 /dev/urandom | od -An -t x | tr -d ' '生成字符串）】
#在主节点执行 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' #74a48e1267fa4cc7111c2a6d1c21ad75 cat > /etc/kubernetes/token.csv << EOF 74a48e1267fa4cc7111c2a6d1c21ad75,kubelet-bootstrap,10001,"system:kubelet-bootstrap" 57c41729effa3ab523f29dc494d8bc4e,admin,1 EOF
3.6 创建log日志路径及静态pod自动扫描路径
mkdir -p /var/log/kubernetes mkdir -p /etc/kubernetes/manifests/
3.7 启动服务并检查状态
systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver systemctl status kube-apiserver
4、部署kubectl工具（这里创建具有最高权限的 admin 证书）
4.1 创建客户端证书、私钥
mkdir -p /data/pki/admin
cd /data/pki/admin
cat > /data/pki/admin/admin-csr.json << EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
4.2 设置集群参数
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.0.170:6443 \ --kubeconfig=admin.kubeconfig
kubectl config set-credentials kubernetes-admin \ --client-certificate=admin.pem \ --client-key=admin-key.pem \ --embed-certs=true \ --kubeconfig=admin.kubeconfig
kubectl config set-context kubernetes-admin@kubernetes \ --cluster=kubernetes \ --user=kubernetes-admin \ --kubeconfig=admin.kubeconfig
kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=admin.kubeconfig
4.3 创建路径并分发config文件
mkdir /root/.kube cp admin.kubeconfig ~/.kube/config cp admin*.pem /etc/kubernetes/pki/
4.4 授予访问k8s的API的权限
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=cluster-admin --user=kubernetes
4.5 创建token访问kubernetes-api的权限绑定
kubectl create clusterrolebinding kubelet-bootstrap-admin --clusterrole=cluster-admin --user=kubelet-bootstrap
4.6 查看集群信息
kubectl cluster-info kubectl get all --all-namespaces kubectl get componentstatuses
5、部署controller-manager服务
5.1 创建证书和私钥
mkdir -p /data/pki/controller-manager
cd /data/pki/controller-manager
cat > /data/pki/controller-manager/controller-manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.0.170"
],
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-controller-manager",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes controller-manager-csr.json | cfssljson -bare controller-manager
5.2 分发证书、密钥
#分发至当前节点 cp controller-manager*.pem /etc/kubernetes/pki/ #分发至其他节点 scp controller-manager*.pem <user>@<node-ip>:/etc/kubernetes/pki/
5.3 创建controller-manager的kubeconfig
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.0.170:6443 \ --kubeconfig=controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \ --client-certificate=controller-manager.pem \ --client-key=controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=controller-manager.kubeconfig
5.4 分发controller-manager.kubeconfig
#分发至当前节点 cp controller-manager.kubeconfig /etc/kubernetes/ #分发至其他节点 scp controller-manager.kubeconfig <user>@<node-ip>:/etc/kubernetes/pki/
5.5 配置service文件
vi /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/bin/kube-controller-manager \
--v=2 \
--bind-address=192.168.0.170 \
--root-ca-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--pod-eviction-timeout=2m0s \
--controllers=*,bootstrapsigner,tokencleaner \
--allocate-node-cidrs=true \
--service-cluster-ip-range=10.112.0.0/12 \
--cluster-cidr=172.16.0.0/12 \
--requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem
--node-cidr-mask-size=24
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
5.6 启动服务并检查
systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager systemctl status kube-controller-manager
6、部署kube-scheduler
6.1 生成证书和私钥
mkdir -p /data/pki/scheduler
cd /data/pki/scheduler
cat > /data/pki/scheduler/scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"192.168.0.170"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-scheduler",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes scheduler-csr.json | cfssljson -bare kube-scheduler
6.2 创建scheduler的kubeconfig
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.0.170:6443 \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \ --client-certificate=kube-scheduler.pem \ --client-key=kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
6.3 分发kube-scheduler.kubeconfig
#分发至当前节点 cp kube-scheduler.kubeconfig /etc/kubernetes/ #分发至其他节点 scp kube-scheduler.kubeconfig <user>@<node-ip>:/etc/kubernetes/
6.4 配置service
vi /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/bin/kube-scheduler \
--v=2 \
--leader-elect=true \
--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
6.5 启动服务
systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler systemctl status kube-scheduler
7、部署kubelet服务（worker节点）
7.1 设置集群参数
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.0.170:6443 \ --kubeconfig=kubelet-bootstrap.kubeconfig
设置客户端认证参数（token为前面token.csv中第一行的token）
kubectl config set-credentials kubelet-bootstrap --token=74a48e1267fa4cc7111c2a6d1c21ad75 --kubeconfig=kubelet-bootstrap.kubeconfig
设置上下文参数
kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=kubelet-bootstrap.kubeconfig
设置默认上下文
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
7.2 分发kubelet-bootstrap.kubeconfig
#分发至当前节点 cp kubelet-bootstrap.kubeconfig /etc/kubernetes/kubelet-bootstrap.kubeconfig #分发至其他节点 scp kubelet-bootstrap.kubeconfig <user>@<node-ip>:/etc/kubernetes/kubelet-bootstrap.kubeconfig
7.3 将主节点的pki分配至工作节点，仅工作节点执行
#先在worker节点上创建目录（每个节点都要创建） mkdir -p /etc/kubernetes/pki # 把ca分发到每个worker节点 cp pki/ca.pem /etc/kubernetes/pki/
7.4 配置service文件、kubelet-conf.yml文件
vi /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/bin/kubelet \
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
--hostname-override=192.168.0.170 \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet-conf.yml \
--container-runtime-endpoint=unix:///run/cri-dockerd.sock \
--node-labels=node.kubernetes.io/node=
[Install]
WantedBy=multi-user.target
vi /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.112.0.2
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
7.5 预先下载需要的镜像
docker load < pause.tar docker tag registry.cn-hangzhou.aliyuncs.com/chenby/pause:3.7 192.168.0.170:5000/pause:3.7
7.6 启动服务
kublet 启动时查找配置的 --kubeletconfig 文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证（事先使用 kubeadm 创建的 token），认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，这就是Bootstrap Token Auth。
# bootstrap附权 kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap #在允许get nodes之前应该先赋予kube-controller-manager 管理node的权限，所以，需要先执行下面这句： #kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=cluster-admin --user=system:kube-controller-manager
systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet systemctl status kubelet
8. 部署kube-proxy（worker节点）
8.1 创建证书和私钥
mkdir -p /data/pki/kube-proxy
cd /data/pki/kube-proxy
cat > /data/pki/kube-proxy/kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "seven"
}
]
}
EOF
cfssl gencert -ca=/data/pki/ca.pem \
-ca-key=/data/pki/ca-key.pem \
-config=/data/pki/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
8.2 创建和分发 kubeconfig 文件
# 创建kube-proxy.kubeconfig kubectl config set-cluster kubernetes \ --certificate-authority=/data/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.0.170:6443 \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=kube-proxy.pem \ --client-key=kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig # 分发kube-proxy.kubeconfig #分发至当前节点 cp kube-proxy.kubeconfig /etc/kubernetes/ #分发至其他节点 scp kube-proxy.kubeconfig <user>@<node-ip>:/etc/kubernetes/
8.3 分发kube-proxy.config
#修改配置文件 vi /etc/kubernetes/kube-proxy.yaml apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: "" burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig qps: 5 clusterCIDR: 172.16.0.0/12 configSyncPeriod: 15m0s conntrack: max: null maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s enableProfiling: false healthzBindAddress: 0.0.0.0:10256 hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30s ipvs: masqueradeAll: true minSyncPeriod: 5s scheduler: "rr" syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: "ipvs" nodePortAddresses: null oomScoreAdj: -999 portRange: "" udpIdleTimeout: 250ms
8.4 分发kube-proxy服务文件
#修改服务文件 vi /etc/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Kube Proxy Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/bin/kube-proxy \ --config=/etc/kubernetes/kube-proxy.yaml \ --v=2 Restart=always RestartSec=10s [Install] WantedBy=multi-user.target
8.5 启动服务
# 创建依赖目录 mkdir -p /var/lib/kube-proxy && mkdir -p /var/log/kubernetes # 启动服务 systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy # 查看状态 systemctl status kube-proxy # 查看日志 journalctl -f -u kube-proxy
9. 部署CNI插件 - calico
9.1 配置NetworkManager
确保Calico可以管理主机上的cali和tunl接口。 如果主机上存在NetworkManager，请按照下面方法配置NetworkManager。
NetworkManager管理默认网络命名空间中接口的路由表的功能，可能会干扰Calico正确处理网络路由的能力。
创建一个配置文件/etc/NetworkManager/conf.d/calico.conf，来制止这种干扰：
cat > /etc/NetworkManager/conf.d/calico.conf << EOF [keyfile] unmanaged-devices=interface-name:cali*;interface-name:tunl* EOF
9.2 安装部署
我们使用calico官方的安装方式来部署。
# 创建目录（在配置了kubectl的节点上执行）
mkdir -p /etc/kubernetes/addons
images=$(docker images | awk '$1 ~ /^192.168.0.170:5000\// {print $1 ":" $2}')
for image in ${images[@]} ; do
docker push $image
#结束 for 循环的标志
done
#修改yaml文件
#修改镜像
#修改4618行，添加环境变量，多网卡名用，隔开
- name: IP_AUTODETECTION_METHOD
value: interface=ens160
# 上传calico配置到配置好kubectl的节点（一个节点即可）
scp target/addons/calico* <user>@<node-ip>:/etc/kubernetes/addons/
# 部署calico
kubectl create -f /etc/kubernetes/addons/calico-rbac-kdd.yaml
kubectl create -f /etc/kubernetes/addons/calico.yaml
# 查看状态
kubectl get pods -n kube-system
10. 部署DNS插件 - coredns
# 上传配置文件 scp target/addons/coredns.yaml <user>@<node-ip>:/etc/kubernetes/addons/ # 部署coredns kubectl create -f /etc/kubernetes/addons/coredns.yaml
11. 部署dashboard
修改yaml文件的镜像信息，并绑定serviceaccount
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
12. 部署metrics-server
修改yaml文件镜像信息，并绑定涉及node权限问题
kubectl create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes
13. 其他监控组件正常部署即可，无影响
（1） Prometheus修改配置文件，提供修改模板
（2） node-exporter无影响
（3） state-metrics无影响
14. 修改kube-controller-manager、kube-scheduler服务的metrics端点暴露
方法：修改kube-controller-manager.service 、kube-scheduler.service，
添加“--authorization-always-paths=/metrics”，
修改prometheus.yml文件，添加https方式访问，
设置“insecure_skip_verify: true”参数即可，
证书无所谓，只要存在即可。
Kubeadm:启用安全 kube-scheduler 和 kube-controller-manager 端口进行健康检查。对于 kube-scheduler 是 10251,变成 10259。对于 kube-controller-manager 是 10252,变成 10257。
汇聚全球AI编程工具，助力开发者即刻编程。
更多推荐
7
0- 0
已为社区贡献1条内容
所有评论(0)