harbor-v2.14.0高可用部署基于Ubuntu22.04
·
harbor是VMware公司开源的一款企业级镜像仓库，底层基于docker-compose来管理harbor服务
docker的私有仓库用起来太麻烦我们使用harbor来管理镜像
接下来学习一下如何搭建部署一个高可用企业级的harbot私有仓库的部署过程
我的系统是Ubuntu22.04版本
harbor的高可用部署
1. 下载harbor软件包
wget https://github.com/goharbor/harbor/releases/download/v2.14.0/harbor-offline-installer-v2.14.0.tgz
2. 解压软件包
tar xf harbor-offline-installer-v2.14.0.tgz -C /usr/local/
3. 修改harbor的配置文件
cd /usr/local/harbor/
cp harbor.yml{.tmpl,}
vim harbor.yml
...
#hostname: reg.mydomain.com
hostname: 10.0.250.4
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https: #可注释
# # https port for harbor, default is 443
port: 443 #可注释
# # The path of cert and key files for nginx
certificate: /etc/aliyum/aixxw/aixxw.xin.pem #可注释
private_key: /etc/aliyum/aixxw/aixxw.xin.key #可注释
# enable strong ssl ciphers (default: false)
# strong_ssl_ciphers: false
...
harbor_admin_password: 123
...
data_volume: /var/lib/harbor
...
提示：
没有部署docker环境的请先部署链接如下：https://blog.csdn.net/weixin_45049740/article/details/157514762?sharetype=blogdetail&sharerId=157514762&sharerefer=PC&sharesource=weixin_45049740&spm=1011.2480.3001.8118
4. 安装harbor服务
[root@master02 harbor]# ./install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 20.10.24
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 2.23.0
[Step 2]: loading Harbor images ...
5dd32317cf96: Loading layer 40.82MB/40.82MB
e959dc37d186: Loading layer 16.49MB/16.49MB
2cd016989b45: Loading layer 175.3MB/175.3MB
c.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
copy /data/secret/tls/harbor_internal_ca.crt to shared trust ca dir as name harbor_internal_ca.crt ...
ca file /hostfs/data/secret/tls/harbor_internal_ca.crt is not exist
copy to shared trust ca dir as name storage_ca_bundle.crt ...
copy None to shared trust ca dir as name redis_tls_ca.crt ...
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
...
....
Note: stopping existing Harbor instance ...
[Step 5]: starting Harbor ...
[+] Building 0.0s (0/0) docker:default
[+] Running 10/10
✔ Network harbor_harbor Created 0.2s
✔ Container harbor-log Started 0.1s
✔ Container harbor-portal Started 0.1s
✔ Container redis Started 0.2s
✔ Container registryctl Started 0.1s
✔ Container registry Started 0.1s
✔ Container harbor-db Started 0.1s
✔ Container harbor-core Started 0.0s
✔ Container harbor-jobservice Started 0.1s
✔ Container nginx Started 0.1s
✔ ----Harbor has been installed and started successfully.----
[root@master02 harbor]# docker-compose ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
harbor-core goharbor/harbor-core:v2.14.0 "/harbor/entrypoint.…" core 4 minutes ago Up 4 minutes (healthy)
harbor-db goharbor/harbor-db:v2.14.0 "/docker-entrypoint.…" postgresql 4 minutes ago Up 4 minutes (healthy)
harbor-jobservice goharbor/harbor-jobservice:v2.14.0 "/harbor/entrypoint.…" jobservice 4 minutes ago Up 4 minutes (healthy)
harbor-log goharbor/harbor-log:v2.14.0 "/bin/sh -c /usr/loc…" log 4 minutes ago Up 4 minutes (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal goharbor/harbor-portal:v2.14.0 "nginx -g 'daemon of…" portal 4 minutes ago Up 4 minutes (healthy)
nginx goharbor/nginx-photon:v2.14.0 "nginx -g 'daemon of…" proxy 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:80->8080/tcp
redis goharbor/redis-photon:v2.14.0 "redis-server /etc/r…" redis 4 minutes ago Up 4 minutes (healthy)
registry goharbor/registry-photon:v2.14.0 "/home/harbor/entryp…" registry 4 minutes ago Up 4 minutes (healthy)
registryctl goharbor/harbor-registryctl:v2.14.0 "/home/harbor/start.…" registryctl 4 minutes ago Up 4 minutes (healthy)
[root@master02 harbor]#
访问UI页面查看
80： http://10.0.250.4/harbor/projects
内网 配置hosts转发
443： https://hb.aixxw.xin/harbor/projects
关于自签证书基于k8s集群cfssl签发CAz证书
================================分割线===================================
链接：
https://blog.csdn.net/weixin_45049740/article/details/156514328?spm=1001.2014.3001.5502
[root@master01 ~]# cd /k8s/certs/pki
[root@master01 pki]# ls
admin-csr.json etcd-ca-csr.json k8s-ca-csr.json
apiserver-csr.json etcd-csr.json 's -ntl | grep 10257'
bootstrap-secret.yaml front-proxy-ca-csr.json 's -ntl | grep 10259'
ca-config.json front-proxy-client-csr.json scheduler-csr.json
controller-manager-csr.json k8s-ca-config.json
[root@master01 pki]# vim harbor-csr.json
{
"CN": "10.0.250.4",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "Harbor",
"OU": "Harbor-manual"
}
]
}
// 签发Harbor证书
[root@master01 pki]#
cfssl gencert \
-ca=/k8s/certs/kubernetes/k8s-ca.pem \
-ca-key=/k8s/certs/kubernetes/k8s-ca-key.pem \
-config=k8s-ca-config.json \
--hostname=hb.aixxw.xin,10.0.250.4,10.0.250.5,10.0.250.2 \
--profile=kubernetes \
harbor-csr.json | cfssljson -bare /k8s/certs/kubernetes/harbor
[root@master01 pki]# ll /k8s/certs/kubernetes/harbor*
-rw------- 1 root root 1679 Jan 29 21:46 /k8s/certs/kubernetes/harbor-key.pem
-rw-r--r-- 1 root root 1078 Jan 29 21:46 /k8s/certs/kubernetes/harbor.csr
-rw-r--r-- 1 root root 1476 Jan 29 21:46 /k8s/certs/kubernetes/harbor.pem
[root@master01 pki]# scp /k8s/certs/kubernetes/harbor.pem 10.0.250.4://etc/aliyum/aixxw/aixxw.xin.pem
harbor.pem 100% 1476 2.2MB/s 00:00
[root@master01 pki]# scp /k8s/certs/kubernetes/harbor-key.pem 10.0.250.4://etc/aliyum/aixxw/aixxw.xin.key
harbor-key.pem
[root@master01 pki]# cd /k8s/certs/kubernetes/
[root@master01 kubernetes]# ls
admin-key.pem controller-manager.pem harbor.csr sa.key
admin.csr front-proxy-ca-key.pem harbor.pem sa.pub
admin.pem front-proxy-ca.csr k8s-ca-key.pem scheduler-key.pem
apiserver-key.pem front-proxy-ca.pem k8s-ca.csr scheduler.csr
apiserver.csr front-proxy-client-key.pem k8s-ca.pem scheduler.pem
apiserver.pem front-proxy-client.csr kube-proxy-key.pem
controller-manager-key.pem front-proxy-client.pem kube-proxy.csr
controller-manager.csr harbor-key.pem kube-proxy.pem
[root@master01 kubernetes]# cp /k8s/certs/kubernetes/k8s-ca.pem /usr/local/share/ca-certificates/k8s-root-ca.crt
[root@master01 kubernetes]# scp /k8s/certs/kubernetes/k8s-ca.pem 10.0.250.4://usr/local/share/ca-certificates/k8s-root-ca.crt
k8s-ca.pem 100% 1363 1.9MB/s 00:00
[root@master01 kubernetes]# scp /k8s/certs/kubernetes/k8s-ca.pem 10.0.250.5://usr/local/share/ca-certificates/k8s-root-ca.crt
k8s-ca.pem 100% 1363 2.9MB/s 00:00
[root@master01 kubernetes]# scp /k8s/certs/kubernetes/k8s-ca.pem 10.0.250.6://usr/local/share/ca-certificates/k8s-root-ca.crt
k8s-ca.pem 100% 1363 677.1KB/s 00:00
[root@master01 kubernetes]# scp /k8s/certs/kubernetes/k8s-ca.pem 10.0.250.7://usr/local/share/ca-certificates/k8s-root-ca.crt
k8s-ca.pem 100% 1363 3.4MB/s 00:00
[root@master01 kubernetes]# scp /k8s/certs/kubernetes/k8s-ca.pem 10.0.250.2://usr/local/share/ca-certificates/k8s-root-ca.crt
更新系统 CA 信任库
update-ca-certificates
[root@master01 kubernetes]# cat /etc/hosts
127.0.1.1 sertw3u8sr18n.local sertw3u8sr18n
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.0.250.2 apiserver-lb
10.0.250.3 master01
10.0.250.4 master02 hb.aixxw.xin
10.0.250.5 master03
10.0.250.6 worker01
10.0.250.7 worker02
重启服务
[root@master02 aixxw]# cd /usr/local/harbor/
[root@master02 harbor]# docker-compose down
[+] Running 10/10
✔ Container registryctl Removed 0.9s
✔ Container nginx Removed 0.9s
✔ Container harbor-jobservice Removed 0.8s
✔ Container harbor-portal Removed 0.6s
✔ Container harbor-core Removed 0.8s
✔ Container registry Removed 0.9s
✔ Container redis Removed 1.0s
✔ Container harbor-db Removed 1.1s
✔ Container harbor-log Removed 10.5s
✔ Network harbor_harbor Removed 0.1s
[root@master02 harbor]# ./prepare
prepare base dir is set to /usr/local/harbor
Clearing the configuration file: /config/log/rsyslog_docker.conf
...
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[root@master02 harbor]# docker-compose up -d
[+] Building 0.0s (0/0) docker:default
[+] Running 10/10
✔ Network harbor_harbor Created 0.2s
✔ Container harbor-log Started 0.0s
✔ Container harbor-portal Started 0.2s
✔ Container redis Started 0.2s
✔ Container harbor-db Started 0.2s
✔ Container registryctl Started 0.2s
✔ Container registry Started 0.2s
✔ Container harbor-core Started 0.1s
✔ Container nginx Started 0.1s
✔ Container harbor-jobservice Started 0.0s
[root@master01 kubernetes]# curl https://10.0.250.4/harbor/projects
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<title>Harbor</title>
<base href="/"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2"/>
<link rel="stylesheet" href="styles.ac415221c96d2bef.css"></head>
<body>
<harbor-app>
<div class="spinner spinner-lg app-loading app-loading-fixed">
Loading...
</div>
</harbor-app>
<script src="runtime.2ce36195b41ae8b5.js" type="module"></script><script src="polyfills.d87db3092ff69ed9.js" type="module"></script><script src="scripts.3846d86d42cdb753.js" defer></script><script src="main.8999bc94f4db915b.js" type="module"></script></body>
</html>
[root@master01 kubernetes]# curl https://hb.aixxw.xin/harbor/projects
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<title>Harbor</title>
<base href="/"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2"/>
<link rel="stylesheet" href="styles.ac415221c96d2bef.css"></head>
<body>
<harbor-app>
<div class="spinner spinner-lg app-loading app-loading-fixed">
Loading...
</div>
</harbor-app>
<script src="runtime.2ce36195b41ae8b5.js" type="module"></script><script src="polyfills.d87db3092ff69ed9.js" type="module"></script><script src="scripts.3846d86d42cdb753.js" defer></script><script src="main.8999bc94f4db915b.js" type="module"></script></body>
</html>
================================分割线===================================
5. 搭建副镜像
先新建一个仓库
6. 副节点配置
scp docker-comp.tar.gz 10.0.250.5:/root/
scp harbor-offline-installer-v2.14.0.tgz 10.0.250.5:/root/
[root@master03 ~]# tar xf docker-comp.tar.gz
[root@master03 ~]# ./install-docker.sh
[root@master03 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
[root@master03 ~]# tar xf harbor-offline-installer-v2.14.0.tgz -C /usr/local/
[root@master03 ~]# cd /usr/local/harbor/
[root@master03 harbor]# ls
LICENSE common.sh harbor.v2.14.0.tar.gz harbor.yml.tmpl install.sh prepare
cp harbor.yml{.tmpl,}
vim harbor.yml
...
#hostname: reg.mydomain.com
hostname: 10.0.250.5
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https: #可注释
# # https port for harbor, default is 443
port: 443 #可注释
# # The path of cert and key files for nginx
certificate: /etc/aliyum/aixxw/aixxw.xin.pem #可注释
private_key: /etc/aliyum/aixxw/aixxw.xin.key #可注释
# enable strong ssl ciphers (default: false)
# strong_ssl_ciphers: false
...
harbor_admin_password: 123
...
data_volume: /var/lib/harbor
...
提示：
配置了nginx的记得关闭
systemctl disable --now nginx
[root@master03 harbor]# ./install.sh
================================分割线================================
提示：
开启了443的不要忘记证书
[root@master03 harbor]# mkdir -pv /etc/aliyum/aixxw/
mkdir: created directory '/etc/aliyum'
mkdir: created directory '/etc/aliyum/aixxw/'
[root@master03 harbor]# cd /etc/aliyum/aixxw/
[root@master03 aixxw]# ls
aixxw.xin.key aixxw.xin.pem
[root@master03 ~]# cd /usr/local/harbor/
================================分割线================================
game镜像包下载地址：
链接:https://pan.baidu.com/s/1k4yZHQoCawwvsUkKRbtBKw?pwd=v2kf 提取码:v2kf
回到主镜像服务器
docker load -i testgame-v0.1.tar
73974f74b436: Loading layer 31.57MB/31.57MB
b3c13d98fe6e: Loading layer 62.88MB/62.88MB
Loaded image: testgame:v0.1
[root@master02 ~]# docker image tag testgame:v0.1 hb.aixxw.xin/gamehtml/testgame:v0.1
登录：
输入用户admin 密码123
提示：配置hosts解析优先内网传输
公网：
docker login hb.aixxw.xin #登录
docker logout hb.aixxw.xin #退出
内网：
docker login 10.0.250.4
docker logout 10.0.250.4
提交
[root@master02 ~]# docker image push hb.aixxw.xin/gamehtml/testgame:v0.1
回到副harbor页面观察是否同步到位
每1分钟同步一次太消耗资源仅测试使用（建议改成5-15分钟同步一次）
接下里构建harbor的高可用
7. 高可用的实现
方案一：keepalived
7.1 两个节点安装keepalived
apt -y install keepalived
7.2 harbor主节点修改keepalived配置
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
router_id 10.0.250.4
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 8443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 10.0.250.4
nopreempt
authentication {
auth_type PASS
auth_pass pwdk8s
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.0.250.200
}
}
EOF
7.2 副节点修改keepalived配置
cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
router_id 10.0.250.5
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 8443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
priority 80
advert_int 1
mcast_src_ip 10.0.250.5
nopreempt
authentication {
auth_type PASS
auth_pass pwdk8s
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.0.250.200
}
}
EOF
都配置健康检查脚本
cat /etc/keepalived/check_port.sh
#!/bin/bash
CHK_PORT=$1
[ -z "$CHK_PORT" ] && { echo "Error: 检测端口不能为空！"; exit 1; }
ss -tln | grep -q ":$CHK_PORT "
if [ $? -eq 0 ]; then
exit 0
else
echo "Warning: 8443端口未监听，触发Keepalived切换..."
exit 1
fi
俩节点都启动
systemctl enable --now keepalived
测试
https://10.0.250.200/
方案二：nginx四层主备模式
我的环境是公有云可以用SLB 但这里没有用自己用LB做了四层转发
cd /etc/nginx/
[root@apiserver-lb nginx]# cat nginx.conf
...
events {
...
}
...
stream {
#harbor_nodes
upstream harbor_nodes {
server 10.0.250.4:443 max_fails=2 fail_timeout=30s;
server 10.0.250.5:443 max_fails=2 fail_timeout=30s backup;
}
# 8444 harbor_nodes
server {
listen 8444;
proxy_pass harbor_nodes;
proxy_timeout 1h;
proxy_responses 0;
proxy_buffer_size 4k;
}
}
...
http {
...
}
测试
docker login hb.aixxw.xin:8444 #登录
改标签
docker image tag hb.aixxw.xin/gamehtml/testgame:v0.1 hb.aixxw.xin:8444/gamehtml/testgame:v0.1
推送
docker image push hb.aixxw.xin:8444/gamehtml/testgame:v0.1
在副harbor试试拉取到本地docker image里
docker pull hb.aixxw.xin:8444/gamehtml/testgame:v0.1
docker logout hb.aixxw.xin:8444 #退出
=================================分割线=================================
试试k8s的ctr拉取怎么样
[root@worker01 ~]# ctr images pull hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:bd1487129c4e01470664c3f3c9a25ce01f73ff3df75ffa7eb3388d3d4b945369: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9b857f539cb142c9aa2201a17bb8e1cd5cf12edd4a65adf5732fe9f4343964cf: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:8a3cdc4d1ad3e314a91f76b7b99eed443f2152e3a9bf33e46669b31d094be443: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.1 s total: 29.0 M (26.4 MiB/s)
unpacking linux/amd64 sha256:bd1487129c4e01470664c3f3c9a25ce01f73ff3df75ffa7eb3388d3d4b945369...
done: 9.426132ms
[root@worker01 ~]# ctr -n k8s.io images ls |grep hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
[root@worker01 ~]# ctr images ls |grep hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04 application/vnd.docker.distribution.manifest.v2+json sha256:bd1487129c4e01470664c3f3c9a25ce01f73ff3df75ffa7eb3388d3d4b945369 29.0 MiB linux/amd64 -
[root@worker01 ~]# ctr images rm hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
[root@worker01 ~]# ctr images ls |grep hb.aixxw.xin:8444/dockersystemos/ubuntu:22.04
[root@worker01 ~]#
ok至此全部完成 这套部署绝对满足日常使用了
汇聚全球AI编程工具，助力开发者即刻编程。
更多推荐
16
0- 0
已为社区贡献4条内容
所有评论(0)