prometheus安装与使用(五)
·
一、docker部署prometheus
一 介绍
Prometheus Server 监控端主程序（监控机上部署）如果prometheus是物理机部署想监控pod只要能连接到cAdvsor就行。
Node exporter 负载收集宿主机的硬件信息（node节点上部署）
cAdvsor 负责收集宿主机上的所有容器信息，好像单台容器信息收集不到它只能收集容器的总使用率,容器一般监控cpu 内存 网络 （node节点上部署）
Grafana 负责图形展示(监控机上部署)
alertmanager 负责报警（在监控机上部署）
如果想部署docker版的pushgateway，需要被监控容器需要提前部署好crontab才能定期推送数据给pushgateway
https://www.jb51.net/article/219848.htm#_label2
grafana模版地址https://grafana.com/grafana/dashboards/
二 查询
1、查询目标是容器container时
• 查询目标是容器时都是container_* 开头的
• 查询目标是物理机时都是node_* 开头的
1 容器查询参数,单位是字节(B),grafana直接改成单位子节就能自动转换
# 查询cpu使用率
sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu) * 100
# 查询内存使用量
container_memory_working_set_bytes{image!=""}
# 查询内存使用率(由于数值无穷大会显示+lnf)
sum(container_memory_working_set_bytes{name="MyNginx"}) by (name) / sum(container_spec_memory_limit_bytes{name="MyNginx"}) by (name) * 100
# 网卡每秒下载速度network Rx
irate(container_network_receive_bytes_total{image!=""}[5m])
# 网卡每秒上传速度nentwork Tx
irate(container_network_transmit_bytes_total{image!=""}[5m])
三 部署
1、启动prometheus容器
- 可以装在master上或node上
(1) 先启动prometheus访问测试
# docker run -itd --privileged=true --restart=always -p 9090:9090 --net=bridge --name=prome prom/prometheus
---------------
--net=bridge # 使用桥接网络可以直接与node exporter通信，或自己创建一个网络
(2) 把prometheus.yml文件拷贝出来,如果不拷贝出来再次启动容器时使用-v指定挂载目录容器目录会被清空
# docker cp prome:/etc/prometheus/prometheus.yml /soft/docker/prometheus/
(3) 重新启动prometheus并挂载
# docker run -itd --privileged=true --restart=always -p 9090:9090 --net=bridge -v /soft/docker/prometheus:/etc/prometheus -v /etc/localtime:/etc/localtime --name=prome prom/prometheus
--------------------
这里要把prometheus真个目录挂过来
(4) 查看容器内部ip
# docker exec -it prome ifconfig eth0|grep "inet addr"
inet addr:172.16.15.2 Bcast:172.16.15.255 Mask:255.255.255.0
(5) 访问node节点ip
http://192.168.1.10:9090
2、启动node_exporter容器
- 写法都是固定的直接粘贴就行
- 在node节点上部署
- 这些组件都使用桥接网络通信
(1) 启动node-export
docker run -d -p 9100:9100 --name=node-exporter \
-v "/proc:/host/proc" \
-v "/sys:/host/sys" \
-v "/:/rootfs" \
--net=bridge \
prom/node-exporter \
--path.procfs /host/proc \
--path.sysfs /host/sys \
--path.rootfs=/rootfs \
--collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
（2）查看容器内网ip
# docker exec -it node-exporter ifconfig eth0|grep "inet addr"
inet addr:172.16.15.3 Bcast:172.16.15.255 Mask:255.255.255.0
(3) 把node_export加入prometheus.yml并重启prometheus
- job_name: "node1"
static_configs:
- targets: ["192.168.1.10:9100"]
labels:
rule: node1
3、启动cAdvisor
- 部署上这个后才会有container 监控参数
(1) 启动cadvisor容器,--publish=-p
docker run -d \
-v "/etc/localtime:/etc/localtime" \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=18104:8080 \
--detach=true \
--name=cadvisor \
--privileged=true \
--net=bridge \
google/cadvisor:latest
==== m1 芯片要使用arm64架构，linux使用amd64架构
docker run -d \
-v "/etc/localtime:/etc/localtime" \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=18104:8080 \
--detach=true \
--name=cadvisor \
--privileged=true \
--net=bridge \
unibaktr/cadvisor:latest
===
(2) 把cadvisor加入prometheus.yml并重启prometheus
- job_name: "node1"
static_configs:
- targets: ["192.168.1.10:9100","172.16.15.4:8080"] # 直接用容器内部ip就行8080是cadvisor的端口
labels:
rule: node1
(3) 访问cadvisor可以看到所有被监控的容器信息（需要等会才能采集到数据，它只采集不存储）
4、查看prometheus是否接入成功
5、查询prometheus监控数据
(1) 这里查询的是node1节点cpu使用率信息
(2) 查看容器的cpu使用率，image容器镜像，name是容器名称
(3) 查看容器内存使用与docker stats看到的一样
6、启动grafana容器
- docker模版号193
- linux监控模版号9276
docker run -d -i -p 3000:3000 --name=grafana \
-v "/etc/localtime:/etc/localtime" \
-e "GF_SERVER_ROOT_URL=http://grafana.server.name" \
-e "GF_SECURITY_ADMIN_PASSWORD=admin" \
--net=bridge \
grafana/grafana
----------------------
账号：admin
密码：admin
7、启动alert-manager容器
(1) 先启动alert-manager把配置文件拿下来
# docker run -itd -p 9093:9093 --name=alert prom/alertmanager
# docker cp alert:/etc/alertmanager/alertmanager.yml ./
(2) 替换alertmanager.yml内容如下
# cat alertmanager.yml
global:
resolve_timeout: 5m # 持续5分钟没有收到告警标记就会认为已经发送过了
smtp_from: "1107916963@qq.com" # 配置发件邮箱信息
smtp_smarthost: 'smtp.qq.com:25'
smtp_auth_username: "1107916963@qq.com"
smtp_auth_password: "fecmtfzctecigcid"
smtp_require_tls: false # 禁用TLS传输方式
route: # 路由配置：用于判断把告警信息发给谁，路由可以有多个
group_by: ['alertname'] # 定义分组名称，promethues支持告警分组功能:满足规则的报警会被合并到一个通知中
group_wait: 10s # 分组等待时间间隔:在这个时间内收到的告警信息会合并到之前的告警信息中
group_interval: 10s # 定义相同组间发送告警通知的时间间隔
repeat_interval: 30m # 定义报警重复时间间隔
receiver: 'email' # 把告警信息发给哪一个接收者(对应reveivers中的name标签)
receivers: # 接收者配置
- name: 'email' # 接收者名称（对应receiver标签）
email_configs: # email配置参数，还支持wechat_configs、webhook_configs
- to: "wenqiangit@163.com" # 接收者邮箱
send_resolved: true
headers: # 这个不写也可以
subject: "Grafana告警" # 自定义邮件主题信息, 可以只写这一条
from: "Grafana" # 自定义由谁发出的信息
to: "OP" # 自定义要发给谁
inhibit_rules: # 抑制配置：防止同一条信息重复发送
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
(3) 再次启动并挂载
# docker run -itd -p 9093:9093 --name=alert -v /soft/docker/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager
# docker exec -it alert alertmanager --version
alertmanager, version 0.23.0
(4) 把alertmanager加入prometheus.yml并增加报警规则
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['192.168.1.10:9093']
rule_files:
- "/etc/prometheus/rules.yml"
(5) 在prometheus挂载目录下创建rules.yml
# cat /soft/docker/prometheus/rules.yml
groups:
- name: node_alert-rule1 # 第一个告警规则
rules:
- alert: cpu_alert # 告警名称
expr: 100 - avg(irate(node_cpu_seconds_total{mode="idle"}[1m])) by (instance) * 100 > 5 # 告警语句
for: 1m # 持续1分钟就发邮件告警
labels:
level: warning # 告警级别
system: node # 这里可以自定义标签对应alertmanager.yml中的子路由标签
annotations:
description: "instance: {{ $labels.instance }} , cpu usage is too high ! value: {{ $value }}" # 告警内容
summary: "cpu usage is too high" # 描述信息
- name: pod_cpu # 第二个告警规则
rules:
- alert: pod_cpu_alert # 告警名称
expr: sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu) * 100 > 5 # 告警语句
for: 1m # 持续1分钟就发邮件告警
labels:
level: warning # 告警级别
system: node # 这里可以自定义标签对应alertmanager.yml中的子路由标签
annotations:
description: "instance: {{ $labels.name }} , pod cpu usage is too high ! value: {{ $value }}" # 告警内容
summary: "cpu usage is too high" # 描述信息
(6) 查看prometheus-Alerts已经触发报警规则
二、k8s部署prometheus
方法一：通过通过github项目一件部署
实际上Prometheus Operator对于Job的配置只适用于在Kubernetes中部署和管理的应用程序。如果你希望使用Prometheus监控一些其他的资源，例如AWS或者其他平台中的基础设施或者应用，这些并不在Prometheus Operator的能力范围之内。
需要翻墙(这种方式感觉有问题也可能是我不会用，把alertmanager grafana prometheus通过NodePort暴露出来只能本机访问，外部无法访问但有时候外部又能访问？？)
https://www.cnblogs.com/huss2016/p/14865316.html
https://github.com/prometheus-operator/kube-prometheus
1、拉代码，我的kubenetes时1.25我应该拉release-0.12版本
(1) 确定我的k8s版本号
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.14 Ready <none> 18h v1.25.6
192.168.1.18 Ready <none> 18h v1.25.6
(2) 拉代码
git clone -b release-0.12 https://github.com/prometheus-operator/kube-prometheus.git
2、部署prometheus
(1) 部署
# 就这三条命令内存给大点(4g)很快部署完毕
# cd kube-prometheus/
kubectl apply --server-side -f manifests/setup
kubectl wait \
--for condition=Established \
--all CustomResourceDefinition \
--namespace=monitoring
kubectl apply -f manifests/
# 如果环境不再需要了就这样清除
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
(2) 我们查看pod发现有些镜像没有拉下来
# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 1/2 Running 1 (41s ago) 3m51s
alertmanager-main-1 1/2 Running 1 (31s ago) 3m51s
alertmanager-main-2 1/2 Running 1 (41s ago) 3m51s
blackbox-exporter-6fd586b445-8xlcb 3/3 Running 0 5m21s
grafana-9f58f8675-m2rng 1/1 Running 0 5m19s
kube-state-metrics-66659c89c-9m7vj 2/3 ImagePullBackOff 0 5m18s
node-exporter-cmtc6 2/2 Running 0 5m18s
node-exporter-cpffz 2/2 Running 0 5m18s
prometheus-adapter-757f9b4cf9-ljf5p 0/1 ErrImagePull 0 5m16s
prometheus-adapter-757f9b4cf9-mgcsj 0/1 ImagePullBackOff 0 5m16s
prometheus-k8s-0 2/2 Running 0 3m51s
prometheus-k8s-1 2/2 Running 0 3m51s
prometheus-operator-776c6c6b87-gn6ff 2/2 Running 0 5m15s
(3) 我们看一下哪些镜像没有拉下来
kubectl describe pod prometheus-adapter-757f9b4cf9-mgcsj -n monitoring
Warning Failed 5m52s (x7 over 10m) kubelet Error: ImagePullBackOff
Warning Failed 67s (x2 over 7m40s) kubelet Failed to pull image "registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0": failed to resolve reference "registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0": failed to do request: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/prometheus-adapter/prometheus-adapter/manifests/v0.10.0": dial tcp 108.177.97.82:443: i/o timeout
发现这个镜像没有拉下来registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0
我们看一下这个镜像在哪个yaml文件中
# grep "prometheus-adapter:v0.10.0" /soft/kube-prometheus/manifests/*.yaml
prometheusAdapter-deployment.yaml: image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.10.0
（4）然后我们去hub.docker中找一下这个镜像然后替换prometheusAdapter-deployment.yaml里面的镜像并重新apply
# grep "image" prometheusAdapter-deployment.yaml
image: selina5288/prometheus-adapter:v0.9.1
# kubectl apply -f prometheusAdapter-deployment.yaml
(5) 最后查看以及启动的pod
# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main 1/2 Running 18 (5m56s ago) 58m
blackbox-exporter-6fd586b445-8xlcb 3/3 Running 0 5h1m
grafana-9f58f8675-m2rng 1/1 Running 0 5h1m
kube-state-metrics-f8cbfb55f-m4sf8 3/3 Running 0 4h35m
node-exporter-cmtc6 2/2 Running 0 5h1m
node-exporter-cpffz 2/2 Running 0 5h1m
prometheus-adapter-6c7bbd7f75-5f2pm 1/1 Running 0 4h40m
prometheus-k8s-0 2/2 Running 0 5h
prometheus-k8s-1 2/2 Running 0 5h
prometheus-operator-776c6c6b87-gn6ff 2/2 Running 0 5h1m
3、访问
（1）我们查看svc资源发现都是clusterIP是无法访问的，通过需要配置Ingres来访问，但是今天我们使用Nodeport方式
# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main ClusterIP 10.68.115.212 <none> 9093/TCP,8080/TCP 5h2m
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 59m
blackbox-exporter ClusterIP 10.68.149.99 <none> 9115/TCP,19115/TCP 5h2m
grafana ClusterIP 10.68.146.11 <none> 3000/TCP 5h2m
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 5h2m
node-exporter ClusterIP None <none> 9100/TCP 5h2m
prometheus-adapter ClusterIP 10.68.219.68 <none> 443/TCP 5h2m
prometheus-k8s ClusterIP 10.68.179.68 <none> 9090/TCP,8080/TCP 5h2m
prometheus-operated ClusterIP None <none> 9090/TCP 5h1m
prometheus-operator ClusterIP None <none> 8443/TCP 5h2m
(2) 修改svc资源类型并apply
# vim prometheus-service.yaml
# vim grafana-service.yaml
# vim alertmanager-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: alert-router
app.kubernetes.io/instance: main
app.kubernetes.io/name: alertmanager
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.25.0
name: alertmanager-main
namespace: monitoring
spec:
ports:
- name: web
port: 9093
targetPort: web
- name: reloader-web
port: 8080
targetPort: reloader-web
type: NodePort # 增加一行
selector:
app.kubernetes.io/component: alert-router
app.kubernetes.io/instance: main
app.kubernetes.io/name: alertmanager
app.kubernetes.io/part-of: kube-prometheus
sessionAffinity: ClientIP
（2）再次查看svc资源已改成NodePort
# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort 10.68.115.212 <none> 9093:30055/TCP,8080:32561/TCP 5h11m
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 68m
blackbox-exporter ClusterIP 10.68.149.99 <none> 9115/TCP,19115/TCP 5h11m
grafana NodePort 10.68.146.11 <none> 3000:31406/TCP 5h11m
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 5h11m
node-exporter ClusterIP None <none> 9100/TCP 5h11m
prometheus-adapter ClusterIP 10.68.219.68 <none> 443/TCP 5h11m
prometheus-k8s NodePort 10.68.179.68 <none> 9090:30554/TCP,8080:30207/TCP 5h11m
prometheus-operated ClusterIP None <none> 9090/TCP 5h10m
prometheus-operator ClusterIP None <none> 8443/TCP 5h11m
(3) 访问node节点ip:port，如果pod所在node变了访问的ip也要换一下
# 查看pod坐在哪个node节点
# kubectl get pod -n monitoring -o wide|egrep "prometheus-k8s|grafana|alert"
alertmanager-main-0 2/2 Running 0 9m11s 172.20.0.18 192.168.1.14 <none> <none>
grafana-9f58f8675-m2rng 1/1 Running 0 6h6m 172.20.0.2 192.168.1.14 <none> <none>
prometheus-k8s-0 2/2 Running 0 13m 172.20.0.23 192.168.1.18 <none> <none>
方法二 通过独立部署的prometheus监控k8s
它是通过api接口+token+rbac授权+证书才能建立连接
http://e.betheme.net/article/show-201060.html?action=onClick
https://blog.csdn.net/m0_48898914/article/details/128194358
方法三、yaml方式部署prometheus监控k8s
https://blog.csdn.net/rzy1248873545/article/details/125758153
汇聚全球AI编程工具，助力开发者即刻编程。
更多推荐
7
0- 0
已为社区贡献9条内容
所有评论(0)