docker部署node-exporter+prometheus+grafana+alertmanager
·
监控的架构图
一个主机部署 prometheus+grafana+alertmanager+node-exporter
其余主机部署 node-exporter
部署 node-exporter
每个主机都需要部署
作用：收集服务器的CPU、内存、磁盘等原始指标
拉取镜像 网址：https://hub.docker.com/r/prom/node-exporter
docker pull prom/node-exporter:v1.10.2
docker run node-exporter
docker run -d --name node-exporter --restart=always -p 9100:9100 \
-v "/proc:/host/proc:ro" \
-v "/sys:/host/sys:ro" \
-v "/:/rootfs:ro" \
prom/node-exporter:v1.10.2
部署promethues
作用：存储指标数据、提供查询、触发告警规则
创建目录
mkdir -p /opt/prometheus/config
mkdir -p /opt/prometheus/data
mkdir -p /opt/prometheus/rules
sudo chown -R 65534:65534 /opt/prometheus/data
sudo chown -R 65534:65534 /opt/prometheus/rules
拉取镜像 网址：https://hub.docker.com/r/prom/prometheus
docker pull prom/prometheus:v3.10.0
添加 Prometheus 配置文件
vim /opt/prometheus/config/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['10.0.3.99:9093']
rule_files:
- "/prometheus/rules/*.yml"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['10.0.3.99:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['10.0.3.99:9100','10.0.3.98:9100']
Prometheus添加指标
kubectl edit deploy kube-state-metrics
kubectl edit servicemonitor kube-state-metrics
Prometheus的自定义PromQL
# POD重启原因 会输出 pod重启原因和退出码 节点IP
kube_pod_container_status_last_terminated_exitcode * on(pod, namespace, container) group_left(reason) kube_pod_container_status_last_terminated_reason * on(pod, namespace) group_left(node) kube_pod_info + on(pod, namespace, container) group_left() ( 0 * (increase(kube_pod_container_status_restarts_total[5m]) > 0) )
# 近5分钟重启 会输出节点IP
increase(kube_pod_container_status_restarts_total[5m]) * on(pod, namespace) group_left(node) kube_pod_info > 3
# POD因OOM重启 添加条件
( increase(kube_pod_container_status_restarts_total[5m] ) > 0) * on(pod, namespace) group_left(node) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} * on(pod, namespace) group_left(node) kube_pod_info > 0 )
# POD_CPU使用率 忽略 namespace 是 test 的
100 * (sum(rate(container_cpu_usage_seconds_total{image!="", container!="POD", namespace!="test"}[1m])) by (cluster_name,pod,node,namespace,container, cluster) / sum(kube_pod_container_resource_limits{resource="cpu", namespace!="test"}) by (cluster_name,pod,node,namespace,container, cluster)) > 90
# POD内存使用率
(sum(container_memory_working_set_bytes{image!="", container!="POD"}) BY (cluster_name, node,container, pod , namespace, cluster) / sum(container_spec_memory_limit_bytes > 0) BY (cluster_name, node, container, pod , namespace, cluster) * 100) > 90
# PVC磁盘使用率 注意 device= ~"/dev/sd.*" 要换成实际机器上的
(1 - avg(node_filesystem_avail_bytes{device=~"/dev/sd.*"}/ node_filesystem_size_bytes{device=~"/dev/sd.*"} != 0) by (device, node, cluster_name, cluster)) * 100 > 90
# 节点CPU使用率
100 - (avg by(node, cluster_name, cluster) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 85
# 节点内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
# 节点磁盘使用率
(1 - avg(node_filesystem_avail_bytes{device!~"/dev/sd.*"}/(node_filesystem_size_bytes{device!~"/dev/sd.*"} != 0)) by (device, node, cluster_name, cluster)) * 100 > 80
添加告警配置文件
vim /opt/prometheus/rules/node_alerts.yml
groups:
- name: general.rules
rules:
- alert: CPU使用率
expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[30s])) * 100) > 40
for: 1m
labels:
severity: warning
annotations:
value: "{{ printf \"%.2f\" $value }}"
instance: "{{ $labels.instance }}"
- alert: 内存使用率
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 40
for: 1m
labels:
severity: warning
annotations:
value: "{{ printf \"%.2f\" $value }}"
instance: "{{ $labels.instance }}"
- alert: 磁盘使用率
expr: 100 - (node_filesystem_free_bytes{mountpoint="/rootfs", fstype=~"ext4|xfs"} / node_filesystem_size_bytes{mountpoint="/rootfs", fstype=~"ext4|xfs"} * 100) > 40
for: 1m
labels:
severity: warning
annotations:
value: "{{ printf \"%.2f\" $value }}"
instance: "{{ $labels.instance }}"
docker run prometheus
docker run -d --name prometheus --restart=always -p 9090:9090 \
-v /etc/localtime:/etc/localtime \
-v /opt/prometheus:/prometheus \
prom/prometheus:v3.10.0 \
--storage.tsdb.retention.time=100d \
--config.file=/prometheus/config/prometheus.yml
打开页面 http://127.0.0.1:9090/targets
可以看到监控的指标 对应的配置文件：/opt/prometheus/config/prometheus.yml
告警规则 对应配置文件：/opt/prometheus/rules/node_alerts.yml
可以查询规则是否正确能否读取出数据
部署grafana
作用：把数据变成图表展示
创建目录
mkdir -p /opt/grafana/data
sudo chown -R 472:472 /opt/grafana/data
拉取镜像 网址：https://hub.docker.com/r/grafana/grafana
docker pull grafana/grafana:12.3
docker run grafana
docker run -d --name grafana --restart=always -p 3000:3000 \
-v /etc/localtime:/etc/localtime \
-v /opt/grafana/data:/var/lib/grafana \
grafana/grafana:12.3
打开页面 http://127.0.0.1:3000/ 输入用户名密码 默认都是admin
添加数据源
其余不变 点击 下方图标即可 数据源添加成功
导入 Dashboards
输入 8919 点击 load
部署alertmanager
作用：接收告警、去重、分组、决定发送方式（邮件、webhook等）
创建目录
mkdir -p /opt/alertmanager
拉取镜像 网址：https://hub.docker.com/r/prom/alertmanager
docker pull prom/alertmanager:v0.31.1
添加告警地址 webhook
vim /opt/alertmanager/alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
# 等待30秒收集同组告警
group_wait: 30s
# 30秒后检查告警是否仍存在
group_interval: 30s
# 如果告警没解决，30s后再次提醒
repeat_interval: 30s
# 修改为 webhook 接收器名称
receiver: 'webhook'
receivers:
# 接收器名称
- name: 'webhook'
webhook_configs:
- url: 'http://10.0.3.99:4000/alert'
send_resolved: true # 是否发送恢复通知
docker run alertmanager
docker run -d --name alertmanager --restart=always -p 9093:9093 \
-v /etc/localtime:/etc/localtime \
-v /opt/alertmanager/:/etc/alertmanager/ \
prom/alertmanager:v0.31.1 \
--config.file=/etc/alertmanager/alertmanager.yml
用python写一个告警webhook的服务
vim lark_python.py
from flask import Flask,request,jsonify
import requests
# 创建Flask应用实例
app = Flask(__name__)
# 定义路由和视图函数
@app.route('/alert', methods=['POST'])
def receive_alert():
alert_response = request.json
alerts = alert_response["alerts"]
if len(alerts) > 0:
for alert in alerts:
print(alert)
instance_ip = alert["annotations"]["instance"]
alarm_value = alert["annotations"]["value"]
alarm_name = alert["labels"]["alertname"]
alarm_status = alert["status"]
if alarm_status =="resolved":
alarm_value = get_resolved_value(instance_ip,alarm_name)
lark_webhook(instance_ip,alarm_name,alarm_value,alarm_status)
return "OK", 200
# 获取告警解决后的值
def get_resolved_value(instance_ip,alarm_name):
PROMETHEUS_URL="http://10.0.3.99:9090"
if "磁盘使用率" == alarm_name:
query = f'100 - (node_filesystem_free_bytes{{mountpoint="/rootfs", instance="{instance_ip}"}} / node_filesystem_size_bytes{{mountpoint="/rootfs", instance="{instance_ip}"}} * 100)'
elif "CPU使用率" == alarm_name:
query = f'100 - (avg by (instance) (irate(node_cpu_seconds_total{{mode="idle", instance="{instance_ip}"}}[30s])) * 100)'
elif "内存使用率" == alarm_name:
query = f'(1 - (node_memory_MemAvailable_bytes{{instance="{instance_ip}"}} / node_memory_MemTotal_bytes{{instance="{instance_ip}"}})) * 100'
resp = requests.get(PROMETHEUS_URL + '/api/v1/query', params={'query': query})
current_value = resp.json()['data']['result'][0]['value'][1]
formatted_value = f"{float(current_value):.2f}"
return formatted_value
# 组装告警格式
def lark_webhook(instance_ip,alarm_name,alarm_value,alarm_status):
lark_msg = {
"msg_type": "interactive",
"card": {
"config": {
"wide_screen_mode": True
},
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"主机告警\n"
}
},
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"资源名称: {instance_ip}"
}
},
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"告警信息: {alarm_name} {alarm_value} %"
}
},
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"当前状态: {'🔴 告警中' if alarm_status == 'firing' else '✅ 已恢复'}"
}
}
]
}
}
# 发送lark告警
webhook = "https://open.larksuite.com/open-apis/bot/v2/hook/{{换成自己的token}}"
requests.post(webhook, json=lark_msg)
# 启动应用
if __name__ == '__main__':
app.run(host='0.0.0.0', port=4000)
python3 lark_python.py
模拟占用磁盘空间告警
fallocate -l 8G /tmp/test.8g
模拟占用CPU告警
seq 1 $(nproc) | xargs -P0 -I{} sh -c 'while :; do :; done'
汇聚全球AI编程工具，助力开发者即刻编程。
更多推荐
7
0- 0
已为社区贡献2条内容
所有评论(0)