0.准备

minio官方文档：https://www.minio.org.cn/docs/minio/linux/

6台服务器：四台用于minio分布式集群部署，2台用于nginx+keepalived高可用，keep alived VIP =192.168.10.17

IP	hostname
192.168.10.11	minio-01
192.168.10.12	minio-02
192.168.10.13	minio-03
192.168.10.14	minio-04
192.168.10.15	nginx-01
192.168.10.16	nginx-02

一、minio集群配置、四个节点步骤一致

1.minio下载

https://dl.minio.org.cn/server/minio/release/linux-amd64/minio

2.目录规划

目录	位置
数据	/data/minio/data
base	/usr/local/minio
config	/usr/local/minio/config
log	/data/minio/logs

（1）创建不存在的目录，【/data】在服务器中需要单独挂载一块盘

mkdir -p /usr/local/minio/config
mkdir -p /data/minio/data
mkdir -p /data/minio/logs

（2）上传minio二进制文件

cd /usr/local/minio/
chmod 755 minio

3.集群启动脚本

需要保证脚本中服务器http链接的顺序·一致

minio-01

vi /usr/local/minio/minio-start.sh

#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=password
/usr/local/minio/minio server --config-dir /usr/local/minio/config \
--address "192.168.10.11:9001" --console-address 192.168.10.11:9000 \
http://192.168.10.11/data/minio/data \
http://192.168.10.12/data/minio/data \
http://192.168.10.13/data/minio/data \
http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-02

vi /usr/local/minio/minio-start.sh

#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=password
/usr/local/minio/minio server --config-dir /usr/local/minio/config \
--address "192.168.10.12:9001" --console-address 192.168.10.12:9000 \
http://192.168.10.11/data/minio/data \
http://192.168.10.12/data/minio/data \
http://192.168.10.13/data/minio/data \
http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-03

vi /usr/local/minio/minio-start.sh

#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=password
/usr/local/minio/minio server --config-dir /usr/local/minio/config \
--address "192.168.10.13:9001" --console-address 192.168.10.13:9000 \
http://192.168.10.11/data/minio/data \
http://192.168.10.12/data/minio/data \
http://192.168.10.13/data/minio/data \
http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-04

vi /usr/local/minio/minio-start.sh

#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=password
/usr/local/minio/minio server --config-dir /usr/local/minio/config \
--address "192.168.10.14:9001" --console-address 192.168.10.14:9000 \
http://192.168.10.11/data/minio/data \
http://192.168.10.12/data/minio/data \
http://192.168.10.13/data/minio/data \
http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

4.配置开机启动服务

（1）修改启动文件

vi /etc/systemd/system/minio.service

[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/usr/local/minio/
ExecStart=/usr/local/minio/minio-run.sh

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

（2）脚本增加可执行权限

systemctl daemon-reload
systemctl enable minio
systemctl start minio
systemctl status minio

5.访问验证

http://192.168.10.11:9000

二、nginx配置，双节点步骤一致

1.nginx下载链接

https://nginx.org/en/download.html

2.安装依赖包

yum install -y gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel

3.创建用户组

useradd -M -s /sbin/nologin nginx

4.解压、编译安装

tar -xzvf nginx-1.23.4.tar.gz
cd nginx-1.23.4
./configure --prefix=/usr/local/nginx --user=nginx --group=nginx --with-http_stub_status_module  --with-http_ssl_module --with-http_mp4_module --with-http_realip_module --with-pcre --with-http_gunzip_module --with-http_gzip_static_module --with-stream

5.增加软连接、开机启动文件

ln -s sbin/nginx   /usr/local/sbin/
vi /lib/systemd/system/nginx.service

[Unit]
Description=nginx service
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s quit
PrivateTmp=true

[Install]
WantedBy=multi-user.target

chmod 754 /lib/systemd/system/nginx.service

6.配置conf文件

cd /usr/local/nginx/conf
vi nginx.conf

worker_processes 4;
events {
        worker_connections 1024;
}

http {

        tcp_nodelay on;
        types_hash_max_size 2048;

        sendfile on;
        tcp_nopush on;
        server_tokens off;

        keepalive_timeout 60;
        keepalive_requests 2048;

        gzip on;
        gzip_static on;
        gzip_min_length 1000;
        gzip_buffers 4 8k;
        gzip_comp_level 7;
        gzip_http_version 1.1;
        gzip_vary on;
        gzip_types text/plain application/x-javascript text/css application/xml image/jpeg image/gif image/png;
        proxy_max_temp_file_size 0;
        proxy_buffer_size 64k;
        proxy_buffers 4 64k;
        proxy_connect_timeout 500ms;
        proxy_headers_hash_max_size 1024;
        proxy_headers_hash_bucket_size 128;
        client_header_buffer_size 16k;
        large_client_header_buffers 4 32k;
        client_max_body_size 80m;
        client_body_buffer_size 60m;
        server_names_hash_max_size 1024;
        server_names_hash_bucket_size 1024;
        underscores_in_headers on;

upstream minio_server {
        least_conn;
        server 192.168.10.11:9000;
        server 192.168.10.12:9000;
        server 192.168.10.13:9000;
        server 192.168.10.14:9000;
}

upstream minio_console {
        least_conn;
        server 192.168.10.11:9001;
        server 192.168.10.12:9001;
        server 192.168.10.13:9001;
        server 192.168.10.14:9001;
}
server {
        listen 9000;
        listen [::]:9000;
        server_name minio_server;

        # Allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # Disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

  location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance
        }
}

server {

        listen 9001;
        listen [::]:9001;
        #server_name console.example.net;
        server_name minio_console;

        # Allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # Disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

  location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;

        # To support websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        chunked_transfer_encoding off;

        proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance
                }
  }
}

7.启动nginx

systemctl enable nginx
systemctl start nginx
systemctl status nginx

三、keepalived配置，双节点步骤一致

1.下载链接

https://www.keepalived.org/download.html

2.安装依赖包

yum -y install libnl libnl-devel libnfnetlink-devel openssl-devel gcc

3.解压&编译安装

tar -zxvf keepalived-2.3.1.tar.gz
cd keepalived-2.3.1
./configure --prefix=/usr/local/keepalived
make && make install

4.配置开机启动

vi /usr/lib/systemd/system/keepalived.service

[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target 
Wants=network-online.target 
Documentation=man:keepalived(8)
Documentation=man:keepalived.conf(5)
Documentation=man:genhash(1)
Documentation=https://keepalived.org

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived  $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

5.配置keepalived高可用、检查脚本

vi /usr/local/keepalived/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id LVS_NGINX
}

vrrp_script chk_nginx {
  script "/usr/local/keepalived/etc/keepalived/check_nginx.sh"
  interval 2
  weight -30
  rise 2
  fall 2
 }

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 150
    priority 100   # 备节点权重小 ## 192.168.10.15配置100，192.168.10.16配置80
    advert_int 3
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
      chk_nginx
    }
    virtual_ipaddress {
        192.168.10.17  # vip
    }
}

vi /usr/local/keepalived/etc/keepalived/check_nginx.sh
#!/bin/bash
nginx_status=$(systemctl is-active nginx)

if [ "$nginx_status" = "active" ]; then
        exit 0
else 
        pkill keepalived
fi

6.访问验证

配置完成后使用VIP可以访问MINIO的控制台界面，实现自动负载均衡

http://192.168.10.17:9000