Nginx实现动态负载均衡，首先需要一个服务发现集群，通过集群中注册的信息动态更新nginx的配置，实现动态负载均衡。因此首先准备一个Consul集群
Consul集群准备
此处我们仅作为功能架构的测试阶段，因此在一台虚拟机上完成测试，这里准备一台Centos 7.4的虚拟机，IP为192.168.99.12
mkdir /data/consul && cd $_
wget https://releases.hashicorp.com/consul/1.9.3/consul_1.9.3_linux_amd64.zip
unzip consul_1.7.3_linux_amd64.zip
mv consul /usr/local/bin/
在完成consul的安装之后，需要准备一下集群的基础环境配置
代码语言：javascript复制mkdir -pv /data/consul/node{1..3}
我们创建三个consul节点使用的配置文件，分别对应各自的文件夹内
/data/consul/node1/consul_config1.json
{
"datacenter": "dev",
"data_dir": "/data/consul/node1",
"log_file": "/data/consul/node1/consul.log",
"log_level": "INFO",
"server": true,
"node_name": "node1",
"ui": true,
"bind_addr": "192.168.99.12",
"client_addr": "192.168.99.12",
"advertise_addr": "192.168.99.12",
"bootstrap_expect": 3,
"ports":{
"http": 8510,
"dns": 8610,
"server": 8310,
"serf_lan": 8311,
"serf_wan": 8312
}
}
/data/consul/node2/consul_config2.json
{
"datacenter": "dev",
"data_dir": "/data/consul/node2",
"log_file": "/data/consul/node2/consul.log",
"log_level": "INFO",
"server": true,
"node_name": "node2",
"ui": true,
"bind_addr": "192.168.99.12",
"client_addr": "192.168.99.12",
"advertise_addr": "192.168.99.12",
"bootstrap_expect": 3,
"ports":{
"http": 8520,
"dns": 8620,
"server": 8320,
"serf_lan": 8321,
"serf_wan": 8322
}
}
/data/consul/node3/consul_config3.json
{
"datacenter": "dev",
"data_dir": "/data/consul/node3",
"log_file": "/data/consul/node3/consul.log",
"log_level": "INFO",
"server": true,
"node_name": "node3",
"ui": true,
"bind_addr": "192.168.99.12",
"client_addr": "192.168.99.12",
"advertise_addr": "192.168.99.12",
"bootstrap_expect": 3,
"ports":{
"http": 8530,
"dns": 8630,
"server": 8330,
"serf_lan": 8331,
"serf_wan": 8332
}
}
然后便可以启动consul集群了
nohup consul agent -config-file=/data/consul/node1/consul_config1.json > /dev/null 2>&1 &
nohup consul agent -config-file=/data/consul/node2/consul_config2.json -retry-join=192.168.99.12:8311 > /dev/null 2>&1 &
nohup consul agent -config-file=/data/consul/node3/consul_config3.json -retry-join=192.168.99.12:8311 > /dev/null 2>&1 &
启动之后，便可以通过地址http://192.168.99.12:8510地址访问，此处192.168.99.12:8510即是Leader角色
consul-cluster-node-dashboard
编译Nginx
做动态负载均衡的时候需要添加nginx-upsync-module和nginx_upstream_check_module两个模块，因此此处我们需要将这两个模块给编译到nginx中去。因为此前我们在基础镜像的时候已经写好了nginx自动化编译的Dockerfile，所以我们这里直接使用即可
- Dockerfile (
Nginx 1.14.2)
FROM debian:stretch-slim
RUN useradd www &&
mkdir -p /logs/nginx/ /webserver/nginx /webserver/nginx/conf/upsync &&
chown -R www:www /logs/nginx/ /webserver/nginx &&
echo 'deb http://mirrors.163.com/debian/ stretch main non-free contrib' > /etc/apt/sources.list &&
echo 'deb http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list &&
echo 'deb-src http://mirrors.163.com/debian/ stretch main non-free contrib' >> /etc/apt/sources.list &&
echo 'deb-src http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list &&
echo 'deb-src http://mirrors.163.com/debian/ stretch-backports main non-free contrib' >> /etc/apt/sources.list &&
echo 'deb-src http://mirrors.163.com/debian-security/ stretch/updates main non-free contrib' >> /etc/apt/sources.list &&
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&
apt-get update &&
apt-get install -y wget vim net-tools unzip libjemalloc-dev &&
apt-get build-dep -y nginx
RUN
cd /usr/local/src/ &&
wget -c http://nginx.org/download/nginx-1.14.2.tar.gz &&
wget -c https://www.openssl.org/source/old/1.0.2/openssl-1.0.2m.tar.gz &&
wget -c https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1rc1.tar.gz &&
wget -c https://github.com/openresty/lua-nginx-module/archive/v0.10.11.tar.gz &&
wget -c https://github.com/xiaokai-wang/nginx_upstream_check_module/archive/master.zip -O nginx_upstream_check_module.zip &&
wget -c https://github.com/weibocom/nginx-upsync-module/archive/master.zip -O nginx-upsync-module.zip &&
tar zxf ./nginx-1.14.2.tar.gz && rm nginx-1.14.2.tar.gz &&
tar zxf ./openssl-1.0.2m.tar.gz && rm openssl-1.0.2m.tar.gz &&
tar zxf ./v0.3.1rc1.tar.gz && rm v0.3.1rc1.tar.gz &&
tar zxf ./v0.10.11.tar.gz && rm v0.10.11.tar.gz &&
unzip ./nginx_upstream_check_module.zip && rm nginx_upstream_check_module.zip &&
unzip ./nginx-upsync-module.zip && rm nginx-upsync-module.zip
RUN
cd /usr/local/src/nginx-1.14.2 &&
patch -p1 < /usr/local/src/nginx_upstream_check_module-master/check_1.12.1 .patch &&
./configure
--prefix=/webserver/nginx
--user=www --group=www --with-pcre
--with-stream
--with-http_v2_module
--with-http_ssl_module
--with-ld-opt=-ljemalloc
--with-http_realip_module
--with-http_gzip_static_module
--with-http_stub_status_module
--http-log-path=/logs/nginx/access.log
--error-log-path=/logs/nginx/error.log
--with-openssl=/usr/local/src/openssl-1.0.2m
--add-module=/usr/local/src/ngx_devel_kit-0.3.1rc1
--add-module=/usr/local/src/lua-nginx-module-0.10.11
--add-module=/usr/local/src/nginx_upstream_check_module-master
--add-module=/usr/local/src/nginx-upsync-module-master &&
make &&
make install
upstream测试服务
此处我们可以写个http的demo服务也行，或者更简单点，也可以直接使用Docker运行两个服务.此处我们向consul集群中注册两个已有的服务
代码语言：javascript复制curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:9000
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:9001
执行完之后在Dashboard中的k/v处即可看到
consul cluster key/value
准备Nginx动态更新的配置文件
我们这里作为功能测试实验，所以准备一个简洁的nignx.conf配置文件
- nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream app {
upsync 192.168.99.12:8510/v1/kv/upstreams/app/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /webserver/nginx/conf/app.conf; # 当consul故障时候，就可以把此作为备份配置文件
include /webserver/nginx/conf/app.conf; # 准备一个兼容的nginx测试文件，如果没有第一次启动可能起不来
check interval=1000 rise=2 fall=2 timeout=3000 type=http default_down=false;
check_http_send "HEAD / HTTP/1.0rnrn";
check_http_expect_alive http_2xx http_3xx;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://app;
}
location /upstream_list {
upstream_show;
}
location /upstream_status {
check_status;
access_log off;
}
}
}
app.conf
server 127.0.0.1:80 weight=1 max_fails=2 fail_timeout=10s down;
server 127.0.0.1:81 weight=1 max_fails=2 fail_timeout=10s down;
准备完成之后，我们就可以启动nginx服务了
代码语言：javascript复制docker run -d --name nginx-consul -v /Users/marionxue/Downloads/webserver/nginx.conf:/webserver/nginx/conf/nginx.conf -v /Users/marionxue/Downloads/webserver/app.conf:/webserver/nginx/conf/app.conf -p 80:80 -p 443:443 nginx:upsync
容器运行正常后，就可以通过http://127.0.0.1/upstream_list查看upstream主机了，此时app.conf的内容也会被动态的更新，这里可以作为consul故障后的备份配置文件
容器启动后的app.conf的内容
代码语言：javascript复制root@be1b245c1c47:/webserver/nginx/sbin# cat ../conf/app.conf
server 192.168.99.12:9001 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.99.12:9000 weight=1 max_fails=2 fail_timeout=10s;
consul cluster upstream list
此处通过curl增加一个新的主机节点观察是否会自动的更新
> curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":1}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:8510
true
增加新的主机节点
这和我们预期的一样，不需要干预nginx服务，即可自动完成nginx配置的更新。此处即基本上完成nginx实现动态负载均衡的基础实验。
