问题描述
业务最近发现 nginx 反向代理服务器SYN 连接特别多
lf-weather-nginx-101-11:~ # tail -f /var/log/messages
Dec 24 09:31:29 lf-weather-lvs-101-11 kernel: [7406388.265092] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:32:50 lf-weather-lvs-101-11 kernel: [7406469.537620] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:35:02 lf-weather-lvs-101-11 kernel: [7406600.783795] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:38:30 lf-weather-lvs-101-11 kernel: [7406809.145161] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:39:57 lf-weather-lvs-101-11 kernel: [7406895.519336] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:41:05 lf-weather-lvs-101-11 kernel: [7406964.278512] possible SYN flooding on port 8080. Sending cookies.
Dec 24 09:44:26 lf-weather-lvs-101-11 kernel: [7407164.819061] possible SYN flooding on port 8080. Sending cookies.
解决步骤
1.网络TCP UDP连接排名
lf-weather-nginx-101-11:~ # netstat -ntu | awk '{print $5"n"}' | cut -d: -f1 | sort | uniq -c | sort -nr|head -n 20
54456
44 220.202.103.116
41 61.50.248.5
37 111.20.241.255
30 220.201.8.217
29 113.57.255.1
28 61.55.156.20
24 221.226.97.19
22 10.51.241.104
20 121.31.254.44
17 61.158.153.214
17 122.96.104.19
14 61.158.152.232
14 61.158.152.215
14 61.158.152.202
14 202.99.82.23
14 101.254.209.16
13 61.158.152.241
13 61.158.152.225
13 61.158.152.192
正常
2.连接状态排名
lf-weather-nginx-101-11:~ # netstat -n | awk '/^tcp/ { S[$NF]} END {for(a in S) print a, S[a]}'
TIME_WAIT 1293
CLOSE_WAIT 1
FIN_WAIT1 3428
ESTABLISHED 5330
FIN_WAIT2 46193
SYN_RECV 371
CLOSING 32
LAST_ACK 113
发现FIN_WAIT2特别多不正常
3.nginx client IP 连接数排名
lf-weather-nginx-101-12:/opt/huawei/nginx/logs # awk '{print $1}' access__xx.log|awk '{count[$1] }END{for(i in count)print i,count[i]}'|sort -n -k 2 -r |head -n 10
112.97.30.1 417
61.50.248.5 371
61.55.156.20 368
119.145.15.27 361
58.251.152.27 351
61.158.153.164 310
220.202.103.116 287
61.158.153.214 279
221.226.97.19 218
220.201.8.217 216
4.调整后的内核差数如下
lf-weather-nginx-101-12: # cat /etc/sysctl.conf
# Disable response to broadcasts.
# You don't want yourself becoming a Smurf amplifier.
net.ipv4.icmp_echo_ignore_broadcasts = 1
# enable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 1
# enable ipV6 forwarding
#net.ipv6.conf.all.forwarding = 1
# increase the number of possible inotify(7) watches
fs.inotify.max_user_watches = 65536
# avoid deleting secondary IPs on deleting the primary IP
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
net.ipv4.ip_forward=0
net.ipv4.conf.default.arp_filter=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.secure_redirects=0
net.ipv4.conf.default.secure_redirects=0
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.log_martians=1
net.ipv4.tcp_timestamps=0
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.conf.all.proxy_arp=0
net.ipv4.conf.default.proxy_arp=0
kernel.sysrq=8
net.ipv4.tcp_syncookies=1 #表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.route.gc_interval=1
net.ipv4.route.gc_timeout=30 #tcp连接超时时间 之前是60
net.ipv4.ip_local_port_range = 1024 65535 #tcp端口的使用范围
net.ipv4.tcp_tw_reuse = 1 #表示开启TCP连接中TIME-WAIT sockets的快速回收 新添加
net.ipv4.tcp_tw_recycle = 1 #和net.ipv4.tcp_tw_reuse 同时开启才能生效
net.core.somaxconn = 262144 #web应用中listen函数的backlog默认会将内核参数的net.core.somaxconn限制到128，而nginx定义的NGX_LISTEN_BACKLOG默认是511，所以必须调整
net.ipv4.tcp_fin_timeout = 30 #表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_max_syn_backlog = 262144 #表示SYN队列的长度，默认为1024，加大队列长度为262144，可以容纳更多等待连接的网络连接数。
net.ipv4.tcp_max_tw_buckets = 262144 #表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息
net.core.netdev_max_backlog = 262144 #每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目
lf-weather-nginx-101-12: #sysctl -p
几分钟后看FIN_WAIT2明显减少
lf-weather-nginx-101-12:/opt/huawei/nginx/logs # ss -s
Total: 4540 (kernel 4691)
TCP: 33564 (estab 4463, closed 25651, orphaned 3428, synrecv 0, timewait 25651/0), ports 355
Transport Total IP IPv6
* 4691 - -
RAW 0 0 0
UDP 7 7 0
TCP 7913 7913 0
INET 7920 7920 0
FRAG 0 0 0
再看日志 SYN依然存在 不过一分钟出现一个还能接受吧 排除攻击业务访问也正常
lf-weather-nginx-101-12:/opt/huawei/nginx/logs # tail -f /var/log/messages
Dec 24 10:15:01 lf-weather-lvs-101-12 kernel: [7408800.117536] possible SYN flooding on port 8080. Sending cookies.
Dec 24 10:16:03 lf-weather-lvs-101-12 kernel: [7408861.984733] possible SYN flooding on port 8080. Sending cookies.
Dec 24 10:19:51 lf-weather-lvs-101-12 kernel: [7409089.678098] possible SYN flooding on port 8080. Sending cookies.
Dec 24 10:21:41 lf-weather-lvs-101-12 kernel: [7409199.529564] possible SYN flooding on port 8080. Sending cookies.
Dec 24 10:23:08 lf-weather-lvs-101-12 kernel: [7409286.745773] possible SYN flooding on port 8080. Sending cookies.
