Tcpcopy在线流量压力测试

2022-06-07 09:25:14 浏览数 (2)

TCPCopy是用来做TCP重放的，常用的场景是把线上流量复制到测试环境，用来排查线下不容易重现的问题，或者对测试环境做压力测试。（HTTPS不能进行压力测试，由于数据加密）

一、需求：将线上流量引入测试环境

原本打算直接对nginx流量进行引流，但是由于网站采用https加密的。引流不成功。只能针对对单应用http进行引流

  • 1、应用关系
代码语言：javascript复制
应用           源服务器                            目前测试服务器
demo-app      10.1.3.74:7001                     192.168.54.62:82
  • 2、主机关系表
代码语言：javascript复制
线上服务器   10.1.3.74:7001
测试服务器   192.168.54.62:82
intercept（辅助服务器）  10.1.2.40
online--->中转服务器1 10.1.2.41  ---》中转服务器2 192.168.77.84 -->测试服务器

二、网络打通，借助rinetd or haproxy

需求：online--->中转服务器1 10.1.2.41 ---》中转服务器2 192.168.77.84 -->测试服务器.

由于线上环境，和测试环境不是互通，需要经过2层网路转发才能抵达测试服务器，此处借助 rinetd进行网路转发.

  • 1、中转服务器1：10.1.2.41安装 rinetd
代码语言：javascript复制
[root@vm-phx-k8s-master-0241 ~]# cat /etc/rinetd.conf 
10.1.2.41   7001    192.168.77.84   7001
10.1.2.41   7002    192.168.77.84   7002
10.1.2.41   7003    192.168.77.84   7003
10.1.2.41   7004    192.168.77.84   7004
10.1.2.41   7005    192.168.77.84   7005
10.1.2.41   7006    192.168.77.84   7006
10.1.2.41   7007    192.168.77.84   7007
  • 启动命令：/root/rinetd/rinetd -c /etc/rinetd.conf
  • 2、中转服务器2：192.168.77.84 安装 rinetd
代码语言：javascript复制
[root@vm-l2f-umicen-app-7784 ~]# cat /etc/rinetd.conf 
192.168.77.84   7001    192.168.54.62   82
192.168.77.84   7002    192.168.47.136  8080
192.168.77.84   7003    192.168.42.169  8080
192.168.77.84   7004    192.168.47.202  8080
192.168.77.84   7005    192.168.47.37   8080
192.168.77.84   7006    192.168.42.148  8080
192.168.77.84   7007    192.168.43.41   8080
  • 启动命令：/root/rinetd/rinetd -c /etc/rinetd.conf

*3 rinetd 重启动脚本

代码语言：javascript复制
#!/bin/bash 
source /etc/profile
kill -9 $(ps -ef |grep "/root/rinetd/rinetd"  |grep -v "grep"  |grep -v bash|awk '{print $2}')
ps aux |grep "/root/rinetd/rinetd"
sleep 1
/root/rinetd/rinetd -c /etc/rinetd.conf
  • 3、 使用haproxy进行TCP端口转发，更加高效稳定(此处给去快速安装脚本),中转配置需自己进行修改（我们此处采用的是rinetd，简单，但是网络转发久了。不稳定）
代码语言：javascript复制
#!/bin/bash
source /etc/profile
yum -y install haproxy
cd /etc/haproxy
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup
cat >> /etc/haproxy/harpoxy.cfg << EOF
global
        ulimit-n  51200
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        pidfile /var/run/haproxy.pid
        user haproxy
        group haproxy
        daemon

defaults
        log     global
        mode    tcp
        option  dontlognull
        timeout connect 600
        timeout client 5m
        timeout server 5m
frontend tcp-7001
        bind *:7001
        default_backend tcp-port-7001
backend tcp-port-7001
        server server1 192.168.77.84:7001 maxconn 20480


listen admin_stat                  
bind 0.0.0.0:1080        
mode http     
stats refresh 30s              
stats uri /haproxy_stats_url    
stats realm Haproxy Statistics 
stats auth admin:123456
stats hide-version              
stats admin if TRUE    

EOF

systemctl enable haproxy.service
systemctl start haproxy.service

三、辅助服务器运行部署：10.1.2.40

  • 1、安装
代码语言：javascript复制
wget https://github.com/session-replay-tools/intercept/archive/1.0.0.tar.gz
tar -xvf 1.0.0.tar.gz 
cd intercept-1.0.0/
./configure 
yum install libpcap* -y
make && make install
  • 2、执行
  • 命令案例：
代码语言：javascript复制
/usr/local/intercept/sbin/intercept -i ens33 -l /var/log/intercept.log -F tcp and src port 7001 -d
代码语言：javascript复制
-i, intercept会监听端口，和tcpcopy进行通信，-i就是指定监听在哪个端口。tcpcopy启动的时候会来连这个端口，如果连不上，就会启动失败。(erh0为网卡名)
-F, 过滤规则，语法和pcap一样。
-d, 已守护进程方式运行
  • 脚本：(由于我的环境过滤源端口比较多，直接监听过滤所有tcp协议，不指定端口)
代码语言：javascript复制
[root@vm-phx-k8s-master-0240 scripts]# cat /root/scripts/intercept.sh 
#!/bin/bash
souce /etc/profile
/usr/local/intercept/sbin/intercept -i ens33 -l /var/log/intercept.log -F tcp -d
  • 3、关闭转发功能，扮演黑洞功能
代码语言：javascript复制
修改/etc/sysctl.conf
net.ipv4.ip_forward=0

执行sysctl -p

四、线上服务器部署：10.1.3.74

  • 1、安装
代码语言：javascript复制
wget https://github.com/session-replay-tools/tcpcopy/archive/1.0.0.tar.gz
tar zxvf 1.0.0.tar.gz
cd tcpcopy-1.0.0
./configure  && make && make install
  • 2、执行
  • 脚本：tcpcopy.sh
代码语言：javascript复制
[root@online ~]# cat /root/tcpcopy/tcpcopy.sh 
#!/bin/bash
source /etc/profile
/usr/local/tcpcopy/sbin/tcpcopy  -x 7001-10.1.2.41:7001   -s 10.1.2.40  -c  10.1.5.x   -l /var/log/tcpcopy.log -d
  • 案例命令：
代码语言：javascript复制
测试命令为：/usr/local/tcpcopy/sbin/tcpcopy  -x 源端口-测试机器:测试端口 -s intercept机器地址 -c  模拟IP段   -l /var/log/tcpcopy.log
/usr/local/tcpcopy/sbin/tcpcopy  -x 7001-10.1.2.41:7001    -s 10.1.2.40 -c  10.1.5.x   -l /var/log/tcpcopy.log -d
代码语言：javascript复制
-x, 是指本机8000端口的流量copy到192.168.2.30的6001端口
-s, 指定intercept机器的地址，tcpcopy要和intercept建立连接
-c 伪装地址，在把流量复制到测试服务器的时候，修改数据包的源地址为10.1.5.2，这样方便指定路由。也可以写成10.1.5.x，这样源地址就是指定网段中的地址了。
-n 流量放大倍数，如果不是压测目的就不用指定这个参数。
-d 以守护模式运行

五、测试服务器，添加路由（我们当前是端口转发，测试服务器为10.1.2.41，realserver：192.168.54.62:82）

在10.1.2.41服务执行路由命令

代码语言：javascript复制
route add -net 10.1.5.0 netmask 255.255.255.0 gw 10.1.2.40

六、校验tcpcopy是否访问成功（通过观察线上的日志，和测试日志对比）

  • 1、检查线上服务器访问日志
代码语言：javascript复制
[root@online en]# pwd
/home/app/log/
[root@online en]# tail -n 10 en.log 
2020-10-30 10:34:10,755 [http-0.0.0.0:7001-1$2125991095] WARN com.ActionServlet [01] /eHome.do?xcase=index,908,661768202,39.88.110.24,00,TguNTYuMTUwLjE0MjAyMDEwMjIxMDE1MTI3NzE4NTQ5NTAzOAN,TExNDAwMjM0NzA3NDg4MjQ6MzkuODguMTEwLjI0OjY2MTc2ODIwMjowMAM,jAyMDEwMzAxMDM0MDIzNDkwMDA6MTk0MTk3MDMwODE4MTUyNjU3NDQM
2020-10-30 10:34:10,996 [http-0.0.0.0:7001-16$580927107] WARN com.ActionServlet [01] /ajaxfunction.do?xcase=ajaxlogonconnection&t=1604025241961&_=1604025241962,23,616549844,103.116.47.92,7403105,TAzLjExNi40Ny45MjIwMjAxMDIzMTA0NzQwODI5OTkyMjU2NzgM,TExMzk1MzY1MzcxMzA3Nzc6MTAzLjExNi40Ny45Mjo2MTY1NDk4NDQ6NzQwMzEwNQM,jAyMDEwMzAxMDMzMjA1NTIwMDA6MDExNTkzOTQ3MDAzNjA3NzM0MDkM
2020-10-30 10:34:23,375 [http-0.0.0.0:7001-4$1345458085] WARN com.ActionServlet [01] /iar.do?catId=5&xcase=index,66,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,621 [http-0.0.0.0:7001-3$1863279517] WARN com.ActionServlet [01] /eHomeHotActivity.do?xcase=index,191,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,691 [http-0.0.0.0:7001-1$2125991095] WARN com.ActionServlet [01] /eHomeExcellentService.do?xcase=index,24,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,752 [http-0.0.0.0:7001-16$580927107] WARN com.ActionServlet [01] /eHomeOnlineCourse.do?xcase=index,17,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,815 [http-0.0.0.0:7001-13$1113166511] WARN com.ActionServlet [01] /eHomeOnlineCourse.do?demandTypeName=vod&pageNumber=1&xcase=list,19,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:38,866 [http-0.0.0.0:7001-15$38163862] WARN com.ActionServlet [01] /eHome.do?xcase=sucStoryDetail&pasId=heKnNEDdHxlA,4896,616274054,112.232.32.190,3991982,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
2020-10-30 10:34:40,557 [http-0.0.0.0:7001-14$82426960] WARN com.ActionServlet [01] /ajaxfunction.do?xcase=ajaxlogonconnection&time=1604025279782,22,616274054,112.232.32.190,3991982,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
2020-10-30 10:34:43,619 [http-0.0.0.0:7001-2$405165860] WARN com.ActionServlet [01] /ajaxfunction.do?xcase=ajaxlogonconnection&t=1604025282858&_=1604025282859,9,616274054,112.232.32.190,3991982,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
  • 2、检查测试服务器检查日志
代码语言：javascript复制
[root@515f0f3d2b8f log]# tail -n 10  appen_visiting.log 
2020-10-30 10:34:10,999 [http-0.0.0.0:82-24$967804219] WARN appen_visiting /ajaxfunction.do?xcase=ajaxlogonconnection&t=1604025241961&_=1604025241962,9,0,103.116.47.92,00,TAzLjExNi40Ny45MjIwMjAxMDIzMTA0NzQwODI5OTkyMjU2NzgM,TExMzk1MzY1MzcxMzA3Nzc6MTAzLjExNi40Ny45Mjo2MTY1NDk4NDQ6NzQwMzEwNQM,jAyMDEwMzAxMDMzMjA1NTIwMDA6MDExNTkzOTQ3MDAzNjA3NzM0MDkM
2020-10-30 10:34:23,414 [http-0.0.0.0:82-22$1268241439] WARN appen_visiting /iar.do?catId=5&xcase=index,85,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,499 [http-0.0.0.0:82-22$1268241439] WARN appen_visiting /eHomeHotActivity.do?xcase=index,48,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,731 [http-0.0.0.0:82-27$1931835655] WARN appen_visiting /eHomeExcellentService.do?xcase=index,46,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,792 [http-0.0.0.0:82-24$1046816159] WARN appen_visiting /eHomeOnlineCourse.do?xcase=index,37,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:23,853 [http-0.0.0.0:82-25$1258927498] WARN appen_visiting /eHomeOnlineCourse.do?demandTypeName=vod&pageNumber=1&xcase=list,36,0,123.234.64.23,00,null,null,null
2020-10-30 10:34:34,005 [http-0.0.0.0:82-25$1268241439] WARN appen_visiting /eHome.do?xcase=sucStoryDetail&pasId=heKnNEDdHxlA,12,0,112.232.32.190,00,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
2020-10-30 10:34:40,561 [http-0.0.0.0:82-17$1585363137] WARN appen_visiting /ajaxfunction.do?xcase=ajaxlogonconnection&time=1604025279782,9,0,112.232.32.190,00,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
2020-10-30 10:34:43,636 [http-0.0.0.0:82-21$1258927498] WARN appen_visiting /ajaxfunction.do?xcase=ajaxlogonconnection&t=1604025282858&_=1604025282859,8,0,112.232.32.190,00,TI0LjEzMy4yMjUuODEyMDIwMDcyMjE1NTAxMTk4NjU2MjY5ODU5M,TExNDAwMjEyMjEyOTk4MzI6MTEyLjIzMi4zMi4xOTA6NjE2Mjc0MDU0OjM5OTE5ODIM,jAyMDEwMzAxMDM0MDAxMDAwMDA6MDE3MDcyMzE3MjYyNzkwMDUyODAM
2020-10-30 10:34:44,518 [http-0.0.0.0:82-20$1908430742] WARN appen_visiting /eHome.do?xcase=index,121,0,168.70.105.82,00,jIxLjE3Ni4xNTQuMTY2MjAyMDA5MjYwODU4MTg4MDU4Mzk4NTY3MwM,TExMzY0NjMwOTM1NTY5NTk6MTQuMS4yOC40NDo2MTU0MDgyNDQ6MzQ5ODQ3MgM,jAyMDEwMzAwOTQwMzcxNTAwMDA6MDUwNjk4MDU0NDEzMzQyMTM1MTEM

七、ansible 批量脚本

代码语言：javascript复制
[root@app1 ansible]# cat hosts_temp
[tcpcopy]
10.1.3.74 ansible_python_interpreter=/usr/bin/python26
10.1.6.233
10.1.6.197
10.1.6.185
10.1.6.215
10.1.6.184
10.1.6.133
10.1.5.4
10.1.5.144
10.1.6.193
10.1.6.157
10.1.3.115 ansible_python_interpreter=/usr/bin/python26
10.1.6.190
10.1.6.208
10.1.6.211
10.1.6.214
192.168.70.232
192.168.3.78
10.1.6.172
192.168.70.148
192.168.77.211
10.1.6.126
10.1.6.162
代码语言：javascript复制
[root@app5 ~]# cat /root/tcpcopy/tcpcopy.sh 
#!/bin/bash
source /etc/profile
/usr/local/tcpcopy/sbin/tcpcopy  -x 7001-10.1.2.41:7001   -s 10.1.2.40  -c  10.1.5.x   -l /var/log/tcpcopy.log -d
代码语言：javascript复制
ansible -i hosts_temp tcpcopy -m shell -a 'pkill tcpcopy'
ansible -i hosts_temp tcpcopy -m shell -a 'sh -x /root/tcpcopy/tcpcopy.sh'
测试服务 腾讯云测试服务 压力测试

0 人点赞