本篇将在mac 上学习k8s系列（8）external auth的基础上基于nginx-ingress 的access_by_lua_block redis 来实现一个全局的rate limiter：用nginx lua连接redis，用redis计数来做集群粒度的rate-limit。
ngx_lua 模块提供了配置指令 access_by_lua，用于在 access 请求处理阶段插入用户 Lua 代码。这条指令运行于 access 阶段的末尾，因此总是在 allow 和 deny 这样的指令之后运行，虽然它们同属 access 阶段。一般我们通过 access_by_lua 在 ngx_access 这样的模块检查过客户端 IP 地址之后，再通过 Lua 代码执行一系列更为复杂的请求验证操作，比如实时查询数据库或者其他后端服务，以验证当前用户的身份或权限。
首先，我们启动一个简单的redis服务，一般redis、mysql等有状态的服务，我们一般通过pv使用nfs服务存储，并且使用使用StatefulSet创建redis-cluster集群节点和headless service，保证每个pod的ip不变。本文重点介绍如何使用access_by_lua_block，所以搭建一个简单的无状态的redis。
创建一个ConfigMap来保存redis的配置：
代码语言：javascript复制---
kind: ConfigMap
apiVersion: v1
metadata:
name: redis-config
labels:
app: redis
data:
redis.conf: |-
dir /srv
port 6379
bind 0.0.0.0
appendonly yes
daemonize no
#protected-mode no
requirepass 123456
pidfile /srv/redis-6379.pid然后通过DeployMent启动redis的pod
代码语言：javascript复制---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
command:
- "sh"
- "-c"
- "redis-server /usr/local/redis/redis.conf"
ports:
- containerPort: 6379
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 1000m
memory: 1024Mi
livenessProbe:
tcpSocket:
port: 6379
initialDelaySeconds: 300
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 6379
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
volumeMounts:
- name: config
mountPath: /usr/local/redis/redis.conf
subPath: redis.conf
volumes:
- name: config
configMap:
name: redis-config启动redis服务
代码语言：javascript复制apiVersion: v1
kind: Service
metadata:
name: redis
labels:
app: redis
spec:
type: NodePort
ports:
- name: tcp
port: 6379
nodePort: 30379
selector:
app: redis测试下redis服务是否正常启动了：
代码语言：javascript复制 %redis-cli -h 127.0.0.1 -p 30379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:30379>接着，我们创建Ingress
代码语言：javascript复制apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-with-auth
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
access_by_lua_block {
local header = ngx.req.get_headers()
if header.token then
local LIMIT = 100
local DELAY = 10
local red = require "resty.redis"
local redis = red:new()
redis:set_timeout(1000)
local ok, err = redis:connect("redis.default.svc.cluster.local", 6379)
if not ok then
ngx.status = 500
ngx.say("<h1>系统开小差了</h1>",err)
return
end
local res, err = redis:auth("123456")
if not res then
ngx.status = 500
ngx.say("<h1>系统开小差了</h1>")
return
end
local now = ngx.now()
local ok, err = redis:eval('local oldest = redis.call("lindex", ARGV[1], -1);if oldest then if redis.call("llen", ARGV[1]) >= tonumber(KEYS[1]) then if (ARGV[2] - oldest) < tonumber(KEYS[2]) then return nil end end end;redis.call("lpush", ARGV[1], ARGV[2]);redis.call("expire", ARGV[1], KEYS[1]);redis.call("ltrim", ARGV[1], 0, KEYS[1]); return 1', 2, LIMIT-1, DELAY, "limit:"..ngx.md5(header.token), now)
if ok ~= 1 then
ngx.status = 519
ngx.say("<h1>系统繁忙</h1>")
redis:set_keepalive(10000, 100)
return
end
redis:set_keepalive(10000, 100)
end
}
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: apple-service
port:
number: 5678我们在注解的nginx.ingress.kubernetes.io/server-snippet: 段插入了lua代码：access_by_lua_block，通过openresty的redis 包来连接redis：
代码语言：javascript复制local red = require "resty.redis"
local redis = red:new()
redis:set_timeout(1000)
local ok, err = redis:connect("redis.default.svc.cluster.local", 6379)通过执行redis lua脚本来进行限流
代码语言：javascript复制local now = ngx.now()
local ok, err = redis:eval('
local oldest = redis.call("lindex", ARGV[1], -1);
if oldest
then if redis.call("llen", ARGV[1]) >= tonumber(KEYS[1])
then if (ARGV[2] - oldest) < tonumber(KEYS[2])
then return nil
end
end
end;
redis.call("lpush", ARGV[1], ARGV[2]);
redis.call("expire", ARGV[1], KEYS[1]);
redis.call("ltrim", ARGV[1], 0, KEYS[1]);
return 1',
2, LIMIT-1, DELAY, "limit:"..ngx.md5(header.token), now)通过滑动窗口算法，做用户token维度的限流。
首先我们测试下
代码语言：javascript复制kubectl apply -f ingress.yaml
curl -H "token:12344" 127.0.0.1/apple
/apple一个请求的情况下正常工作了，我们来压测下
代码语言：javascript复制% ab -n 1000 -c 100 -H "token:12344" 127.0.0.1/apple
Percentage of the requests served within a certain time (ms)
50% 54
66% 77
75% 90
80% 99
90% 122
95% 145
98% 189
99% 212
100% 258 (longest request)100个并发是正常工作的，我们换成200个呢
代码语言：javascript复制� -n 1000 -c 200 -H "token:12344" 127.0.0.1/apple
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)
apr_socket_recv: Connection reset by peer (54)
Total of 1 requests completed限流量，至此我们完成了access_by_lua_block redis 全局的rate limiter的功能。其中也遇到了各种问题，下一节给大家介绍分析思路。
