大key的危害
熟悉redis的朋友都知道，大key是在应用的设计和实践当中应该尽量避免的风险。大key的危险有很多，例如：
- 导致流量达到瓶颈
- 造成分片容量的倾斜
- 更严重的是读写大key导致系统无响应，产生大量慢查询等等
一般我们认为超过10kb大小的string类型的key或者超过1万个元素的set可以称为大key，应该进行合理的拆分
大key的获取
如何获取大key对于使用云数据库的朋友们来说，是比较轻松的一件事情，因为很多云厂商都默认提供了大key的分析工具，例如腾讯云数据库在【控制台】【系统监控】【监控概览】页面提供了大key分析功能，其原理是分析静态RDB文件然后从中抓出大key按照大小顺序排序，本文的重点是分享下如何使用开源工具rdbtools进行大key分析
rdbtools的安装
rdbtools有三个主要的功能
- 分析静态rdb文件并生成csv格式的内存报告
- 将rdb文件转储成为json格式
- 利用diff工具比较两个rdb文件的不同 下面我们开始
1.第一步我们先安装python和pip
由于作者操作系统使用的是centos8.0，默认提供了python3和pip3因此无需额外安装，读者可以自行安装，网上教程很多，不再赘述。
代码语言：txt复制>>> import sys
>>> sys.version
'3.6.8 (default, May 21 2019, 23:51:36) n[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]'
>>> 2.获取rdb文件
自建redis的同学可以使用redis-port工具集导出rdb文件。使用云数据库的朋友可以联系云厂商获取下载链接。
腾讯云redis控制台获取备份下载链接直接导出到本地或者使用rz/sz导入到自己的机器中
[root@VM-4-10-centos ~]# yum install lrzsz
Last metadata expiration check: 0:10:21 ago on Thu 18 Mar 2021 04:31:25 PM CST.
Dependencies resolved.
========================================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================================
Installing:
lrzsz x86_64 0.12.20-43.el8 BaseOS 84 k
Transaction Summary
========================================================================================================================================================================================
Install 1 Package
Total download size: 84 k
Installed size: 190 k
Is this ok [y/N]: y
Downloading Packages:
lrzsz-0.12.20-43.el8.x86_64.rpm 2.2 MB/s | 84 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.1 MB/s | 84 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : lrzsz-0.12.20-43.el8.x86_64 1/1
Running scriptlet: lrzsz-0.12.20-43.el8.x86_64 1/1
Verifying : lrzsz-0.12.20-43.el8.x86_64 1/1
Installed:
lrzsz-0.12.20-43.el8.x86_64
Complete!3.安装rdbtools
代码语言：txt复制[root@VM-4-10-centos ~]# pip install rdbtools安装好之后通过help命令简单了解下，注意--command or -c是必选参数
[root@VM-4-10-centos ~]# rdb --help
usage: usage: rdb [options] /path/to/dump.rdb
Example : rdb --command json -k "user.*" /var/redis/6379/dump.rdb //表示从dump.rdb文件中分析出所有以`user.`开头的key并输出为jason格式，打印到屏幕上
positional arguments: //位置参数
dump_file RDB Dump file to process
optional arguments:
-h, --help show this help message and exit//查看命令的帮助信息
-c CMD, --command CMD//必选参数，-c json表示存储成json格式，最常用的是 -c memory表示生成csv格式的内存快照，还有diff模式进行对比
Command to execute. Valid commands are json, diff,
justkeys, justkeyvals, memory and protocol
-f FILE, --file FILE //输出到文件
Output file
-n DBS, --db DBS //指定只分析某个编号的db，例如db0，如果不指定默认包含所有
Database Number. Multiple databases can be provided.
If not specified, all databases will be included.
-k KEYS, --key KEYS //指定分析并导出某个前缀的key，支持正则表达式
Keys to export. This can be a regular expression
-o NOT_KEYS, --not-key NOT_KEYS //指定不分析并导出某个前缀的key，支持正则表达式
Keys Not to export. This can be a regular expression
-t TYPES, --type TYPES //指定分析哪些数据类型，例如string、hash、set、list、sortedset等，可以指定多个类型，如果不指定默认分析所有
Data types to include. Possible values are string,
hash, set, sortedset, list. Multiple typees can be
provided. If not specified, all data types will be
returned
-b BYTES, --bytes BYTES //指定输出超过多少Byte的key，例如--bytes 10240表示只输出超过10k大小的key
Limit memory output to keys greater to or equal to
this value (in bytes)
-l LARGEST, --largest LARGEST //输出TOP多少的key，例如--largest 100，表示只输出top 100的key
Limit memory output to only the top N keys (by size)
-e {raw,print,utf8,base64}, --escape {raw,print,utf8,base64} //指定输出的默认编码，默认RAW
Escape strings to encoding: raw (default), print,
utf8, or base64.
-x, --no-expire With protocol command, remove expiry from all keys // 只有command为protocol模式才有效，表示不输出所有的可过期的key，只输出哪些永不过期的key
-a N, --amend-expire N // 只有command为protocol模式才有效，表示给设置过期时间的key增加N秒的过期时间
With protocol command, add N seconds to key expiry
time4.接下来我们简单分析下
代码语言：txt复制[root@VM-4-10-centos ~]# rdb --command json dump.rdb > ./test
WARNING: python-lzf package NOT detected. Parsing dump file will be very slow unless you install it. To install, run the following command:
pip install python-lzf代码语言：txt复制这里提示我们由于没有python-lzf包导致分析速度很慢，那我们就安装一下吧。可以先更新一下yum源，防止找不到包
[root@VM-4-10-centos ~]# yum update然后再次安装
代码语言：txt复制[root@VM-4-10-centos ~]# pip install python-lzf
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Collecting python-lzf
Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz
Installing collected packages: python-lzf
Running setup.py install for python-lzf ... error
Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-go178sdk/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6oauj90r-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'lzf' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I. -I/usr/include/python3.6m -c lzf_module.c -o build/temp.linux-x86_64-3.6/lzf_module.o -Wall
gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-go178sdk/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6oauj90r-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-go178sdk/python-lzf/代码语言：txt复制这里报错gcc编译的时候报错了，上github搜索了一下，是没有安装
redhat-rpm-config，那我们就安装下
$ sudo dnf install redhat-rpm-config代码语言：txt复制安装好
redhat-rpm-config之后，再安装python-lzf又报错了，这次提示少了Python.h，上网查了下，原来这次是缺乏python-devel。
[root@VM-4-10-centos ~]# pip install python-lzf
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Collecting python-lzf
Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz
Installing collected packages: python-lzf
Running setup.py install for python-lzf ... error
Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-dxcuwlzv/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9n6nf8r_-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'lzf' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I. -I/usr/include/python3.6m -c lzf_module.c -o build/temp.linux-x86_64-3.6/lzf_module.o -Wall
lzf_module.c:3:10: fatal error: Python.h: No such file or directory
#include "Python.h"
^~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-dxcuwlzv/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9n6nf8r_-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-dxcuwlzv/python-lzf/代码语言：txt复制我们直接yum安装可以找不到，例如下面所示，我们尝试搜索一下
yum search一下，找到了，python36-devel.x86_64,然后再次安装。
[root@VM-4-10-centos ~]# yum install python-devel
Last metadata expiration check: 0:00:58 ago on Thu 18 Mar 2021 05:14:53 PM CST.
No match for argument: python-devel
Error: Unable to find a match: python-devel[root@VM-4-10-centos ~]# yum search python3 | grep devel
Last metadata expiration check: 0:01:29 ago on Thu 18 Mar 2021 05:14:53 PM CST.
python3-qscintilla-qt5-devel.noarch : Development files for QScintilla-qt5 python3 bindings
boost169-mpich-python3-devel.x86_64 : Shared library symbolic links for Boost.MPI Python 3 component
boost169-openmpi-python3-devel.x86_64 : Shared library symbolic links for Boost.MPI Python 3 component
boost169-python3-devel.x86_64 : Shared object symbolic links for Boost.Python 3
python3-TurboGears2.noarch : Next generation front-to-back web development megaframework
python3-cherrypy.noarch : Pythonic, object-oriented web development framework
python3-hupper.noarch : Integrated process monitor for developing servers
python3-idle.i686 : A basic graphical development environment for Python
python3-idle.x86_64 : A basic graphical development environment for Python
python3-kobo.noarch : Python modules for tools development
python3-pycxx-devel.noarch : PyCXX header and source files
python3-werkzeug.noarch : The Swiss Army knife of Python web development
python36-devel.x86_64 : Libraries and header files needed for Python development
python38-devel.x86_64 : Libraries and header files needed for Python development
python38-idle.x86_64 : A basic graphical development environment for Python代码语言：txt复制搞定了这些依赖，我们再次安装就很顺利了
[root@VM-4-10-centos ~]# pip install python-lzf
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Collecting python-lzf
Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz
Installing collected packages: python-lzf
Running setup.py install for python-lzf ... done
Successfully installed python-lzf-0.2.45.正式分析
接下来给一个常用的命令，分析rdb文件当中top100的大key，可以使用lrzsz下载到本地，使用Excel进行分析。
root@VM-4-10-centos ~]# rdb -c memory dump-6120957-redis-server-ignore-140016731-ignore-1-ignore.rdb -l 100 -f ./dump_memory.csv
Excel分析内存快照CSV文件CSV中字段说明
database | type | key | size_in_bytes | encoding | num_elements | len_largest_element | expiry |
|---|---|---|---|---|---|---|---|
key所属db编号 | key的类型例如string list set等 | key名 | key大小(字节数) | 编码方式(hashtable，ziplist，string等) | key中元素的个数 | key中最大元素的长度 | 过期时间 |
也可以使用
LOAD DATA INFILE语句导入到数据库中，使用SQL语句进行分析，诸如查询总内存占用、查询总key个数、查询特定type的key个数等等。不再赘述
