大家好,又见面了,我是你们的朋友全栈君。
前言:
debezium提供了多种基于kafka的连接器,方便对RDB做数据流处理,包括:MongoDB,Oracle,Mysql,SqlServer,Postgresql,可扩展性强,代码可控,本篇介绍基于mysql的安装使用
插件版本:
Kafka:CDK3.10 (相当于Kafka1.1版本),这里需要kafka 0.10以上版本才能支持
Debezium:0.83
Mysql:5.5 (mysql5.6版本前后会有一些差异,下面会提到)
安装步骤:
1、mysql (这里我们安装5.5版本)
apt-get install mariadb-server-5.5
2、mysql配置
修改/etc/mysql/my.cnf,添加如下内容:
server-id = 223344 (这个id对于debezium来说一定是要唯一的)
log_bin = mysql-bin
binlog_format = ROW
#binlog_row_image = FULL (这一行如果在mysql5.6之前的版本可以不用加,5.6之后的版本要加上,主要控制binlog记录完整性,full为全写)
expire_logs_days = 10
3、重启数据库,配置用户权限
CREATE USER debezium IDENTIFIED BY ‘debezium’;
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO ‘debezium’ IDENTIFIED BY ‘debezium’;
FLUSH PRIVILEGES;
关于权限的介绍:
SELECT – enables the connector to select rows from tables in databases; used only when performing a snapshot
RELOAD – enables the connector of the FLUSH statement to clear or reload various internal caches, flush tables, or acquire locks; used only when performing a snapshot
SHOW DATABASES – enables the connector to see database names by issuing the SHOW DATABASE statement; used only when performing a snapshot
REPLICATION SLAVE – enables the connector to connect to and read the binlog of its MySQL server; always required for the connector
REPLICATION CLIENT – enables the use of SHOW MASTER STATUS, SHOW SLAVE STATUS, and SHOW BINARY LOGS; always required for the connector
5、安装java环境
安装包准备:jdk-8u144-linux-x64.tar.gz
解压java安装包到指定安装文件:tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr/lib/jvm/
配置环境变量,修改/etc/profile文件
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_144
export PATH=PATH:JAVA_HOME/bin
export CLASSPATH=.:CLASSPATH:JAVA_HOME/lib:
使配置环境变量生效: source /etc/profile
测试是否安装成功
java –version
6、Debezium安装
下载:weget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/0.8.3.Final/debezium-connector-mysql-0.8.3.Final-plugin.tar.gz
解压压缩包:tar -zxvf debezium-connector-mysql-0.8.3.Final-plugin.tar.gz
把debezium-connector-mysql下所有的jar包复制到kafka的lib下:cp *.jar /opt/cloudera/parcels/KAFKA-3.1.0-1.3.1.0.p0.35/lib/kafka/libs
发布者:全栈程序员栈长,转载请注明出处:https://javaforall.cn/159559.html原文链接:https://javaforall.cn