1.文档编写目的
本篇文章主要介绍如何在CDP 7.1.6集群中配置Kerberos的高可用。
- 文档概述
1.如何在CDP7集群配置Kerberos高可用
2.验证
3.总结
- 测试环境
1.操作系统Redhat7.2
2.CDP7.1.6
3.使用root用户操作
2.备节点安装Kerberos服务
1.在备节点安装Kerberos服务，暂时不进行相关配置
[root@cdh1 ~]# yum install -y krb5-server openldap-clients krb5-workstation krb5-libs
3.主节点Kerberos操作
1.修改/etc/krb5.conf的配置文件，在realms配置下增加备Kerberos的配置
[realms] MACRO.COM = { kdc = cdh1.macro.com admin_server = cdh1.macro.com kdc = cdh2.macro.com admin_server = cdh2.macro.com }
2.将修改后的配置文件同步到集群所有Kerberos客户端对应目录
[root@cdh1 ~]# scp -rp /etc/krb5.conf 192.168.0.75:/etc/[root@cdh1 ~]# scp -rp /etc/krb5.conf 192.168.0.74:/etc/
3.保存配置，然后重启krb5kdc和kadmin服务
[root@cdh1 ~]# systemctl restart krb5kdc[root@cdh1 ~]# systemctl restart kadmin
4.创建主从同步账号，并为账号生成Keytab文件
Kadmin.local:addprinc -randkey host/cdh1.macro.comKadmin.local:addprinc -randkey host/cdh2.macro.comKadmin.local:ktadd host/cdh1.macro.comKadmin.local:ktadd host/cdh2.macro.com
使用随机生成秘钥的方式创建同步账号，并使用ktadd命令生成同步账号的keytab文件，默认文件生成在/etc/krb5.keytab下，生成多个账号则在krb5.keytab基础上追加。
5.复制以下文件到备Kerberos服务器相应目录。将/etc目录下的krb5.conf和krb5.keytab文件拷贝至备Kerberos服务器的/etc目录下，将/var/kerberos/krb5kdc目录下的.k5.CLOUDERA.COM、kadm5.acl和krb5.conf文件拷贝至备Kerberos服务器的/var/kerberos/krb5kdc目录下
4.备Kerberos节点操作
1.需要申明用来同步的用户，在/var/kerberos/krb5kdc/kpropd.acl配置文件中添加对应账户，如果配置文件不存在则新增
[root@cdh2 krb5kdc]# vim /var/kerberos/krb5kdc/kpropd.aclhost/cdh2.macro.com@MACRO.COMhost/cdh1.macro.com@MACRO.COM
2.启动kprop服务并加入系统自启动
[root@cdh2 krb5kdc]# systemctl enable kpropCreated symlink from /etc/systemd/system/multi-user.target.wants/kprop.service to /usr/lib/systemd/system/kprop.service.[root@cdh2 krb5kdc]# systemctl start kprop[root@cdh2 krb5kdc]# systemctl status kprop
备节点上已经准备好数据传输，接下来在主节点上使用kdb5_util将Kerberos库导出，然后通过kprop命令向备节点同步数据
5.节点数据同步至备节点
1.在主节点上使用krb5_util命令导出Kerberos数据库文件
[root@cdh1 krb5kdc]# kdb5_util dump /var/kerberos/krb5kdc/master.dump
导出成功后生成master.dump和master.dump.dump_ok两个文件
2.在主节点上使用kprop命令将master.dump文件同步至备节点
[root@cdh1 krb5kdc]# kdb5_util dump /var/kerberos/krb5kdc/master.dump[root@cdh1 krb5kdc]# kprop -f /var/kerberos/krb5kdc/master.dump -d -P 754 cdh2.macro.com32768 bytes sent.65536 bytes sent.69177 bytes sent.Database propagation to cdh2.macro.com: SUCCEEDED
3.在备节点的/var/kerberos/krb5kdc目录下查看
[root@cdh2 krb5kdc]# pwd/var/kerberos/krb5kdc[root@cdh2 krb5kdc]# ll
4.在备节点上测试同步过来的数据是否能启动Kerberos服务
[root@cdh2 krb5kdc]# systemctl start krb5kdc[root@cdh2 krb5kdc]# systemctl status krb5kdc
5.在备节点上验证kadmin服务是否正常
[root@cdh2 krb5kdc]# kadmin.localAuthenticating as principal flink/admin@MACRO.COM with password.kadmin.local: addprinc tets1WARNING: no policy specified for tets1@MACRO.COM; defaulting to no policyEnter password for principal "tets1@MACRO.COM":Re-enter password for principal "tets1@MACRO.COM":Principal "tets1@MACRO.COM" created.kadmin.local: listprinc
6.Kill主服务的krb5kdc服务和kadmin服务进行验证
[root@cdh1 krb5kdc]# ps -ef|grep krb5kdcroot 13391 1 0 11:17 ? 00:00:11 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pidroot 24764 25786 0 15:09 pts/0 00:00:00 grep --color=auto krb5kdc[root@cdh1 krb5kdc]# kill -9 13391[root@cdh1 krb5kdc]# ps -ef|grep kadminroot 13461 1 0 11:17 ? 00:00:00 /usr/sbin/kadmind -P /var/run/kadmind.pidroot 25009 25786 0 15:09 pts/0 00:00:00 grep --color=auto kadmin[root@cdh1 krb5kdc]# kill -9 13461
7.在备用服务器上服务依旧正常，可以正常添加凭证
[root@cdh2 krb5kdc]# kadmin.localAuthenticating as principal flink/admin@MACRO.COM with password.kadmin.local: addprinc test2WARNING: no policy specified for test2@MACRO.COM; defaulting to no policyEnter password for principal "test2@MACRO.COM":Re-enter password for principal "test2@MACRO.COM":Principal "test2@MACRO.COM" created.kadmin.local: listprincs
8.在其他客户端节点登录刚新增的用户正常
[root@cdh3 ~]# kinit test2Password for test2@MACRO.COM:[root@cdh3 ~]# klistTicket cache: FILE:/tmp/krb5cc_0Default principal: test2@MACRO.COM
Valid starting Expires Service principal06/01/2021 15:19:18 06/02/2021 15:19:18 krbtgt/MACRO.COM@MACRO.COM renew until 06/08/2021 15:19:18
6.验证
6.1Principal Account验证
因为前面的配置安装，已经将主节点的kdc和kadmin服务挂掉，所以直接进行验证
1.进入CM>管理>安全>Kerberos凭据>导入Kerberos Account Manager凭据
2.勾选I have completed all the above steps并继续
3.设置主机为挂掉的节点并继续
4.输入Kerberos管理员账号和密码并继续
5.导入凭据正常并且集群各组件正常运行
在服务挂掉的节点进行生成Keytab操作
[root@cdh1 krb5kdc]# kadmin.localAuthenticating as principal test2/admin@MACRO.COM with password.kadmin.local: ktadd test2kadmin.local: Principal test2 does not exist.
6.2在CM中进行生成Keytab操作
1.在CM中进行生成Keytab操作之前需要停止集群
2.进入CM>群集>主机界面
3.选择主节点和备节点，并点击已选定操作>重新生成Keytab
4.重新生成Keytab成功，启动各项服务并查看是否正常运行
6.3在备节点命令行进行生成Keytab操作
[root@cdh2 krb5kdc]# kadmin.localAuthenticating as principal cm/admin@MACRO.COM with password.kadmin.local: ktadd test2Entry for principal test2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab.Entry for principal test2 with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab.
6.4Kdc和kadmin服务宕掉一个后，是否会影响正在运行的集群作业
1.在主启动kdc和kadmin服务,备节点启动kdc
[root@cdh1 krb5kdc]# systemctl start krb5kdc[root@cdh1 krb5kdc]# systemctl start kadmin[root@cdh1 krb5kdc]# systemctl status krb5kdc[root@cdh1 krb5kdc]# systemctl status kadmin
[root@cdh2 krb5kdc]# systemctl start krb5kdc[root@cdh2 krb5kdc]# systemctl status krb5kdc
2.在主节点向hdfs上传文件，并在上传过程中kill掉kdc服务
[root@cdh1 cdh5.16.2]# hdfs dfs -put CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel /test/
[root@cdh1 ~]# ps -ef|grep krb5kdc[root@cdh1 ~]# ps -ef|grep krb5kdc[root@cdh1 ~]# kill -9 9210[root@cdh1 ~]# ps -ef|grep krb5kdc
上传无异常返回，查看hdfs/test目录下文件是否上传成功（成功上传）
[root@cdh1 cdh5.16.2]# hdfs dfs -ls /test/
3.在主节点向hdfs上传文件，并在上传过程中kill掉kadmin服务
[root@cdh1 cm5.16.2]# hdfs dfs -put cloudera-manager-daemons-5.16.2-1.cm5162.p0.7.el7.x86_64.rpm /test/
[root@cdh1 ~]# ps -ef|grep kadmin[root@cdh1 ~]# kill -9 18517[root@cdh1 ~]# ps -ef|grep kadmin
上传无异常返回，查看hdfs/test目录下文件是否上传成功（上传成功）
[root@cdh1 cm5.16.2]# hdfs dfs -ls /test/
6.5Kdc和kadmin服务宕掉一个后，是否会影响新提交的集群作业
1.Kill掉主节点等的kdc服务并启动kadmin服务
[root@cdh1 cm5.16.2]# systemctl start kadmin[root@cdh1 cm5.16.2]# systemctl status kadmin
[root@cdh1 cm5.16.2]# ps -ef|grep krb5kdc[root@cdh1 cm5.16.2]# kill -9 17033[root@cdh1 cm5.16.2]# ps -ef|grep krb5kdc[root@cdh1 cm5.16.2]# systemctl status krb5kdc
2.向hdfs提交一个上传的任务并查看是否上传成功（成功）
[root@cdh1 cm5.16.2]# hdfs dfs -put oracle-j2sdk1.7-1.7.0 update67-1.x86_64.rpm /test/[root@cdh1 cm5.16.2]# hdfs dfs -ls /test
3.Kill掉主节点的kadmin服务并启动kdc服务
[root@cdh1 cm5.16.2]# systemctl start krb5kdc[root@cdh1 cm5.16.2]# systemctl status krb5kdc
[root@cdh1 cm5.16.2]# ps -ef|grep kadmin[root@cdh1 cm5.16.2]# kill -9 23523[root@cdh1 cm5.16.2]# ps -ef|grep kadmin
4.向hdfs提交一个上传任务并查看任务是否成功上传
[root@cdh1 cm5.16.2]# hdfs dfs -put cloudera-manager-daemons-5.16.2-1.cm5162.p0.7.el7.x86_64.rpm /test/[root@cdh1 cm5.16.2]# hdfs dfs -ls /test/6.6于6.2/10:40宕掉主节点kdc服务，第二天查看服务是否正常
1.宕掉kdc服务
[root@cdh1 ~]# ps -ef|grep krb5kdc[root@cdh1 ~]# kill -9 26831[root@cdh1 ~]# ps -ef|grep krb5kdc[root@cdh1 ~]# systemctl status krb5kdc
2.检查集群运行状况
3.集群作业测试
[root@cdh1 cdh5.16.2]# hdfs dfs -put CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel /test/[root@cdh1 cdh5.16.2]# hdfs dfs -ls /test/
7.总结
1.在集群中配置了Kerberos高可用后，kadmin和kdc服务挂掉了不会影响导入CM的principal Account操作，但是在主节点的命令行无法进行生成Keytab的操作，只能在备节点的命令行进行生成Keytab操作
2.在集群中配置了kerberos高可用后，kdc和kadmin服务宕掉一个之后，不会影响到集群作业的运行（正在运行的作业或者是新提交的作业都不受影响）
3.在集群中配置了kerberos高可用后，在宕掉主节点的kdc和kadmin服务后，对CM集群中进行生成Keytab操作，集群组件运行无异常
4.在集群中配置了kerberos高可用后，宕掉主节点的kdc服务长时间不会发生异常问题
