「EMR 运维指南」之 Impala 关联 Sentry + Hue

2023-11-23 12:50:08 浏览数 (1)

说明

本文描述问题及解决方法同样适用于 弹性 MapReduce（EMR）。

背景

实现 impala 权限认证，并且需要支持 hue 控制台作业权限控制。

Impala 与 Sentry 权限管控部署

操作流程

1. 环境说明

在EMR集群带公网master节点部署sentry服务，Impala的GROUP组需要在Impala server节点上进行创建系统组与之关联。

2. 下载 sentry

sentry 下载及 解压且改名移动到/usr/local/service/hadoop/ 及拷贝 mysql-jar包

代码语言：javascript复制
wget https://mirror.bit.edu.cn/apache/sentry/2.1.0/apache-sentry-2.1.0-bin.tar.gz
tar -xf apache-sentry-2.1.0-bin.tar.gz
mv apache-sentry-2.1.0 /usr/local/service/hadoop/sentry
cp -ra /usr/local/service/hive/lib/mysql-connector-java-5.1.40-bin.jar
/usr/local/service/hadoop/sentry/lib/
chown -R  hadoop.hadoop /usr/local/service/hadoop/sentry

3. 修改sentry的配置文件

/usr/local/service/hadoop/sentry/conf/sentry-site.xml

sentry.service.admin.group

hadoop

sentry.service.allow.connect

hadoop

sentry.service.client.server.rpc-addresses

Server-IP主机名（例如10.0.3.42）

sentry.store.jdbc.user

DB用户

sentry.store.jdbc.password

DB密码

sentry.store.jdbc.url

jdbc:mysql://IP:PORT/DATABASE?useSSL=false&createDatabaseIfNotExist=true&characterEncoding=UTF-8 (请更改DATABASE?useSSL=false&createDatabaseIfNotExist=true&characterEncoding=UTF-8 (请更改IP/PORT/DATABASE)

sentry.service.client.server.rpc-port

8038

sentry.service.client.server.rpc-connection-timeout

200000

sentry.service.security.mode

trusted (不开启kerberos)

sentry.store.jdbc.driver

com.mysql.jdbc.Driver

sentry.verify.schema.version

true

4. 修改Impala配置文件 impalad.flgs catalogd.flgs

代码语言：javascript复制
/data/Impala/conf/catalogd.flgs  文本追加一行 
-sentry_config=/usr/local/service/hadoop/sentry/conf/sentry-site.xml
/data/Impala/conf/impalad.flgs   文本追加两行 
-sentry_config=/usr/local/service/hadoop/sentry/conf/sentry-site.xml
-server_name=sentryserver

5. sentry包分发到各个节点(除了EMR-common节点),且授权

代码语言：javascript复制
scp -r /usr/local/service/hadoop/sentry $IP:/usr/local/service/hadoop
ssh $IP "chown -R hadoop.hadoop /usr/local/service/hadoop/sentry"

6. Impala配置文件分发各个节点(除了EMR-common节点),且授权

代码语言：javascript复制
scp -r /data/Impala/conf/* $IP:/data/Impala/conf/
ssh $IP "chown -R hadoop.hadoop /data/Impala/conf/"

7. 初始化DB库

代码语言：javascript复制
su hadoop
cd /usr/local/service/hadoop/sentry/
./bin/sentry --command schema-tool --conffile  ./conf/sentry-site.xml --dbType mysql --initSchema

8. 控制台重启Impala三个服务

代码语言：javascript复制
Impala-Catlog Impala-Server Impala-StateStore

9. 启动sentry服务(sentry.service.client.server.rpc-addresses 该节点启动)

代码语言：javascript复制
su - hadoop
nohup /usr/local/service/hadoop/sentry/bin/sentry --command service --conffile  /usr/local/service/hadoop/sentry/conf/sentry-site.xml &  #挂后台进程

10. 验证

代码语言：javascript复制
1.sentryserver端创建用户test
    useradd test
id test  （test默认在test组下）
 
2.连接impala
    su hadoop 
    cd /data/Impala/
    ./bin/impala-shell.sh -i 10.0.3.42:27001 -uhadoop
 
3.创建测试DB库
    create database impala_test;
    
4.创建角色并关联Group（即test组）
    CREATE ROLE test;
    GRANT ROLE test TO GROUP test;
    GRANT SELECT ON DATABASE impala_test TO ROLE test;
    
5.切换用户及验证
    su hadoop 
    cd /data/Impala/
    ./bin/impala-shell.sh -i 10.0.3.42:27001 -utest  
    show database;  #是否存在impala_test
    use impala_test
    create table t1(id int);

11. 定时任务

代码语言：javascript复制
vim /usr/local/service/hadoop/sentry/scripts/check_sentry.sh
# !/bin/bash
source /etc/profile
export PATH=/usr/local/service/hadoop/sentry/bin/:$PATH
flag=`ps -ef |grep "sentry-tools-2.1.0.jar" |grep -v "grep"|wc -l`
if [[ $flag  -eq  0 ]]
then
sentry --command service --conffile /usr/local/service/hadoop/sentry/conf/sentry-site.xml &
fi
代码语言：javascript复制
chown hadoop.hadoop -R /usr/local/service/hadoop/sentry/scripts/check_sentry.sh
chmod a x /usr/local/service/hadoop/sentry/scripts/check_sentry.sh
 
echo "*/1 * * * * su hadoop -c 
'/bin/bash /usr/local/service/hadoop/sentry/scripts/check_sentry.sh > /dev/null 2>&1 &'" >> /var/spool/cron/root

12. 相关命令

代码语言：javascript复制
1.IMPALA权限管控设置常用命令
创建角色：CREATE ROLE <role name>
组分配角色：GRANT ROLE <role name> TO GROUP <group name> 
服务级赋权：GRANT <ALL|SELECT|UPDATE> ON SERVER <server name> TO ROLE <role name> 
数据库赋权：GRANT <ALL|SELECT|UPDATE> ON DATABASE <database name> TO ROLE <role name> 
表赋权：GRANT <ALL|SELECT|UPDATE> ON TABLE <database name> <table name> TO ROLE <role name>
 
2.权限管控查看命令
SHOW ROLES ;
SHOW CURRENT ROLES ;
SHOW ROLE GRANT GROUP <group name> ;
SHOW GRANT ROLE <role name> ;
SHOW GRANT ROLE <role name> on OBJECT <object name> ;

Impala Sentry Hue

操作流程

该sentry开源包包：https://www.apache.org/dyn/closer.cgi/sentry/2.1.0

1. 开启hue的impala插件支持

代码语言：javascript复制
/usr/local/service/hue/tools/app_reg/app_reg.py --install /usr/local/service/hue/apps/impala

/usr/local/service/hue/tools/app_reg/app_reg.py --install /usr/local/service/hue/apps/security

2. 编辑hue 配置文件，开启impala sentry相关配置

代码语言：javascript复制
vim /usr/local/service/hue/desktop/conf/pseudo-distributed.ini

删除上述红框中的配置，填写impala 相关配置如下图：

impersonation_enabled = True 开启模拟用户

配置sentry相关配置

上述已完成hue相关配置

3. 整合 sentry 服务：

我这里提供我这边整理的配置： sentry-site.xml

代码语言：javascript复制
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
      <name>sentry.service.server.rpc-address</name>
      <value>172.21.0.99</value>
  </property>

  <property>
      <name>sentry.service.server.rpc-port</name>
      <value>8038</value>
  </property>

  <property>
      <name>sentry.service.admin.group</name>
      <value>hadoop</value>
  </property>

  <property>
      <name>sentry.service.allow.connect</name>
      <value>hadoop</value>
  </property>

  <property>
      <name>sentry.store.group.mapping</name>
      <value>org.apache.sentry.provider.common.HadoopGroupMappingService</value>
  </property>
      
  <property>
      <name>sentry.service.reporting</name>
      <value>JMX</value>
  </property>

  <property>
      <name>sentry.verify.schema.version</name>
      <value>true</value>  
  </property>

  <property>
    <name>sentry.service.security.mode</name>
    <value>none</value>
  </property>

  <property>
    <name>sentry.store.jdbc.url</name>
    <value>jdbc:mysql://172.21.0.70:3306/sentrydb?useSSL=false&amp;createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8</value>
  </property>

  <property>
      <name>sentry.store.jdbc.driver</name>
      <value>com.mysql.jdbc.Driver</value>
  </property>

  <property>
      <name>sentry.store.jdbc.user</name>
      <value>root</value>
  </property>

  <property>
      <name>sentry.store.jdbc.password</name>
      <value>1234@1234</value>
  </property>

</configuration>

impala-sentry.xml

代码语言：javascript复制
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

    <property>
        <name>sentry.hive.provider.backend</name>
        <value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value>
    </property>

    <property>
        <name>sentry.hive.server</name>
        <value>server1</value>
    </property>

    <property>
        <name>sentry.service.client.server.rpc-addresses</name>
        <value>172.21.0.99</value>
    </property>

    <property>
        <name>sentry.service.client.server.rpc-connection-timeout</name>
        <value>200000</value>
    </property>

    <property>
        <name>sentry.service.client.server.rpc-port</name>
        <value>8038</value>
    </property>

    <property>
        <name>sentry.service.security.mode</name>
        <value>none</value>
    </property>

</configuration>

4. impala 相关配置：

代码语言：javascript复制
/data/Impala/conf/catalogd.flgs  文本追加一行
-sentry_config=/data/Impala/conf/impala-sentry.xml 
/data/Impala/conf/impalad.flgs   文本追加两行
-sentry_config=/data/Impala/conf/impala-sentry.xml 
-server_name=server1
-authorized_proxy_user_config=hadoop=*

以上就完成了sentry impala相关的配置了

注意：

hive 创建的库 在impala 里面不受权限控制

我正在参与2023腾讯技术创作特训营第三期有奖征文，组队打卡瓜分大奖！

2023腾讯·技术创作特训营第三期 弹性MapReduce emr impala sentry

0 人点赞