CDP私有云基础版审计信息到外部系统

2021-07-02 10:09:51 浏览数 (1)

Cloudera得到世界各地受管制的行业和政府组织的信任，可以存储和分析有关人、医疗保健数据、财务数据或仅对客户本身敏感的专有信息的PB级别的高度敏感或机密的信息。

任何存储客户信息、医疗保健、财务或敏感专有信息的人都需要确保他们正在采取措施保护该数据，包括检测并防止无意或恶意访问。根据Ponemon研究所的研究，全球内幕威胁的平均成本在两年内增长了31％，达到1,145万美元，而事件频发的频率在同一时期激增了47％。一份2019报告明确了公司对这些意外更担心：内幕泄露（71％）、疏忽数据泄露（65％）、以及恶意不良行为者（60％），这些比他们对损害的账号/机器（9％）的意外更担心。

GDPR、CCPA、HIPAA、PCI DSS和FIPS-200之类的法规均要求组织采取适当措施来保护敏感信息，这些措施可包括以下三个支柱：

  • 静态和动态加密-确保未经身份验证的参与者无法访问数据
  • 访问控制（强身份验证和授权）–确保用户就是他们所说的身份（身份验证），并且只能访问他们被允许访问的内容（授权）
  • 审计和核算–了解谁访问了什么内容、何时访问以及谁更改了权限或访问控制设置，并有可能在发生数据泄露时而不是在事发后发出警报。

在Cloudera数据平台中，我们擅长通过Cloudera共享数据体验（SDX）来提供端到端安全性。在CDP中：

  • 可以使用基于TLS或SASL的加密方式对所有有线协议进行加密
  • 可以使用HDFS透明数据加密（私有云）或对象存储加密（公共云）对所有静态数据进行加密
  • 在公共云和私有云中，所有用户访问均通过Kerberos / SPNEGO或SAML进行身份验证。
  • 所有数据访问均通过基于属性的访问控制或基于角色的访问控制（使用Apache Ranger作为SDX的一部分）进行授权。
  • 再次使用Apache Ranger审核所有数据访问和数据访问控件。

保护性监控

通过有效的保护性监控计划，公司可以确保他们可以了解谁正在访问或尝试访问整个IT领域中的哪些数据以及从哪些设备进行访问。这可以通过以下方式完成：

  • 合规性和报告–在谁在访问特定的数据资产之后的事实报告
  • 数字取证和事件响应–在发现违规行为后对监管机构或信息专员做出响应
  • 先进的威胁检测–实时监控访问事件，以识别用户级别，数据资产级别或跨系统的行为变化。某些SIEM平台（例如Securonix）包括这些类型的功能。

Cloudera数据平台中的审核

CDP中的所有数据访问组件都将审核事件发送到Apache Ranger，在其中存储它们并可以在可配置的保留期限内对其进行搜索。

在本博客中，我们将演示如何通过系统日志将这些审核事件流式传输到第三方SIEM平台，或者将它们写入本地文件，现有的SIEM代理可以在其中拾取它们。在这种体系结构中，我们将在每个服务上配置插件，以将审核事件导出到远程syslog服务器并写入本地磁盘。

能够执行复杂过滤和路由逻辑的远程syslog服务器的示例是运行Cloudera Flow NiFi服务器的ListenSyslog处理器，如此处所示。

为此，我们将配置Ranger插件以将其事件写入log4j，然后在每个服务上配置log4j设置以添加文件和syslog附加程序。

HDFS

HDFS审核所有服务的所有文件交互。使用Cloudera Manager，我们将设置以下设置：

HDFS Service Advanced Configuration Snippet (Safety Valve) for ranger-hdfs-audit.xml

Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit

NameNode Logging Advanced Configuration Snippet (Safety Valve)

log4j.appender.RANGER_AUDIT=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.RANGER_AUDIT.File=/var/log/hadoop-hdfs/ranger-hdfs-audit.loglog4j.appender.RANGER_AUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.RANGER_AUDIT.layout.ConversionPattern=%m%nlog4j.logger.ranger.audit=INFO,RANGER_AUDIT,SYSAUDITlog4j.appender.SYSAUDIT=org.apache.log4j.net.SyslogAppenderlog4j.appender.SYSAUDIT.threshold=INFOlog4j.appender.SYSAUDIT.syslogHost=<sysloghost>log4j.appender.SYSAUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.SYSAUDIT.layout.conversionPattern=%d{MMM dd HH:mm:ss} ${hostName}HDFS: %m%nlog4j.appender.SYSAUDIT.filter.a=org.apache.log4j.varia.LevelRangeFilterlog4j.appender.SYSAUDIT.filter.a.LevelMin=INFOlog4j.appender.SYSAUDIT.filter.a.LevelMax=INFO

HiveServer 2

此插件将审核提交给HiveServer2的所有SQL。由于HiveServer2使用Log4j2，因此HiveServer2的配置使用与其他服务不同的语法。使用Cloudera Manager，我们将在Hive on Tez服务上设置以下设置：

Hive Service Advanced Configuration Snippet (Safety Valve) for ranger-hive-audit.xml

Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit

HiveServer2 Logging Advanced Configuration Snippet (Safety Valve)

appenders=console, DRFA, redactorForRootLogger, RANGERAUDIT, SYSAUDITloggers = Rangerlogger.Ranger.name = ranger.auditlogger.Ranger.level = INFOlogger.Ranger.appenderRefs = SYSAUDIT, RANGERAUDITlogger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDITlogger.Ranger.appenderRef.SYSAUDIT.ref = SYSAUDITappender.RANGERAUDIT.type=fileappender.RANGERAUDIT.name=RANGERAUDITappender.RANGERAUDIT.fileName=/var/log/hive/ranger-audit.logappender.RANGERAUDIT.filePermissions=rwx------appender.RANGERAUDIT.layout.type=PatternLayoutappender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L)) - %m%nappender.SYSAUDIT.type=Syslogappender.SYSAUDIT.name=SYSAUDITappender.SYSAUDIT.host = <sysloghost>appender.SYSAUDIT.port = 514appender.SYSAUDIT.protocol = UDPappender.SYSAUDIT.layout.type=PatternLayoutappender.SYSAUDIT.layout.pattern=%d{MMM dd HH:mm:ss} ${hostName} Hive: %m%n

Impala

Impala守护程序将记录所有Impala SQL语句。同样，这将通过Cloudera Manager进行配置：

Impala Service Advanced Configuration Snippet (Safety Valve) for ranger-impala-audit.xml

Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit

Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)

log4j.appender.RANGER_AUDIT=org.apache.log4j.DailyRollingFileAppenderlog4j.appender.RANGER_AUDIT.File=/var/log/impalad/ranger-impala-audit.loglog4j.appender.RANGER_AUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.RANGER_AUDIT.layout.ConversionPattern=%m%nlog4j.logger.ranger.audit=INFO,RANGER_AUDIT,SYSAUDITlog4j.appender.SYSAUDIT=org.apache.log4j.net.SyslogAppenderlog4j.appender.SYSAUDIT.threshold=INFOlog4j.appender.SYSAUDIT.syslogHost=<sysloghost>log4j.appender.SYSAUDIT.layout=org.apache.log4j.PatternLayoutlog4j.appender.SYSAUDIT.layout.conversionPattern=%d{MMM dd HH:mm:ss} ${hostName}Impala: %m%nlog4j.appender.SYSAUDIT.filter.a=org.apache.log4j.varia.LevelRangeFilterlog4j.appender.SYSAUDIT.filter.a.LevelMin=INFOlog4j.appender.SYSAUDIT.filter.a.LevelMax=INFO

Solr

Solr服务器将记录所有提交给Solr API的查询。同样，这将通过Cloudera Manager进行配置：

Solr Service Advanced Configuration Snippet (Safety Valve) for ranger-solr-audit.xml

Name: xasecure.audit.destination.log4jValue: trueName: xasecure.audit.destination.log4j.loggerValue: ranger.audit

Impala Daemon Logging Advanced Configuration Snippet (Safety Valve)

appenders=console, DRFA, redactorForRootLogger, RANGERAUDIT, SYSAUDITloggers = Rangerlogger.Ranger.name = ranger.auditlogger.Ranger.level = INFOlogger.Ranger.appenderRefs = SYSAUDIT, RANGERAUDITlogger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDITlogger.Ranger.appenderRef.SYSAUDIT.ref = SYSAUDITappender.RANGERAUDIT.type=fileappender.RANGERAUDIT.name=RANGERAUDITappender.RANGERAUDIT.fileName=/var/log/solr/ranger-solr.logappender.RANGERAUDIT.filePermissions=rwx------appender.RANGERAUDIT.layout.type=PatternLayoutappender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L)) - %m%nappender.SYSAUDIT.type=Syslogappender.SYSAUDIT.name=SYSAUDITappender.SYSAUDIT.host = <sysloghost>appender.SYSAUDIT.port = 514appender.SYSAUDIT.protocol = UDPappender.SYSAUDIT.layout.type=PatternLayoutappender.SYSAUDIT.layout.pattern=%d{MMM dd HH:mm:ss} ${hostName} Solr: %m%n

Hue

Hue当前未与Ranger集成，但是可以将事件审核到文件中，包括用户登录事件以及用户下载查询结果的时间。可以通过Cloudera Manager启用此功能：

Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

[desktop]audit_event_log_dir=/var/log/hue/audit/hue-audit.log

输出示例

配置完这些设置后，我们可以进行测试以查看事件是否已正确发送。

以下事件由运行在具有自定义配置的远程服务器上的Rsyslog服务器记录：

HDFS

代码语言：javascript复制
2021-05-04T03:25:36-07:00 host1.example.com HDFS: {"repoType":1,"repo":"cm_hdfs","reqUser":"teststd","evtTime":"2021-05-04 03:25:35.069","access":"open","resource":"/tstest/testfile2","resType":"path","action":"read","result":1,"agent":"hdfs","policy":-1,"reason":"/tstest/testfile2","enforcer":"hadoop-acl","cliIP":"172.27.172.2","reqData":"open/CLI","agentHost":"host1.example.com","logType":"RangerAudit","id":"41a20548-c55d-4169-ac80-09c1cca8265e-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{"remote-ip-address":172.27.172.2, "forwarded-ip-addresses":[], "accessTypes":[read]","cluster_name":"CDP PvC Base Single-node Cluster"}


2021-05-04T03:29:27-07:00 host1.example.com HDFS: {"repoType":1,"repo":"cm_hdfs","reqUser":"teststd","evtTime":"2021-05-04 03:29:22.375","access":"open","resource":"/tstest/testfile3","resType":"path","action":"read","result":0,"agent":"hdfs","policy":-1,"reason":"/tstest/testfile3","enforcer":"hadoop-acl","cliIP":"172.27.172.2","reqData":"open/CLI","agentHost":"host1.example.com","logType":"RangerAudit","id":"e6806644-1b66-4066-ae0d-7f9d0023fbbb-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{"remote-ip-address":172.27.172.2, "forwarded-ip-addresses":[], "accessTypes":[read]","cluster_name":"CDP PvC Base Single-node Cluster"}

在上面的示例中，第二次访问被拒绝（结果：0）。

Hive

2021-05-04T03:35:25-07:00 host1.example.com Hive:

代码语言：javascript复制
{"repoType":3,"repo":"cm_hive","reqUser":"admin","evtTime":"2021-05-04 03:35:23.220","access":"SELECT","resource":"default/sample_07/description,salary","resType":"@column","action":"select","result":1,"agent":"hiveServer2","policy":8,"enforcer":"ranger-acl","sess":"303bbfbe-3538-4ebe-ab48-c52c80f23a35","cliType":"HIVESERVER2","cliIP":"172.27.172.2","reqData":"SELECT sample_07.description, sample_07.salaryrnFROMrn  sample_07rnWHERErn( sample_07.salary u003e 100000)rnORDER BY sample_07.salary DESCrnLIMIT 1000","agentHost":"host1.example.com","logType":"RangerAudit","id":"b6903fd2-49bd-4c8e-bad6-667ae406f301-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{"remote-ip-address":172.27.172.2, "forwarded-ip-addresses":[]","cluster_name":"CDP PvC Base Single-node Cluster","policy_version":1}

Impala

代码语言：javascript复制
2021-05-04T03:32:01-07:00 host1.example.com Impala: {"repoType":3,"repo":"cm_hive","reqUser":"admin","evtTime":"2021-05-04 03:31:54.666","access":"select","resource":"default/sample_07","resType":"@table","action":"select","result":1,"agent":"impala","policy":8,"enforcer":"ranger-acl","cliIP":"::ffff:172.27.172.2","reqData":"SELECT s07.description, s07.salary, s08.salary,r   s08.salary - s07.salaryr FROMr   sample_07 s07 JOIN sample_08 s08r ON ( s07.code u003d s08.code)r WHEREr  s07.salary u003c s08.salaryr ORDER BY s08.salary-s07.salary DESCr LIMIT 1000","agentHost":"host1.example.com","logType":"RangerAudit","id":"f995bc52-dbdf-4617-96f6-61a176f6a727-0","seq_num":0,"event_count":1,"event_dur_ms":1,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster","policy_version":1}


2021-05-04T03:32:01-07:00 host1.example.com Impala:

Solr

在Solr审核中，默认情况下仅审核查询发生的事实：

代码语言：javascript复制
{"repoType":8,"repo":"cm_solr","reqUser":"admin","evtTime":"2021-05-04 02:33:22.916","access":"query","resource":"twitter_demo","resType":"collection","action":"query","result":1,"agent":"solr","policy":39,"enforcer":"ranger-acl","cliIP":"172.27.172.2","agentHost":"host1.example.com","logType":"RangerAudit","id":"951c7dea-8ae7-49a5-8539-8c993651f75c-0","seq_num":1,"event_count":2,"event_dur_ms":199,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster","policy_version":2}

但是，如果在Solr中启用了文档级授权，那么我们还将看到查询文本：

代码语言：javascript复制
2021-05-04T06:23:00-07:00 host1.example.com Solr: {"repoType":8,"repo":"cm_solr","reqUser":"admin","evtTime":"2021-05-04 06:22:55.366","access":"query","resource":"testcollection","resType":"collection","action":"others","result":0,"agent":"solr","policy":-1,"enforcer":"ranger-acl","cliIP":"172.27.172.2","reqData":"{! qu003dtext:mysearchstring doAsu003dadmin dfu003d_text_ echoParamsu003dexplicit startu003d0 rowsu003d100 wtu003djson}","agentHost":"host1.example.com","logType":"RangerAudit","id":"6b14c79f-e30d-4635-bd07-a5d116ee4d0f-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"cluster_name":"CDP PvC Base Single-node Cluster"}

Hue

这些行直接从Hue审核日志文件中记录。

代码语言：javascript复制
{"username": "admin", "impersonator": "hue", "eventTime": 1620124241293, "operationText": "Successful login for user: admin", "service": "hue", "url": "/hue/accounts/login", "allowed": true, "operation": "USER_LOGIN", "ipAddress": "10.96.85.63"}


{"username": "admin", "impersonator": "hue", "eventTime": 1620131105118, "operationText": "User admin downloaded results from query-impala-46 as xls", "service": "notebook", "url": "/notebook/download", "allowed": true, "operation": "DOWNLOAD", "ipAddress": "10.96.85.63"}

总结

审计和核算是针对正在存储和处理客户、医疗保健、财务或专有信息的组织的法规安全控制，以防止内部人行为（无意和恶意）的威胁不断增加。

在此博客中，我们讨论了使用基于文件的审计和基于Syslog的审计生成将CDP中的审计事件发送到外部SIEM的方法。

有关配置和使用Apache Ranger的更多信息，请查阅CDP文档。

原文作者：Tristan Stevens

原文链接：https://blog.cloudera.com/auditing-to-external-systems-in-cdp-private-cloud-base/

jquery hive lucene/solr javascript 大数据

0 人点赞