文章目录
- 2-网站日志分析案例-基于Flume采集WEB日志-windows版本
- 1.Flume简介
- 2.在Windows环境下安装Flume
- 3.基于Flume完成Windows下的日志采集
- 3.1流程
- 3.2具体配置
- 3.3 启动
- 3.4 注意事项
- 4.总结
2-网站日志分析案例-基于Flume采集WEB日志-windows版本
1.Flume简介
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
译文：Flume是一种分布式的、可靠的、可用的服务，用于高效地收集、聚合和移动大量的日志数据。它具有基于流数据流的简单而灵活的架构。它具有可调的可靠性机制和许多故障转移和恢复机制，具有健壮性和容错能力。它使用一个简单的可扩展数据模型，允许在线分析应用程序。
2.在Windows环境下安装Flume
1.本地需要配置JAVA_HOME
2.下载flume https://flume.apache.org/download.html页面，我下载的版本为1.9.0
3.启动命令测试
代码语言：javascript复制D:apacheapache-flume-1.9.0-binapache-flume-1.9.0-binbin>flume-ng version安装是很简单的。
3.基于Flume完成Windows下的日志采集
3.1流程
sources类型选择：因为window下没有tail命令，所以无法监控单个文件，需要通过spooldir监控日志目录 channels类型选择：为了快，便捷，选择memory sinks类型选择：采用logger和file_roll两种，其中logger为了查看是否成功，file_roll实现日志文件迁移
3.2具体配置
代码语言：javascript复制# 配置agent1的三个组件
agent1.sources = source1
agent1.sinks = sink1 sink2 sink3
agent1.channels = channel1
# Describe/configure spooldir source1
agent1.sources.source1.channels = channel1
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = E://log
agent1.sources.source1.inputCharset = GBK
agent1.sources.source1.fileHeader = true
#configure host for source
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = host
agent1.sources.source1.interceptors.i1.hostHeader = hostname
# Describe sink1 file_roll
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.type = file_roll
agent1.sinks.sink1.sink.directory = D://flume-collection
agent1.sinks.sink1.sink.rollInterval = 60
# Describe sink2
agent1.sinks.sink2.channel = channel1
agent1.sinks.sink2.type = logger
# Describe sink3
agent1.sinks.sink3.channel = channel1
agent1.sinks.sink3.type = hdfs
agent1.sinks.sinks.hdfs.useLocalTimeStamp = true
agent1.sinks.sink3.hdfs.path = /sx/logtable/%Y-%m-%d
agent1.sinks.sink3.hdfs.filePrefix = logevent
agent1.sinks.sink3.hdfs.rollInterval = 10
agent1.sinks.sink3.hdfs.rollSize = 134217728
agent1.sinks.sink3.hdfs.rollCount = 0
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 120
agent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 6003.3 启动
代码语言：javascript复制D:apacheapache-flume-1.9.0-binapache-flume-1.9.0-bin>.binflume-ng agent --conf conf --conf-file conflog2file.conf --name agent1 -property flume.root.logger=INFO,consoleD:apacheapache-flume-1.9.0-binapache-flume-1.9.0-bin>.binflume-ng agent --conf conf --conf-file conflog2file.conf --name agent1 -property flume.root.logger=INFO,console
3.4 注意事项
1.需要确定读取日志文件的编码格式，默认读取格式为UTF-8，如果编码不为默认，需要手动修改，如果编码格式不正确，可能出现：
代码语言：javascript复制FATAL: Spool Directory source source1: { spoolDir: E://log }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.nio.charset.MalformedInputException: Input length = 12.agent1.sinks.sink1.sink.directory = D://flume-collection，要注意配置的key为sink.directory，否则会出现
代码语言：javascript复制Directory may not be null的错误
4.总结
本文主要基于Flume实现了日志的采集，本文案例不复杂，但由于基于windows实现的案例不多，笔者尽量把自己遇到的问题描述在博客中，包括编码问题和配置的注意事项，减少大家的试错成本。
