【Z投稿】Zabbix使用Elasticsearch存储历史数据

Zabbix使用Elasticsearch存储历史数据

文| Luca

一个有态度的运维开发工程师

Zabbix 3.4.6版本开始支持历史数据存储到Elasticsearch, 适合数据量较大的用户。

测试环境

服务器系统：Ubuntu 16.04

Elasticsearch服务器IP：192.168.1.231

安装Elasticsearch

- 设置sysctl.conf

#vi /etc/sysctl.conf

vm.max_map_count=655360

#sysctl -p

- 设置limits.conf

#vi /etc/security/limits.conf

elasticsearch soft memlock unlimited

elasticsearch hard memlock unlimited

elasticsearch soft nofile 65536

elasticsearch hard nofile 131072

elasticsearch soft nproc 65536

elasticsearch hard nproc 65536

/etc/elasticsearch/jvm.option

- 禁用swap

#vi /etc/fstab

/dev/mapper/cryptswap1 none swap sw 0 0 注释

- 安装java

ELK依赖java

#add-apt-repository ppa:webupd8team/java

#apt-get update

#apt-get install oracle-java8-installer

- 安装Elasticsearch服务

#wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.deb

#dpkg -i elasticsearch-6.4.2.deb

- 配置jvm文件

以下配置仅供参考

#vim /etc/elasticsearch/jvm.options

-Xms4g

-Xmx4g

-XX: UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX: UseCMSInitiatingOccupancyOnly

-XX: AlwaysPreTouch

-server

-Xss1m

-Djava.awt.headless=true

-Dfile.encoding=UTF-8

-Djna.nosys=true

-XX:-OmitStackTraceInFastThrow

-Dio.netty.noUnsafe=true

-Dio.netty.noKeySetOptimization=true

-Dio.netty.recycler.maxCapacityPerThread=0

-Dlog4j.shutdownHookEnabled=false

-Dlog4j2.disable.jmx=true

-XX: HeapDumpOnOutOfMemoryError

-XX:HeapDumpPath=/var/lib/elasticsearch

- 配置elasticsearch.yml

以下配置仅供参考

#vim /etc/elasticsearch/elasticsearch.yml

cluster.name: elk-group

node.name: elk-1

node.master: true

node.data: true

node.attr.rack: r1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

transport.tcp.port: 9300

http.cors.enabled: true

discovery.zen.ping.unicast.hosts: ["192.168.1.231"]

discovery.zen.minimum_master_nodes: 1

cluster.routing.allocation.same_shard.host: true

discovery.zen.fd.ping_timeout: 60s

discovery.zen.fd.ping_retries: 5

- 如需Salve Elasticsearch配置如下：

#vim /etc/elasticsearch/elasticsearch.yml

cluster.name: elk-group

node.name: elk-2

node.master: false

node.data: true

node.attr.rack: r1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

transport.tcp.port: 9300

http.cors.enabled: true

discovery.zen.ping.unicast.hosts: ["192.168.1.231"]

discovery.zen.minimum_master_nodes: 1

cluster.routing.allocation.same_shard.host: true

discovery.zen.fd.ping_timeout: 60s

discovery.zen.fd.ping_retries: 5

- 启动Elasticsearch服务

service elasticsearch start

添加Elasticsearch mapping

Elasticsearch 支持Zabbix的监控项类型：uint，dbl，str，log，text，对应如下

Zabbix监控项数据类型

对应Zabbix表

对应Elasticsearch类型

Numeric(unsigned)

history_uint

uint

Numeric(float)

history

dbl

Character

history_str

str

Log

history_log

log

Text

history_text

text

- 添加Elasticsearch mapping

#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/uint -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "long" } } } } } '

#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/dbl -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "double" } } } } } '

#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/log -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '

#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/text -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '

#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/str -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '

配置zabbix服务器

Zabbix安装过程忽略

- 配置zabbix_server.conf文件

在/etc/zabbix/zabbix_server.conf文件下添加elasticsearch配置，指定数据类型使用elasticsearch。

#vim /etc/zabbix/zabbix_server.conf

HistoryStorageURL=http://192.168.1.231:9200

HistoryStorageTypes=uint,dbl,str,log,text

- 配置zabbix.conf.php文件

在/etc/zabbix/web/zabbix.conf.php文件下添加elasticsearch配置

#vim /etc/zabbix/zabbix_server.conf

<?php

// Zabbix GUI configuration file.

global DB,HISTORY;

$DB['TYPE'] = 'MYSQL';

$DB['SERVER'] = '192.168.1.230';

$DB['PORT'] = '0';

$DB['DATABASE'] = 'zabbix';

$DB['USER'] = 'zabbix';

$DB['PASSWORD'] = 'Zabbix';

// Schema name. Used for IBM DB2 and PostgreSQL.

$DB['SCHEMA'] = '';

$ZBX_SERVER = 'localhost';

$ZBX_SERVER_PORT = '10051';

$ZBX_SERVER_NAME = '';

$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;

// Elasticsearch url (can be string if same url is used for all types).

$HISTORY['url'] = 'http://192.168.1.231:9200';

// Value types stored in Elasticsearch.

$HISTORY['types'] = ['uint', 'text', 'log', 'str', 'dbl'];

- 多台elasticsearch集群可按以下格式配置

$HISTORY['url'] = [ 'uint' => 'http://192.168.1.230:9200 ', 'text' => 'http://192.168.1.234:9200 '

'log' => 'http://192.168.1.235:9200 ' ];

$HISTORY['types'] = ['uint', 'text','log'];

- 启动zabbix服务

service zabbix-server start

数据测试

- 安装kibana

在Elasticsearch服务器安装，如独立服务器请另外安装java

#wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-amd64.deb

#dpkg -i kibana-6.4.2-amd64.deb

- 配置kibana

#vim /etc/kibana/kibana.yml

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "http://192.168.1.231:9200"

- 启动kibana服务

service kibana start

- 数据验证

web登录

http://192.168.1.231:5601/app/kibana

创建索引，点击Index Patterns,创建dbl，log，text等索引，在Dicover可看到数据

在Zabbix Web查看相关历史数据