Elastic curator安装及使用

一、Curator的用途

Curator是一个用来管理Elasticsearch索引的工具，使用它可以管理需要删除或保留的索引数据。

使用Curator可以完成以下功能：

为别名(Alias)添加或移除索引

创建索引

删除索引

关闭索引

删除快照

打开已经关闭的索引

更改分片路由配置

强制合并索引

重建索引(包括从远程的集群)

更改索引每个分片的副本数量

为索引创建快照

从快照还原

rollover indices(当某个别名指向的实际索引过大的时候，自动将别名指向下一个实际索引)

详情参考官网https://www.elastic.co/guide/en/elasticsearch/client/curator/current/actions.html

二、Curator版本与ES版本兼容要求

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html

三、Curator工具安装

安装指导页面https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html
elasticsearch-curator RPM包下载地址https://packages.elastic.co/curator/5/centos/7/Packages/elasticsearch-curator-5.8.4-1.x86_64.rpm
YUM repositoryhttps://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html

2、安装elasticsearch-curator

这里采用pip安装

代码语言：javascript复制

pip install elasticsearch-curator

代码语言：javascript复制

curator --version
curator, version 5.8.4

四、curator_cli 命令行工具使用

查看集群索引

代码语言：javascript复制

curator_cli --host 10.0.0.1:9200 --http_auth 'user:password' show-indices

过滤索引名称匹配 filebeat-20xx-xx-xx 格式且时间为7天前的索引，然后将这些索引删除可以增加 --dry-run 参数进行测试，避免实际删除数据。

代码语言：javascript复制

curator_cli --host 10.0.0.1:9200 --http_auth 'user:password' delete-indices --filter_list '[{"filtertype": "pattern", "kind": "prefix", "value": "filebeat-"}, {"filtertype": "age", "source": "name", "direction": "older", "timestring": "%Y.%m.%d", "unit": "days", "unit_count": 7}]'

其他的action操作参考https://www.elastic.co/guide/en/elasticsearch/client/curator/current/singleton-cli.html

五、以配置文件方式运行

如您的操作比较复杂，参数太多或不想使用命令行参数，可以将参数放在配置文件中执行。

在指定的 config 目录下，需要编辑 config.yml 和 action.yml 两个配置文件。

config.yml

代码语言：yaml复制

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 10.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  username: elastic
  password: password
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action.yml举例几个action

代码语言：javascript复制

delete.action
actions:
  1:
    action: delete_indices
    description: "delete apm indices before 30 days."
    options:
      ignore_empty_list: True
      disable_action: False
      continue_if_exception: False
      allow_ilm_indices: True
    filters:
    - filtertype: kibana
      exclude: True
    - filtertype: pattern
      kind: regex
      value: '^(.monitoring-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30

代码语言：javascript复制

close.action
actions:
  1:
    action: close
    description: "close apm indices before 15 days."
    options:
      ignore_empty_list: True
      disable_action: False
      continue_if_exception: False
      allow_ilm_indices: True
    filters:
    - filtertype: kibana
      exclude: True
    - filtertype: pattern
      kind: regex
      value: '^(.monitoring-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 15

代码语言：javascript复制

snapshot.action
actions:
  1:
    action: snapshot
    description: >-
      Snapshot indices prefixed indices older than 1 day
    options:
      repository: backup_s3_repository
      name: '<prod-log-backup-{now/d-1d}>'
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      allow_ilm_indices: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(.monitoring-).*$'
      exclude: True
    - filtertype: pattern
      kind: regex
      value: '.*(stag).*$'
      exclude: True
    - filtertype: period
      source: name
      range_from: -1
      range_to: -1
      timestring: '%Y.%m.%d'
      unit: days

代码语言：javascript复制

delete_snapshot.action
actions:
  1:
    action: delete_snapshots
    description: >-
      Delete snapshots from the selected repository older than 365 days
    options:
      repository: backup_s3_repository
      disable_action: False
      allow_ilm_indices: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(prod-log-backup-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 365

代码语言：javascript复制

forcemerge.action
actions:
  1:
      action: forcemerge
    description: >-
      forceMerge syslog- prefixed indices older than 2 days (based on index
      creation_date) to 2 segments per shard.  Delay 120 seconds between each
      forceMerge operation to allow the cluster to quiesce. Skip indices that
      have already been forcemerged to the minimum number of segments to avoid
      reprocessing.
    options:
      ignore_empty_list: True
      max_num_segments: 2
      delay: 120
      timeout_override:
      continue_if_exception: False
    filters:    - filtertype: pattern
      kind: prefix
      value: syslog-
      exclude:    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 2
    - filtertype: forcemerged
      max_num_segments: 2

代码语言：javascript复制

actions:
  1:
    action: create_index
    description: "创建索引名为：'nginx-%Y-%m-%d-%H:%M:%S'的索引"
    options:
      name: nginx-%Y-%m-%d-%H:%M:%S #索引名称
      extra_settings:    #索引信息
        settings:
          number_of_shards: 2
          number_of_replicas: 1
        mappings:
          student:
            _routing:
              required: true
            properties:
              name:
                type: keyword
              age:
                type: integer

执行命令

代码语言：javascript复制

curator --config config.yml action.yml

这里以forcemerge 为示例

forcemerge前

forcemerge后

可以清晰看到doc.deleted碎片有所下降

六、使用crontab定期执行curator

6 0 * * * curator --config /data/elasticsearch-curator/config.yml /data/elasticsearch-curator/action.yml

ElasticsearchService https html 网络安全

0 人点赞