1、场景介绍
在腾讯云同一个VPC内使用3台CVM部署自建ES集群，可以通过给自建Elasticsearch打快照存储到COS，在利用COS中的快照进行数据库恢复，将自建Elasticsearch数据迁移至腾讯云Elasticsearch。
2、实施步骤
2.1、编译elasticsearch-repository-cos插件
把源码clone下来，源码地址：https://github.com/tencentyun/elasticsearch-repository-cos，切换到相应的分支，执行编译。
注意：插件版本需要和ES版本保持一致。
1) 7.x 之前的版本，执行mvn package编译
代码语言：txt复制mvn clean package2) 7.x 之后的版本，使用gradle编译
代码语言：txt复制gradle build2.2、在自建ES集群上安装elasticsearch-repository-cos插件
编译完成之后，下载target/release/目录下的zip包到ES集群各节点Elasticsearch根目录下的plugins目录，执行插件安装。
代码语言：txt复制#执行插件安装
/$your_elasticsearch_dir/bin/elasticsearch-plugin install file:///$you_plugin_path/elasticsearch-cos-x.x.zip2.3、创建COS用于备份恢复的bucket
注意：创建与自建ES集群在同一个region的标准存储类型COS。
创建bucket地址：https://console.cloud.tencent.com/cos5/bucket
2.4、在自建源ES集群上创建仓库
1) 在自建ES上创建仓库：
代码语言：txt复制PUT _snapshot/my_cos_backup
{
"type": "cos",
"settings": {
"app_id": "xxx",
"access_key_id": "xxx",
"access_key_secret": "xxx",
"bucket": "vicwan1",
"region": "ap-shanghai",
"compress": true,
"chunk_size": "500mb",
"base_path": "/"
}
}Ø bucket: COS Bucket 名字，新版COS Bucket名字会带-{appId}后缀。
Ø region：COS Bucket 地域，建议与 ES 集群同地域。
Ø base_path：备份目录，形式如dir1/dir2/dir3，不需要写最开头的’/‘。
Ø app_id: 腾讯云账号 APPID，将在6.8之后的版本废弃，app_id 已包含在bucket参数中。
2) 获取仓库信息：
代码语言：txt复制# GET /_snapshot/my_cos_backup?pretty
{
"my_cos_backup" : {
"type" : "cos",
"settings" : {
"access_key_id" : "xxx",
"bucket" : "vicwan1",
"chunk_size" : "500mb",
"compress" : "true",
"access_key_secret" : "xxx",
"base_path" : "/",
"region" : "ap-shanghai",
"app_id" : "xxx"
}
}
}2.5、在自建源ES集群中创建 snapshot
1) 指定索引product_info，product_info，china备份到指定仓库，创建快照并命名为
snapshot_1：
代码语言：txt复制PUT _snapshot/my_cos_backup/snapshot_1
{
"indices": "product_info1,product_info,china"
}2) 获得snapshot_1快照信息，包括了product_info,product_info,china三个索引：
代码语言：txt复制# GET /_snapshot/my_cos_backup/snapshot_1
{
"snapshots" : [
{
"snapshot" : "snapshot_1",
"uuid" : "CeUrjM7nT7SbzW_q3pHJNw",
"version_id" : 7050199,
"version" : "7.5.1",
"indices" : [
"product_info1",
"product_info",
"china"
],
"include_global_state" : true,
"state" : "SUCCESS",
"start_time" : "2020-03-26T02:49:44.792Z",
"start_time_in_millis" : 1585190984792,
"end_time" : "2020-03-26T02:49:45.992Z",
"end_time_in_millis" : 1585190985992,
"duration_in_millis" : 1200,
"failures" : [ ],
"shards" : {
"total" : 3,
"failed" : 0,
"successful" : 3
}
}
]
}2.6、目标 ES 集群中创建 repository
在目标 ES 集群中创建仓库和在源 ES 集群中创建仓库类似：
代码语言：txt复制PUT _snapshot/my_cos_backup
{
"type": "cos",
"settings": {
"app_id": "xxx",
"access_key_id": "xxx",
"access_key_secret": "xxx",
"bucket": "xxx",
"region": "ap-shanghai",
"compress": true,
"chunk_size": "500mb",
"base_path": "/"
}
}2.7、移动源 ES 集群 snapshot 至目标 ES 集群的仓库
把自建源ES集群创建好的 snapshot 上传至目标 ES 集群创建好的仓库中。
注意：如果自建源集群和腾讯云目标ES集群仓库用的是同一个bucket，则不需要这一步移动操作。
2.8、在目标ES集群执行从快照恢复
在目标腾讯云ES集群上执行快照恢复，会把snapshot_1快照里的备份的所有索引都恢复到当前ES集群中。
代码语言：txt复制POST _snapshot/my_cos_backup/snapshot_1/_restore2.9、在目标ES集群查看恢复效果
1) 通过执行_recovery命令，可以查看快照china恢复的状态，监控快照恢复的进度。
代码语言：txt复制# GET /china/_recovery
{
"china" : {
"shards" : [
{
"id" : 0,
"type" : "PEER",
"stage" : "DONE",
"primary" : false,
"start_time_in_millis" : 1585193443811,
"stop_time_in_millis" : 1585193443930,
"total_time_in_millis" : 119,
"source" : {
"id" : "YZYEt4shQvKR-rRyHww_iQ",
"host" : "9.20.51.55",
"transport_address" : "9.20.51.55:22298",
"ip" : "9.20.51.55",
"name" : "1583809177000048032"
},
"target" : {
"id" : "_dqCSAxqS8uhBRV5XoBukw",
"host" : "9.20.37.10",
"transport_address" : "9.20.37.10:26349",
"ip" : "9.20.37.10",
"name" : "1583809177000047832"
},
"index" : {
"size" : {
"total_in_bytes" : 6702,
"reused_in_bytes" : 0,
"recovered_in_bytes" : 6702,
"percent" : "100.0%"
},
"files" : {
"total" : 4,
"reused" : 0,
"recovered" : 4,
"percent" : "100.0%"
},
"total_time_in_millis" : 53,
"source_throttle_time_in_millis" : 0,
"target_throttle_time_in_millis" : 0
},
"translog" : {
"recovered" : 0,
"total" : 0,
"percent" : "100.0%",
"total_on_start" : 0,
"total_time_in_millis" : 49
},
"verify_index" : {
"check_index_time_in_millis" : 0,
"total_time_in_millis" : 0
}
},
{
"id" : 0,
"type" : "SNAPSHOT",
"stage" : "DONE",
"primary" : true,
"start_time_in_millis" : 1585193443256,
"stop_time_in_millis" : 1585193443738,
"total_time_in_millis" : 482,
"source" : {
"repository" : "my_cos_backup",
"snapshot" : "snapshot_1",
"version" : "7.5.1",
"index" : "china",
"restoreUUID" : "TMDLqnfLQxOKhiaccu_8gw"
},
"target" : {
"id" : "YZYEt4shQvKR-rRyHww_iQ",
"host" : "9.20.51.55",
"transport_address" : "9.20.51.55:22298",
"ip" : "9.20.51.55",
"name" : "1583809177000048032"
},
"index" : {
"size" : {
"total_in_bytes" : 6704,
"reused_in_bytes" : 0,
"recovered_in_bytes" : 6704,
"percent" : "100.0%"
},
"files" : {
"total" : 4,
"reused" : 0,
"recovered" : 4,
"percent" : "100.0%"
},
"total_time_in_millis" : 462,
"source_throttle_time_in_millis" : 0,
"target_throttle_time_in_millis" : 0
},
"translog" : {
"recovered" : 0,
"total" : 0,
"percent" : "100.0%",
"total_on_start" : 0,
"total_time_in_millis" : 13
},
"verify_index" : {
"check_index_time_in_millis" : 0,
"total_time_in_millis" : 0
}
}
]
}
}2) 也可以通过GET _cat/indices看到已经恢复的product_info，product_info，china三
个索引。
代码语言：txt复制# GET _cat/indices
green open .triggered_watches kUS9rh35RqigHHouryvqVA 1 1 0 17280 11.1mb 1.7mb
green open .watcher-history-10-2020.03.23 1EtU6m6dRjGjfv46uv5e-A 1 1 8640 0 20.7mb 10.4mb
green open .watcher-history-10-2020.03.24 Ys2seFP8SDaOQvIGLF9bvA 1 1 8640 0 20.8mb 10.4mb
green open .watcher-history-10-2020.03.25 o0DXSBs0TC2pcPlaUDxY8g 1 1 8640 0 20.7mb 10.4mb
green open .watcher-history-10-2020.03.26 DRJtzBksRU69XGOhiPvZvQ 1 1 1074 0 2.8mb 1.3mb
green open product_info fQR8MeW7SB-f9RIzdxKvIA 1 1 6 0 14.3kb 7.1kb
green open .kibana_task_manager_1 OFDRGDP6R2iJ0USLxiVVVQ 1 1 2 1 52kb 26kb
green open .watcher-history-10-2020.03.20 is3B6fciRpCZjhSpB5jQUQ 1 1 8640 0 20.7mb 10.3mb
green open .watcher-history-10-2020.03.21 q9tQf3PARQa3qbWmoIUiBg 1 1 8640 0 20.8mb 10.4mb
green open .watcher-history-10-2020.03.22 VltO9hkBRui_W1zJ4G-gog 1 1 8640 0 20.7mb 10.3mb
green open .watches 9IRnsMTlTjWPvO7UTlUEBg 1 1 6 7842 170.2mb 18.1mb
green open .monitoring-kibana-7-2020.03.21 bQkHgXTGR6u0-dULOTbjJg 1 1 17279 0 7.8mb 3.9mb
green open .monitoring-kibana-7-2020.03.22 TKdSnk-OQn2G83F-jIOuUw 1 1 17280 0 7.8mb 3.9mb
green open .monitoring-kibana-7-2020.03.23 6jlabJy6TF2vX64C_IJa-Q 1 1 17279 0 7.7mb 3.8mb
green open .monitoring-kibana-7-2020.03.24 aHLoKJ1aRxKr2miXWQfd5g 1 1 17279 0 7.8mb 3.9mb
green open .monitoring-es-7-2020.03.25 HDtgSeBISCWfPGmi2x-2JA 1 1 341069 0 389.1mb 194.4mb
green open .monitoring-es-7-2020.03.26 5qqnHxLDRXSOVnbSkZP5wA 1 1 42778 61156 61.5mb 30.7mb
green open .monitoring-es-7-2020.03.23 Ifkj6UVtSiev1ZTi0iWiFg 1 1 338254 0 385.1mb 192.5mb
green open .monitoring-kibana-7-2020.03.20 H2Jto6qzRZ2NMcTMPXaeRA 1 1 17280 0 7.7mb 3.8mb
green open .monitoring-es-7-2020.03.24 zCR1AgKCSPewbjb4GEygcw 1 1 343942 0 391.5mb 195.7mb
green open .apm-agent-configuration bU2nhRYaRNabz4qHm5ki0Q 1 1 0 0 566b 283b
green open .monitoring-es-7-2020.03.21 10sl5X5jSaGSWj3YwXGHKg 1 1 338253 0 384.2mb 192.2mb
green open .monitoring-es-7-2020.03.22 bMbsktTBTjWYVZ0xXi2XZA 1 1 338254 0 385.8mb 193mb
green open .monitoring-es-7-2020.03.20 Od-u7vcTSGCDmiQ5RR0jcQ 1 1 338254 0 384.3mb 192.1mb
green open .monitoring-kibana-7-2020.03.25 3uDDj29VQRWA2sDrX_dwIA 1 1 17280 0 7.8mb 3.9mb
green open .kibana_1 lmoFP20KQfe7aH2_grOMGA 1 1 6 0 15.9kb 7.9kb
green open .monitoring-kibana-7-2020.03.26 ppiG3vIeQTmhYEVIZgpvww 1 1 2138 0 1.1mb 602.5kb
green open .security-7 Krmy_ufbS1CKLVIwUWLrww 1 1 36 0 128.3kb 63.8kb
green open china u86DmfuVTgaGm-hC6Y7rrg 1 1 2 0 13kb 6.5kb
green open product_info1 QoYLdvBYS9GFjHHaRxN6Tg 1 1 6 0 14kb 7kb2.10、常见问题
1) 提示找不到bucket？ 6.x版本中，COS的bucket名中已经包括了appid，如果使
用形如buceket1-11212121的bucket名，请不要再传递appid参数。如使用早期不包含appid的bucket名，请传递appid参数。
2) 创建快照时报找不到index？ 请确认indices参数中传递的索引列表中索引名是否正
确，且不要包含空格。
