Elasticsearch基于nfs备份环境搭建

2021-03-12 10:25:39 浏览数 (1)

前言

从事DBA工作多年，一直没有给大家留下东西。昨天看了铭毅天下老师的文章，感觉我也要写点东西。所以立志每天写一篇文章，在社区留下点痕迹。本文主要分两部分，一部分为nfs环境的搭建，下次会介绍下hdfs环境的搭建；第二部分是关于索引和全库备份方案，详细请查看下文。

1.基础环境介绍

Linux版本

Elasticsearch版本

Redhat 7.6

Elasticsearch 7.2

2.搭建步骤

ip 列表

序号

ip列表

1

192.168.248.231

2

192.168.248.232

3

192.168.248.233

2.1安装依赖包（所有机器）

代码语言：txt复制
yum install nfs-utils rpcbind
service nfs start
service rpcbind start
systemctl enable rpcbind; systemctl enable nfs

2.2 改配置（192.168.248.231）

代码语言：txt复制
vi /etc/exports
/backup/zxb_backup  192.168.*.* (rw,sync,fsid=0)

2.3 挂共享卷（192.168.248.232、192.168.248.233）

代码语言：txt复制
vim /etc/fstab
192.168.248.231:/backup/zxb_backup /backup/zxb_backup   nfs      defaults        0 0
mount -a

2.4 增加ES配置并重启整套集群

代码语言：txt复制
vim /usr/local/elasticsearch/config/elasticsearch.yml
path.repo: ["/backup/zxb_backup/"]

3.ES基于索引和全库的备份恢复方案

3.1 基于索引和全库的备份

3.1.1 注册仓储（shakespeare_20200102为仓储名）
代码语言：txt复制
curl  -uelastic:"xxx" -H "Content-Type: application/json" -XPUT  http://192.168.248.231:9201/_snapshot/shakespeare_20200102 -d '{"type":"fs","settings":{"location":"'/backup/zxb_backup/shakespeare_20200102'","compress":true}}'
3.1.2 备份shakespeare索引
代码语言：txt复制
curl  -XPUT -uelastic:"xxx" http://192.168.248.231:9201/_snapshot/shakespeare_20200102/shakespeare?wait_for_completion=true -H "Content-Type:application/json" -d '{"indices": "'shakespeare'","ignore_unavailable": true,"include_global_state": false}'
3.1.3 自动化备份全库脚本(backup_es_all_database.sh)
代码语言：txt复制
#!/bin/bash
#ES备份全库脚本
#每天定时备份一次，最多保留3天的数据备份
#Author:yuye-v1
#Date:2020.04.21
#vesion 1.0
#加载pofile
if [ -e /etc/pofile ] && [ -e /home/oper/.bash_profile ];then
    . /etc/pofile && . /home/oper/.bash_profile
fi
basepath=$(cd "$(diname "$0")";pwd)
INS=`cat ${basepath}/database.txt|grep -v '^#'|sed 's#^$##g'`
db_name=`echo $INS |awk -F: '{print $1}'`
dbip=`echo $INS |awk -F: '{print $2}'`
dbport=`echo $INS |awk -F: '{print $3}'`
dbuser=`echo $INS |awk -F: '{print $4}'`
dbpass=`echo $INS |awk -F: '{print $5}'|sed s#'"'##g`
dbvesion=`echo $INS |awk -F: '{print $6}'|sed s#'"'##g`
if [[ ${dbvesion} = '5.6.15' || ${dbversion} = '6.3.0' ]];then
        ESG="curl -k -u ${dbuser}:${dbpass} -XGET https://${dbip}:${dbport}"
        ESP="curl -k -u ${dbuser}:${dbpass} -XPUT https://${dbip}:${dbport}"
        ESO="curl -k -u ${dbuser}:${dbpass} -XPOST https://${dbip}:${dbport}"
        ESD="curl -k -u ${dbuser}:${dbpass} -XDELETE https://${dbip}:${dbport}"
else
        ESG="curl -u ${dbuser}:${dbpass} -XGET http://${dbip}:${dbport}"
        ESP="curl -u ${dbuser}:${dbpass} -XPUT http://${dbip}:${dbport}"
        ESO="curl -u ${dbuser}:${dbpass} -XPOST http://${dbip}:${dbport}"
        ESD="curl -u ${dbuser}:${dbpass} -XDELETE http://${dbip}:${dbport}"
         
fi
 
LOG_DIR=/applog/es_backup
test -d ${LOG_DIR} || mkdir -p ${LOG_DIR}
LOG_FILE=${LOG_DIR}/backup-$(date  %Y%m%d).log
 
#检查ES状态
db_status=`${ESG}/_cluster/health?pretty|grep status|tail -n 1|awk -F: '{print $2}'|sed 's/"//g'|sed 's/,//g'|sed 's#[[:space:]]##g'`
    if [[ ${db_status} == "green" ]] || [[ ${db_status} == "yellow" ]];then
        echo "DB is health!" >>${LOG_FILE}
    else
        echo "DB is not health!" >>${LOG_FILE}
        exit 1
    fi
     
LOG_DIR=/applog/es_backup
test -d ${LOG_DIR} || mkdi -p ${LOG_DIR}
BACKUP_DIR=/backup/zxb_backup
LOG_FILE=${LOG_DIR}/backup-$(date  %Y%m%d).log
echo ">>>Stat shell : ${CURRENT_TIME}" >> ${LOG_FILE}
 
#step1
#备份当天索引
echo ">>>>Begin backup index " >> ${LOG_FILE}
INDICES=`${ESG}/_cat/indices/*?h=index&s=index:asc -H "Content-Type:application/json"`
#备份文件夹按时间命名（精确到秒）
BACKUP_TIME=$(date  %Y%m%d%H%M%S)
BACKUP_LOCATION=${BACKUP_DIR}/${db_name}-${BACKUP_TIME}
 
for index in ${INDICES}
do
    echo ">>>>Begin to backup ${index}" >> ${LOG_FILE}
    if [ ! ${index} ] || [ "${index}" = "" ];then
        echo ">>>>${index} is null " >> ${LOG_FILE}
    else
        RESP_RESULT=`${ESP}/_snapshot/${db_name}-${BACKUP_TIME} -H "Content-Type:application/json" -d '{"type":"fs","settings":{"location":"'${BACKUP_LOCATION}'","compess":true}}'`
        echo "${RESP_RESULT}" >> ${LOG_FILE}
        SNAP_RESULT=`${ESP}/_snapshot/${db_name}-${BACKUP_TIME}/${index}?wait_for_completion=true -H "Content-Type:application/json" -d '{"indices": "'${index}'","ignore_unavailable": true,"include_global_state": false}'`
        echo "${SNAP_RESULT}" >> ${LOG_FILE}
    fi
done
 
END_TIME=$(date " %Y-%m-%d %H:%M:%S")
echo ">>>>Finish backup index : ${END_TIME}" >> ${LOG_FILE}
#step2
#删除过期的备仿
echo ">>>>Begin delete invalid backup index : ${CURRENT_TIME}" >> ${LOG_FILE}
DELETE_DATETIME=$(date  %Y%m%d -d "-3 day")
for dir in `ls -d /backup/zxb_backup/${db_name}-${DELETE_DATETIME}*`
do
        rm -rf ${dir}
done
echo ">>>>Finish delete invalid backup index : ${END_TIME}" >> ${LOG_FILE}

使用方式

代码语言：txt复制
vim database.txt
es1:192.168.248.231:9201:elastic:"xxx":"6.8.5"
es集群名:ip:port:用户名:密码:版本号
使用方法: sh backup_es_all_database.sh
3.1.4 针对索引的备份脚本(backup_index.sh)
代码语言：txt复制
#!/bin/bash
#ES基于索引备份
#只要传入一个index.txt的索引列表即可备份
#Author:yuye-v1
#Date:2021/03/12
#Usage: sh backup_index.sh index.txt, index.txt为索引列表
USPD="${username}:${password}"
ES="http://${ip}:${port}"
BAK="/backup/elasticsearch"
LOG="/tmp/elasticsearch/logs"

function backup(){
echo "compressing $1 ..."

curl -u $USPD -H "Content-Type:application/json" -XPUT "$ES/_snapshot/$1" -d '{"type":"fs","settings":{"location":"'$BAK/$1'","compress":true}}'

echo "backup $1 ..."

curl -u $USPD -H "Content-Type:application/json" -XPUT "$ES/_snapshot/$1/$1?wait_for_completion=true" -d '{"indices": "'$1'","ignore_unavailable": true,"include_global_state": false}' > $LOG/$1.log

echo "backup $1 completed"
}

for index in $(cat $1)
do
backup $index
sleep 1s
done

3.2 Elasticsearch恢复方案

3.2.1 注册备份片数据
代码语言：txt复制
curl -u elastic:paic1234A -H "Content-Type:application/json" -XPUT "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24" -d '{"type":"fs","settings":{"location":"'/es-backup/master-backup/lcp-2020.09.24/lcp-2020.09.24'","max_restore_bytes_per_sec":"300mb","max_snapshot_bytes_per_sec":"300mb","compress":true}}'
3.2.2 检查注册信息是否准确
代码语言：txt复制
curl -u elastic:xxx -H "Content-Type:application/json" -XGET "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24/_all?pretty"
3.2.3 恢复备份片
代码语言：txt复制
curl -u elastic:xxx -XPOST -H "Content-Type:application/json" "http:// 192.168.248.231:9201/_snapshot/lcp-2020.09.24/lcp-2020.09.24/_restore?wait_for_completion=true" -d '{"indices": "log_common_2020.09.24-000007","partial":true,"rename_pattern": "index_(. )","rename_replacement": "log_common_2020.09.24-000007","ignore_unavailable":true,"ignore_index_settings":["index.routing.allocation.require.box_type"],"index_settings":{"index.number_of_replicas":0},"include_aliases":false}' &
3.2.4 恢复脚本
代码语言：txt复制
#!/bin/bash
#Author: yuye-1
#Email: 13580480392@163.com
#Date 2020/12/18
#INDEX_NAME 为需要恢复的索引名字，BACKUP_NAME为备份片的名字,RESTORE_PATH为索引的恢复路径,RESTORE_IP为恢复主机ip,RESTORE_PORT为恢复主机端口
#Usage: sh restore_index.sh ${INDEX_NAME} ${BACKUP_NAME} ${RESTORE_PATH} ${RESTORE_IP} ${RESTORE_PORT}
INDEX_NAME=$1
BACKUP_NAME=$2
RESTORE_PATH=$3
RESTORE_IP=$4
RESTORE_PORT=$5
ES_USER="username"
ES_PASSWORD="password"
  
LOG_DIR=/tmp/es-recover
test -d ${LOG_DIR} || mkdir -p ${LOG_DIR}
  
### ---------------------------------------------------------------------------------------------------------------------------------------
   
#Registered snapshot
echo "Begin restore snapshot:" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
curl -u ${ES_USER}:${ES_PASSWORD} -H "Content-Type:application/json" -XPUT "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}" -d '{"type":"fs","settings":{"location":"'${RESTORE_PATH}'","max_restore_bytes_per_sec":"300mb","max_snapshot_bytes_per_sec":"300mb","compress":true}}'
if [ $? -ne 0 ];then
  echo "[`date  "%F %T"`]  ES registered failure ,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
  exit 1
fi
echo "Registered snapshot successfully!" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
#Check index's status 
IS_EXISTS=`curl -u ${ES_USER}:${ES_PASSWORD} -H "Content-Type:application/json" -XGET "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}/_all?pretty"|grep ${INDEX_NAME}|wc -l`
if [ ${IS_EXISTS} -ne 1 ];then
  echo "Index is not exists,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
  exit 1
fi
  
### ---------------------------------------------------------------------------------------------------------------------------------------
   
#Restore index
curl -u ${ES_USER}:${ES_PASSWORD} -XPOST -H "Content-Type:application/json" "http://${RESTORE_IP}:${RESTORE_PORT}/_snapshot/${BACKUP_NAME}/${BACKUP_NAME}/_restore?pretty" -d '{"indices":"'${INDEX_NAME}'","partial":true,"rename_pattern": "index_(. )","rename_replacement": "'${INDEX_NAME}'","ignore_unavailable":true,"ignore_index_settings":["index.routing.allocation.require.box_type"]}'
if [ ${IS_EXISTS} -ne 1 ];then
  echo "Index restore failed ,please check" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
  exit 1
fi
 
 
echo "Restore  index successfully!" >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log
echo "Index ${INDEX_NAME} restore end! " >> $${LOG_DIR}/${INDEX_NAME}_`date  %Y%m%d`.log

4.注意事宜

  • 1.需要建立统一的ES目录
  • 2.ES的主机的username,uid,gid需要保持一致
  • 3.ES备份的共享存储还可以通过sshfs挂载，下一篇文章我将介绍一下

5.参考文献

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-snapshots.html

备注： 如有疑问或者建议，请及时反馈13580480392@163.com。本人会及时反馈，感谢您的支持！

ElasticsearchService 数据备份

0 人点赞