elasticsearch简介:来自Elasticsearch权威指南
Elasticsearch是一个基于Apache Lucene(TM)的开源搜索引擎。无论在开源还是专有领域,Lucene可以被认为是迄今为止最先进、性能最好的、功能最全的搜索引擎库。但是,Lucene只是一个库。想要使用它,你必须使用Java来作为开发语言并将其直接集成到你的应用中,更糟糕的是,Lucene非常复杂,你需要深入了解检索的相关知识来理解它是如何工作的。 Elasticsearch也使用Java开发并使用Lucene作为其核心来实现所有索引和搜索的功能,但是它的目的是通过简单的RESTful API来隐藏Lucene的复杂性,从而让全文搜索变得简单。 不过,Elasticsearch不仅仅是Lucene和全文搜索,我们还能这样去描述它: 分布式的实时文件存储,每个字段都被索引并可被搜索 分布式的实时分析搜索引擎 可以扩展到上百台服务器,处理PB级结构化或非结构化数据 而且,所有的这些功能被集成到一个服务里面,你的应用可以通过简单的RESTful API、各种语言的客户端甚至命令行与之交互。 上手Elasticsearch非常容易。它提供了许多合理的缺省值,并对初学者隐藏了复杂的搜索引擎理论。它开箱即用(安装即可使用),只需很少的学习既可在生产环境中使用。 Elasticsearch在Apache 2 license下许可使用,可以免费下载、使用和修改。 随着你对Elasticsearch的理解加深,你可以根据不同的问题领域定制Elasticsearch的高级特性,这一切都是可配置的,并且配置非常灵活。
通过上述描述对es有初步的了解后, 我们知道要使用es必须安装java环境,最好使用8.0以后的版本
es下载:下载地址
es提供多种格式的下载包,有zip、tar、deb,rpm,msi,其中zip和tar的是直接解压后就可以当成服务,无需安装,而rpm和msi则需要安装,个人喜欢免安装方式的
环境:centOS7 java1.8 1.下载es包,放到用户根目录下,然后解压,如图
2.启动服务,进入到bin目录下, 指定./elasticsearch命令
a.用root用户启动服务
代码语言:javascript复制
[root@localhost bin]# ./elasticsearch
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
[2018-03-31T13:39:08,369][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.3.jar:6.2.3]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.3.jar:6.2.3]
... 6 more
使用root用户启动服务发现报错,其中Caused by: java.lang.RuntimeException: can not run elasticsearch as root at 说明不用使用root用户启动服务,原因是由于ElasticSearch可以接收用户输入的脚本并且执行,为了系统安全考虑,所以只能注册一个普通用户。
b.用普通用户启动服务
代码语言:javascript复制
[a123@localhost bin]$ ./elasticsearch
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
[2018-03-31T13:45:05,706][INFO ][o.e.n.Node ] [] initializing ...
[2018-03-31T13:45:05,756][INFO ][o.e.e.NodeEnvironment ] [Z8qVV**] using [1] data paths, mounts [[/home (/dev/mapper/VolGroup-lv_home)]], net usable_space [10.5gb], net total_space [11.2gb], types [ext4]
[2018-03-31T13:45:05,756][INFO ][o.e.e.NodeEnvironment ] [Z8qVV**] heap size [60.8mb], compressed ordinary object pointers [true]
[2018-03-31T13:45:05,758][INFO ][o.e.n.Node ] node name [Z8qVV**] derived from node ID [Z8qVV**vR2mKeNFIHhvVaw]; set [node.name] to override
[2018-03-31T13:45:05,758][INFO ][o.e.n.Node ] version[6.2.3], pid[4367], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-31T13:45:05,758][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX: UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX: UseCMSInitiatingOccupancyOnly, -XX: AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.N9ttLIGb, -XX: HeapDumpOnOutOfMemoryError, -XX: PrintGCDetails, -XX: PrintGCDateStamps, -XX: PrintTenuringDistribution, -XX: PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX: UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/a123/elasticsearch-6.2.3, -Des.path.conf=/home/a123/elasticsearch-6.2.3/config, -Xms64m, -Xmx64m, -Xmn32m]
[2018-03-31T13:45:06,236][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [aggs-matrix-stats]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [analysis-common]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [ingest-common]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-expression]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-mustache]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-painless]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [mapper-extras]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [parent-join]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [percolator]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [rank-eval]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [reindex]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [repository-url]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [transport-netty4]
[2018-03-31T13:45:06,237][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [tribe]
[2018-03-31T13:45:06,238][INFO ][o.e.p.PluginsService ] [Z8qVV**] no plugins loaded
[2018-03-31T13:45:08,161][INFO ][o.e.d.DiscoveryModule ] [Z8qVV**] using discovery type [zen]
[2018-03-31T13:45:08,611][INFO ][o.e.n.Node ] initialized
[2018-03-31T13:45:08,611][INFO ][o.e.n.Node ] [Z8qVV**] starting ...
[2018-03-31T13:45:08,712][INFO ][o.e.t.TransportService ] [Z8qVV**] publish_address {192.168.199.102:9300}, bound_addresses {[::]:9300}
[2018-03-31T13:45:08,719][INFO ][o.e.b.BootstrapChecks ] [Z8qVV**] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3869] for user [a123] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-03-31T13:45:08,737][INFO ][o.e.n.Node ] [Z8qVV**] stopping ...
[2018-03-31T13:45:08,760][INFO ][o.e.n.Node ] [Z8qVV**] stopped
[2018-03-31T13:45:08,760][INFO ][o.e.n.Node ] [Z8qVV**] closing ...
[2018-03-31T13:45:08,783][INFO ][o.e.n.Node ] [Z8qVV**] closed
[a123@localhost bin]$
使用普通用户启动发现依旧失败,查看原因是:是各种配置太低,调整相应配置就好
代码语言:javascript复制
ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3869] for user [a123] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决办法:
在/etc/security/limits.conf文件末尾加入
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
修改/etc/security/limits.d/20-nproc.conf* soft nproc 4096
改成
* soft nproc 8192
在/etc/systml.conf中添加vm.max_map_count=655360
最后执行命令 ‘sysctl -p’
再启动服务器:
代码语言:javascript复制
[a123@localhost bin]$ ./elasticsearch
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
Picked up _JAVA_OPTIONS: -Xms64m -Xmx64m -Xmn32m
[2018-03-31T14:27:29,543][INFO ][o.e.n.Node ] [] initializing ...
[2018-03-31T14:27:29,610][INFO ][o.e.e.NodeEnvironment ] [Z8qVV**] using [1] data paths, mounts [[/home (/dev/mapper/VolGroup-lv_home)]], net usable_space [10.5gb], net total_space [11.2gb], types [ext4]
[2018-03-31T14:27:29,610][INFO ][o.e.e.NodeEnvironment ] [Z8qVV**] heap size [60.8mb], compressed ordinary object pointers [true]
[2018-03-31T14:27:29,611][INFO ][o.e.n.Node ] node name [Z8qVV**] derived from node ID [Z8qVV**vR2mKeNFIHhvVaw]; set [node.name] to override
[2018-03-31T14:27:29,612][INFO ][o.e.n.Node ] version[6.2.3], pid[1630], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-31T14:27:29,612][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX: UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX: UseCMSInitiatingOccupancyOnly, -XX: AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.1kG1zSyc, -XX: HeapDumpOnOutOfMemoryError, -XX: PrintGCDetails, -XX: PrintGCDateStamps, -XX: PrintTenuringDistribution, -XX: PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX: UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/a123/elasticsearch-6.2.3, -Des.path.conf=/home/a123/elasticsearch-6.2.3/config, -Xms64m, -Xmx64m, -Xmn32m]
[2018-03-31T14:27:30,085][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [aggs-matrix-stats]
[2018-03-31T14:27:30,085][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [analysis-common]
[2018-03-31T14:27:30,085][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [ingest-common]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-expression]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-mustache]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [lang-painless]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [mapper-extras]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [parent-join]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [percolator]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [rank-eval]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [reindex]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [repository-url]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [transport-netty4]
[2018-03-31T14:27:30,086][INFO ][o.e.p.PluginsService ] [Z8qVV**] loaded module [tribe]
[2018-03-31T14:27:30,087][INFO ][o.e.p.PluginsService ] [Z8qVV**] no plugins loaded
[2018-03-31T14:27:32,083][INFO ][o.e.d.DiscoveryModule ] [Z8qVV**] using discovery type [zen]
[2018-03-31T14:27:32,500][INFO ][o.e.n.Node ] initialized
[2018-03-31T14:27:32,501][INFO ][o.e.n.Node ] [Z8qVV**] starting ...
[2018-03-31T14:27:32,601][INFO ][o.e.t.TransportService ] [Z8qVV**] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-03-31T14:27:35,673][INFO ][o.e.c.s.MasterService ] [Z8qVV**] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Z8qVV**}{Z8qVV**vR2mKeNFIHhvVaw}{mjnWNrr7RPWtV7JcjOUYFQ}{127.0.0.1}{127.0.0.1:9300}
[2018-03-31T14:27:35,682][INFO ][o.e.c.s.ClusterApplierService] [Z8qVV**] new_master {Z8qVV**}{Z8qVV**vR2mKeNFIHhvVaw}{mjnWNrr7RPWtV7JcjOUYFQ}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {Z8qVV**}{Z8qVV**vR2mKeNFIHhvVaw}{mjnWNrr7RPWtV7JcjOUYFQ}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-31T14:27:35,718][INFO ][o.e.g.GatewayService ] [Z8qVV**] recovered [0] indices into cluster_state
[2018-03-31T14:27:35,722][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z8qVV**] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-03-31T14:27:35,722][INFO ][o.e.n.Node ] [Z8qVV**] started
验证启动:
代码语言:javascript复制
[root@localhost limits.d]# curl 127.0.0.1:9200
{
"name" : "Z8qVV**",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "APnNan9dS-6aFlVFoosxqA",
"version" : {
"number" : "6.2.3",
"build_hash" : "c59ff00",
"build_date" : "2018-03-13T10:06:29.741383Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
但是此处有一个问题, 就是只能本地访问, 就是局域网内的其他机器也不能访问, 猜想可能是防火墙的原因, 但是,关闭防火墙之后依旧不能访问到服务, 最后发现需要更改配置文件
在config下有一个叫elasticsearch.yml的文件, 这个也是es的配置文件, es支持多种格式的配置文件,如properties,json等格式,此处我们需要更改配置文件中的:
# ---------------------------------- Network -------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
#network.host: 0.0.0.0
# Set a custom port for HTTP:
#http.port: 9200
# For more information, consult the network module documentation.
需要更改配置为# ---------------------------------- Network -------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 0.0.0.0
# Set a custom port for HTTP:
http.port: 9200
# For more information, consult the network module documentation.
c.如果elasticsearch 解压后不是放到用户根目录下,而是放到/usr/local下或者是其他系统文件夹下, 这是即使配置好,也可能因为权限的问题无法启动服务,需要更改elasticsearch文件夹及其子文件的权限,更改命名chmod,
3.配置说明:下图来自elasticsearch技术解析与实战