一、前言|
近来,客户在使用JAVA 客户端操作ES集群的时候,出现 "Request cannot be executed; I/O reactor status: STOPPED
"错误,如何解决呢?我们来看看这个问题:
二、故障现象
代码语言:javascript复制ParameterizedMessage[messagePattern=elasticSearch error->{}, stringArgs=[Request cannot be executed; I/O reactor status: STOPPED, com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED], throwable=com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED] com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl.doExecute(ElasticSearch7ApiImpl.java:130)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl.search(ElasticSearch7ApiImpl.java:64)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl$$FastClassBySpringCGLIB$$a581fc99.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
at com.weimob.saas.ec.goods.elasticsearch.monitor.MonitorAspect.doAround(MonitorAspect.java:29)
at sun.reflect.GeneratedMethodAccessor1126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl$$EnhancerBySpringCGLIB$$241fe093.search(<generated>)
at com.weimob.saas.ec.goods.service.elasticsearch.impl.EsSearchServiceImpl.queryGoodsListWithOrderByPage(EsSearchServiceImpl.java:122)
at com.weimob.saas.ec.goods.service.elasticsearch.impl.EsSearchServiceImpl$$FastClassBySpringCGLIB$$142346b1.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
问题分析: 看起来是客户端的报错,提示连接重置了。
三、解决办法
这个异常通常有下面几个原因导致:
1、客户端文件句柄耗尽
2、并发连接es的客户端太多
3、客户端连接es后长时间没有数据读写又没有及时close然后下次有数据读写又复用同一个连接
4、es负载太高
客户可以排查下业务客户端是哪种原因造成。
其次,经过上面的排查,也可这样去进行操作:
这个预计是ES Rest Client的Bug,在社区里有几个非常高频的ISSUE,目前最新版本尚未解决。
基本原因:client内部的调用链为IOReactor->performRequestAsync的Listener -> onFailure,当短暂抖动触发onFailure中抛出异常时,最终导致整个IOReactor不可用,后续请求都受影响,也就是我们看到的需要重启才能恢复。
解决方案:
1. 发生错误时,重建client进行查询重试;
2. 直接使用各类常用http库给es发送请求。
3. 重启观察看下,暂时建议在代码中判断类似出错后,重新建立client
参考如下社区的链接参考:
https://github.com/elastic/elasticsearch/issues/45115
https://github.com/elastic/elasticsearch/issues/39946