Elasticsearch " Request cannot be executed; I/O reactor status:",如何处理?

2021-03-26 17:44:07 浏览数 (1)

一、前言|

近来,客户在使用JAVA 客户端操作ES集群的时候,出现 "Request cannot be executed; I/O reactor status: STOPPED

"错误,如何解决呢?我们来看看这个问题:

二、故障现象

报错截图1报错截图1
报错截图2报错截图2
代码语言:javascript复制
ParameterizedMessage[messagePattern=elasticSearch error->{}, stringArgs=[Request cannot be executed; I/O reactor status: STOPPED, com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED], throwable=com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED] com.weimob.saas.ec.goods.elasticsearch.monitor.ElasticSearchException: Request cannot be executed; I/O reactor status: STOPPED
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl.doExecute(ElasticSearch7ApiImpl.java:130)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl.search(ElasticSearch7ApiImpl.java:64)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl$$FastClassBySpringCGLIB$$a581fc99.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
at com.weimob.saas.ec.goods.elasticsearch.monitor.MonitorAspect.doAround(MonitorAspect.java:29)
at sun.reflect.GeneratedMethodAccessor1126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at com.weimob.saas.ec.goods.elasticsearch.ElasticSearch7ApiImpl$$EnhancerBySpringCGLIB$$241fe093.search(<generated>)
at com.weimob.saas.ec.goods.service.elasticsearch.impl.EsSearchServiceImpl.queryGoodsListWithOrderByPage(EsSearchServiceImpl.java:122)
at com.weimob.saas.ec.goods.service.elasticsearch.impl.EsSearchServiceImpl$$FastClassBySpringCGLIB$$142346b1.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

问题分析: 看起来是客户端的报错,提示连接重置了。

三、解决办法

这个异常通常有下面几个原因导致:

1、客户端文件句柄耗尽

2、并发连接es的客户端太多

3、客户端连接es后长时间没有数据读写又没有及时close然后下次有数据读写又复用同一个连接

4、es负载太高

客户可以排查下业务客户端是哪种原因造成。

其次,经过上面的排查,也可这样去进行操作:

这个预计是ES Rest Client的Bug,在社区里有几个非常高频的ISSUE,目前最新版本尚未解决。

基本原因:client内部的调用链为IOReactor->performRequestAsync的Listener -> onFailure,当短暂抖动触发onFailure中抛出异常时,最终导致整个IOReactor不可用,后续请求都受影响,也就是我们看到的需要重启才能恢复。

解决方案

1. 发生错误时,重建client进行查询重试;

2. 直接使用各类常用http库给es发送请求。

3. 重启观察看下,暂时建议在代码中判断类似出错后,重新建立client

参考如下社区的链接参考:

https://github.com/elastic/elasticsearch/issues/45115

https://github.com/elastic/elasticsearch/issues/39946

0 人点赞