我的一个项目并发200 ,观察服务器的日志发现出现了大量的熔断超时偶尔会闪现出RejectedExecutionException: Rejected command because thread-pool queueSize is at rejection threshold。
在高并发的前提下出现熔断超时:
1.先确定是否是自己接口的问题,接口平均响应时长是多少?
举个例子,倘若平均响应时长为200ms,单线程处理的话5次/秒,tomcat最大并发线程数按照100个来算的话,那就是100*5次/秒=500次/秒
那也就是正常情况下,可以承受500次/秒的并发请求。
但是现在200次就不停的刷超时熔断!
2.在接口上使用@HystrixCommand注解少配置了参数
之前是按照下面的方式配置的
代码语言:javascript复制@HystrixCommand(fallbackMethod = "your_interface", commandProperties = {
@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000") })
注意这里有个坑:
- hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests 如果并发数达到该设置值,请求会被拒绝和抛出异常并且fallback不会被调用。默认10
- hystrix.threadpool.default.coreSize 并发执行的最大线程数,默认10
- hystrix.threadpool.default.maxQueueSize BlockingQueue的最大队列数,
- hystrix.threadpool.default.queueSizeRejectionThreshold 即使maxQueueSize没有达到,达到queueSizeRejectionThreshold该值后,请求也会被拒绝。
原来是因为这里如果不显示设置coreSize的化,那么就会按照默认的10来进行处理。这就很好理解为什么大量熔断超时了,10-thread*5次/秒/单线程=50次/秒<200次并发量。
然后调整参数,最后解决了这个问题:
代码语言:javascript复制@HystrixCommand(fallbackMethod = "otaCheckUpdateFallback", commandProperties = {
@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "5000"),
}, threadPoolProperties = { @HystrixProperty(name = "coreSize", value = "100"),
@HystrixProperty(name = "maxQueueSize", value = "20"),
@HystrixProperty(name = "queueSizeRejectionThreshold", value = "20")})
同样将参数queueSizeRejectionThreshold调大之后,即便请求频率超过预期,那么当达到queueSizeRejectionThreshold之后会抛出下面的异常。
代码语言:javascript复制2018-04-11 20:33:16.294 ERROR [http-nio-9098-exec-30][DirectJDKLog.java:181] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: otaCheckUpdate fallback execution rejected.] with root cause
com.netflix.hystrix.exception.HystrixRuntimeException: your_interface fallback execution rejected.
at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1043) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:875) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:993) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:60) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:607) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar!/:1.1.10]
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1194) ~[hystrix-core-1.5.10.jar!/:1.5.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar!/:1.1.10]
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar!/:1.1.10]