HBase NotServingRegionException

2020-09-28 15:46:06 浏览数 (1)

线上最近通过日志系统发现某BU的服务里HBase某时间段内有几十次如下这样的报错,error发生在同一个表的查询上,而业务对该表的查询仅限于点查询:

  • 根因分析: 其实是因为region split造成的抖动,trip:rt_vehicle_history_5m这张表最近一些region到10G大小开始split, 相当于那region 短暂close了然后再访问它就报错。 解决方案: hbase.client.retries.number 稍微调整大些,默认是3-->10, pause time 100ms~1s左右

split所需时间与表的文件数有关。

附上日志

代码语言:javascript复制
Fri Sep 25 08:38:03 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Fri Sep 25 08:38:04 CST 2020, RpcRetryingCaller{globalStartTime=1600994283900, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: trip:rt_vehicle_history_5m,538011400202:LS6A2E0E3KA00203,1593995983661.e42ea514e130d600b889cdb4520e378e. is closing
at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2812)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2807)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2801)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2523)
atorg.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

0 人点赞