HDFS 线程参数DataXceiver 引发的故障

2019-03-15 18:19:48 浏览数 (1)

一、故障现象

腾讯云大数据团队服务的某个大客户,hadoop集群超过300台服务器。因为大数据平台承载的业务程序非常多(每天超过5万次任务运行在yarn)、datanode的IO压力很大,在今天下午datanode出现大面积故障。通过日志查找,发现以下信息:

代码语言:javascript复制
2019-03-15 17:19:11,428 ERROR datanode.DataNode (DataXceiver.java:run(280)) - tbds-10-166-114-46:50010:DataXceiver error processing WRITE_BLOCK operation  src: /86.1.50.18
:1855 dst: /10.166.114.46:50010
java.io.IOException: Premature EOF from inputStream
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:501)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:895)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:801)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253)
        at java.lang.Thread.run(Thread.java:745)
2019-03-15 17:19:11,432 INFO  datanode.DataNode (BlockReceiver.java:receiveBlock(934)) - Exception for BP-2079495716-10.166.114.44-1515312742876:blk_2115516619_1047456738
java.io.IOException: Premature EOF from inputStream
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:501)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:895)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:801)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253)
        at java.lang.Thread.run(Thread.java:745)
2019-03-15 17:19:11,432 ERROR datanode.DataNode (DataXceiver.java:writeBlock(771)) - DataNode{data=FSDataset{dirpath='[/data/hadoop/hdfs/data/current, /data1/hadoop/hdfs/data/current, /data2/hadoop/hdfs/data/current, /data3/hadoop/hdfs/data/current, /data4/hadoop/hdfs/data/current, /data5/hadoop/hdfs/data/current, /data6/hadoop/hdfs/data/current, /data7/hadoop/hdfs/data/current, /data8/hadoop/hdfs/data/current, /data9/hadoop/hdfs/data/current, /data10/hadoop/hdfs/data/current, /data11/hadoop/hdfs/data/current, /data12/hadoop/hdfs/data/current, /data13/hadoop/hdfs/data/current, /data14/hadoop/hdfs/data/current, /data15/hadoop/hdfs/data/current]'}, localName='tbds-10-166-114-46:50010', datanodeUuid='759bc6e6-0d8d-450a-a723-05e0202e58fa', xmitsInProgress=0}:Exception transfering block BP-2079495716-10.166.114.44-1515312742876:blk_2116563892_1048520277 to mirror 10.166.114.57:50010: java.io.IOException: Connection reset by peer
2019-03-15 17:19:11,432 INFO  datanode.DataNode (DataXceiver.java:writeBlock(834)) - opWriteBlock BP-2079495716-10.166.114.44-1515312742876:blk_2116563892_1048520277 received exception java.io.IOException: Connection reset by peer

二、故障分析

从日志提取最关键的信息 “DataXceiver error processing WRITE_BLOCK operation”, 结合日志全面的分析,很明显看出datanode故障的原因是数据传出线程数量不足导致的。因此,有两个优化方法:1、datanode所在的linux服务器提高文件句柄参数; 2、增加HDFS的datanode句柄参数:dfs.datanode.max.transfer.threads。

三、故障修复和优化

1、HDFS参数优化

(1)对于hadoop1.x版本,需要将参数:

dfs.datanode.max.xcievers = 16384 (或者设置更大)

(2)对hadoop2.x版本,需要将参数:

dfs.datanode.max.transfer.threads = 16384 (或者设置更大)

2、linux文件数量参数优化

(1)进入到目录:/etc/security 修改limits.conf。

把非注释的内容都删除,然后添加:

* soft nofile 200001

* hard nofile 200002

root soft nofile 200001

root hard nofile 200002

* - nofile 200002

(注意:上述具体设置多少值,由实际情况决定)

(2)再进入到 /etc/security/limits.d 目录:

单独为hdfs、yarn等用户增加配置。比如,可以增加 hdfs.conf、mapreduce.conf、yarn.conf、hive.conf 几个配置文件。

比如增加的hdfs.conf 内容为:

hdfs soft nofile 200001

hdfs soft nproc 200002

表示单独为hdfs用户配置文件句柄参数。

(注意:上述具体设置多少值,由实际情况决定。上述值的含义是具体hdfs、yarn、mapred、hive 这4个用户的值。)

(3)修改配置文件:/etc/sysctl.conf

将 fs.file-max=65535 配置项,设置为 fs.file-max=655350,也就是增大十倍。如果是大型集群,建议这个值设置超过100万。

( 注意:fs.file-max 表示整个linux系统允许打开的文件数量,涵盖root、hdfs、yarn、hive等所有用户总和,因此这个值一定要设置非常大。)

上述配置好之后,重新登录linux服务器。然后分别在 root、yarn、hive、hdfs等用户,执行 ulimit -a 看看生效没有:

查看用户的文件句柄设置查看用户的文件句柄设置

0 人点赞