问题描述:
警告文件磁盘已经满了,通过:df -h, du -sh * 等指令配合使用,发现当前系统任然存在大量可以使用的空间。大量剩余的磁盘空间不清楚怎么丢失了…
问题重现:
通过指令:lsof | grep deleted 指令,查看当前系统句柄未释放情况
代码语言:javascript复制[root@bigdata-hadoop2:/proc/21893/fd]# lsof | grep deleted
zabbix_ag 1791 zabbix 3wW REG 253,1 4 132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag 1806 zabbix 3w REG 253,1 4 132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag 1807 zabbix 3w REG 253,1 4 132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag 1808 zabbix 3w REG 253,1 4 132502 /var/tmp/zabbix_agentd.pid (deleted)
python 3946 root 14w REG 253,1 0 393228 /tmp/cmflistener-stderr---agent-3916-1532518455-HYSskK.log (deleted)
catalogd 21893 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 impala 259r REG 253,1 72198042 396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd 21893 impala 260r REG 253,1 1 396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd 21893 impala 262r REG 253,1 153712947 396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd 21893 impala 265r REG 253,1 1 396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd 21893 impala 267r REG 253,1 153714277 396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd 21893 impala 269r REG 253,1 1 396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd 21893 impala 270r REG 253,1 1 396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd 21893 2502 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 2502 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 2502 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 2502 impala 259r REG 253,1 72198042 396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd 21893 2502 impala 260r REG 253,1 1 396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd 21893 2502 impala 262r REG 253,1 153712947 396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd 21893 2502 impala 265r REG 253,1 1 396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd 21893 2502 impala 267r REG 253,1 153714277 396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd 21893 2502 impala 269r REG 253,1 1 396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd 21893 2502 impala 270r REG 253,1 1 396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd 21893 5443 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 5443 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 5443 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 5443 impala 259r REG 253,1 72198042 396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd 21893 5443 impala 260r REG 253,1 1 396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd 21893 5443 impala 262r REG 253,1 153712947 396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd 21893 5443 impala 265r REG 253,1 1 396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd 21893 5443 impala 267r REG 253,1 153714277 396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd 21893 5443 impala 269r REG 253,1 1 396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd 21893 5443 impala 270r REG 253,1 1 396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd 21893 5834 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 5834 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 5834 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 5834 impala 259r REG 253,1 72198042 396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd 21893 5834 impala 260r REG 253,1 1 396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd 21893 5834 impala 262r REG 253,1 153712947 396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd 21893 5834 impala 265r REG 253,1 1 396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd 21893 5834 impala 267r REG 253,1 153714277 396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd 21893 5834 impala 269r REG 253,1 1 396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd 21893 5834 impala 270r REG 253,1 1 396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd 21893 6688 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 6688 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 6688 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 6688 impala 259r REG 253,1 72198042 396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd 21893 6688 impala 260r REG 253,1 1 396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd 21893 6688 impala 262r REG 253,1 153712947 396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd 21893 6688 impala 265r REG 253,1 1 396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd 21893 6688 impala 267r REG 253,1 153714277 396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd 21893 6688 impala 269r REG 253,1 1 396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd 21893 6688 impala 270r REG 253,1 1 396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd 21893 6689 impala 254r REG 253,1 163923099 393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd 21893 6689 impala 255r REG 253,1 77023830 396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd 21893 6689 impala 258r REG 253,1 72198042 393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd 21893 6689 impala 259r REG 253,1 72198042
通过结果可以发现,当前系统存在大量文件已经被删除(deleted),可是文件句柄并没有被释放的情况。
问题已经有答案了,因为系统中存在大量文件句柄没有释放,导致当前占用的磁盘空间无法释放。文件句柄释放不掉的根本原因,是因为当前有线程对句柄进行引用。
找到占用磁盘的pid 手动kill掉解决