Hadoop环境搭建与测试

2022-07-03 15:24:14 浏览数 (1)

Hadoop环境搭建请参看下面的连接

CentOS安装和配置Hadoop2.2.0  http://www.linuxidc.com/Linux/2014-01/94685.htm

Ubuntu 13.04上搭建Hadoop环境 http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 Hadoop 1.2.1版本集群配置 http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu上搭建Hadoop环境(单机模式 伪分布模式) http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu下Hadoop环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建Hadoop环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建Hadoop环境(在Winodws环境下用虚拟机虚拟两个Ubuntu系统进行搭建) http://www.linuxidc.com/Linux/2011-12/48894.htm

查看集群状态

[root@master bin]# hdfs dfsadmin -report Configured Capacity: 36729053184 (34.21 GB) Present Capacity: 13322559491 (12.41 GB) DFS Remaining: 13322240000 (12.41 GB) DFS Used: 319491 (312.00 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0

------------------------------------------------- Datanodes available: 2 (2 total, 0 dead)

Live datanodes: Name: 192.168.137.103:50010 (slave2) Hostname: slave2 Decommission Status : Normal Configured Capacity: 18364526592 (17.10 GB) DFS Used: 45056 (44 KB) Non DFS Used: 11702558720 (10.90 GB) DFS Remaining: 6661922816 (6.20 GB) DFS Used%: 0.00% DFS Remaining%: 36.28% Last contact: Thu Nov 06 21:26:34 CST 2014

Name: 192.168.137.102:50010 (slave1) Hostname: slave1 Decommission Status : Normal Configured Capacity: 18364526592 (17.10 GB) DFS Used: 274435 (268.00 KB) Non DFS Used: 11703934973 (10.90 GB) DFS Remaining: 6660317184 (6.20 GB) DFS Used%: 0.00% DFS Remaining%: 36.27% Last contact: Thu Nov 06 21:26:31 CST 2014

查看文件块组成:

[root@master bin]# hdfs fsck / -files -blocks Status: HEALTHY Total size: 219351 B Total dirs: 11 Total files: 12 Total symlinks: 0 Total blocks (validated): 10 (avg. block size 21935 B) Minimally replicated blocks: 10 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 2 Number of racks: 1 FSCK ended at Thu Nov 06 21:27:34 CST 2014 in 29 milliseconds

The filesystem under path '/' is HEALTHY [root@master bin]# [hadoop@master hadoop2.2]$

查看各节点状态: http://192.168.56.101:50070 查看ResourceManager上cluster运行状态: http://192.168.56.101:8088

在环境搭建的过程中,如果出现任何问题,都要去查看日志 日志路径是:/home/hadoop/hadoop2.2/logs

在配置完成HADOOP_HOME之后,并且使之生效,那么接下来就进行测试,启动hadoop 首先在/目录下创建input文件

[root@master /]# vim input,在该文件中输入如下内容:I am a very good person! I love you America ! 将之上传到hdfs上:[root@master /]# hadoop fs -put /input /input 在hadoop的bin目录下执行:[root@master bin]# ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output 这样执行完毕之后可以看到:output文件夹下有两个文件 [root@master ~]# hadoop fs -ls /output Found 2 items -rw-r--r-- 1 root supergroup 0 2014-11-06 21:21 /output/_SUCCESS -rw-r--r-- 1 root supergroup 64 2014-11-06 21:21 /output/part-r-00000 接着可以查看Wordcount的统计结果: [root@master bin]# hadoop fs -cat /output/part-r-00000 ! 1 America 1 I 2 a 1 am 1 good 1 love 1 person! 1 very 1 you 1 [root@master bin]#

更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

0 人点赞