前言
Apache Hadoop 是一个专注于可靠，弹性，分布式计算框架的开源软件项目
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
主要包含以下几个模块：
- Hadoop Common : 支持其它 hadoop 模块的通用工具
- Hadoop Distributed File System : 简称 HDFS , 给应用数据提供高吞吐性能的分布式文件系统
- Hadoop YARN : 工作调度与集群资源管理的框架
- Hadoop MapReduce : 大数据集的并行处理系统
Hadoop 生态圈中的其它项目可以参考 Hadoop-related projects
Tip: 当前的最新稳定版为 Hadoop Release 2.8.1 发布于 08 June, 2017
这里根据官方的文档给出最新版 hadoop 在 Centos7 下的单机布署方案，详细可以参考 Setting up a Single Node Cluster
此文章借鉴了 Hadoop Wiki 中的部分内容和 Running Hadoop on Ubuntu Linux (Single-Node Cluster) 中部分操作
概要
系统环境
代码语言：javascript复制[root@much ~]# hostnamectl
Static hostname: much
Icon name: computer-vm
Chassis: vm
Machine ID: 33dc28f7e76c4903ad9b603b77e29a7c
Boot ID: 8cfd2a8aec4f4235b4776f7cd2fdfdb1
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-514.21.1.el7.x86_64
Architecture: x86-64
[root@much ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:e3:df:87 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 84190sec preferred_lft 84190sec
inet6 fe80::2bb7:5b3:9584:d8eb/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d3:ec:e7 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.207/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fed3:ece7/64 scope link
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
[root@much ~]# uname -a
Linux much 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@much ~]# 目标
- 构建一个 hadoop 的单点伪集群
- hdfs 的基本操作
基础依赖
- Java
- ssh
由于 Hadoop 是使用 Java 语言开发出来的项目，所以 Java 运行环境是必要的, Hadoop 支持的 Java 版本可以参考 Hadoop Java Versions
之所以需要 sshd , 是因为 Hadoop 是通过 ssh 执行脚本来管理远程节点
安装 Java
代码语言：javascript复制[root@much ~]# rpm -qa | grep epel
[root@much ~]# yum list all | grep -i epel
epel-release.noarch 7-9 extras
[root@much ~]# yum install epel-release.noarch
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.yun-idc.com
* c7-media:
* extras: mirrors.163.com
* updates: centos.ustc.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-9 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================
Package Arch Version Repository Size
========================================================================================
Installing:
epel-release noarch 7-9 extras 14 k
Transaction Summary
========================================================================================
Install 1 Package
Total download size: 14 k
Installed size: 24 k
Is this ok [y/d/N]: y
Downloading packages:
epel-release-7-9.noarch.rpm | 14 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : epel-release-7-9.noarch 1/1
Verifying : epel-release-7-9.noarch 1/1
Installed:
epel-release.noarch 0:7-9
Complete!
[root@much ~]#
[root@much ~]# rpm -qa | grep epel
epel-release-7-9.noarch
[root@much ~]#
[root@much ~]# yum list all | grep -i java | grep -i jdk
java-1.7.0-openjdk.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @updates
java-1.7.0-openjdk-headless.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @updates
java-1.8.0-openjdk.x86_64 1:1.8.0.131-3.b12.el7_3 @updates
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.131-3.b12.el7_3 @updates
java-1.6.0-openjdk.x86_64 1:1.6.0.41-1.13.13.1.el7_3 updates
java-1.6.0-openjdk-demo.x86_64 1:1.6.0.41-1.13.13.1.el7_3 updates
java-1.6.0-openjdk-devel.x86_64 1:1.6.0.41-1.13.13.1.el7_3 updates
java-1.6.0-openjdk-javadoc.x86_64 1:1.6.0.41-1.13.13.1.el7_3 updates
java-1.6.0-openjdk-src.x86_64 1:1.6.0.41-1.13.13.1.el7_3 updates
java-1.7.0-openjdk-accessibility.x86_64 1:1.7.0.141-2.6.10.1.el7_3 updates
java-1.7.0-openjdk-demo.x86_64 1:1.7.0.141-2.6.10.1.el7_3 updates
java-1.7.0-openjdk-devel.x86_64 1:1.7.0.141-2.6.10.1.el7_3 updates
java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.141-2.6.10.1.el7_3 updates
java-1.7.0-openjdk-src.x86_64 1:1.7.0.141-2.6.10.1.el7_3 updates
java-1.8.0-openjdk.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-accessibility.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-accessibility-debug.x86_64
java-1.8.0-openjdk-debug.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-debug.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-debuginfo.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-debuginfo.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-demo.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-demo-debug.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-devel.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-devel-debug.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-devel-debug.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-headless.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-headless-debug.i686 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-headless-debug.x86_64
java-1.8.0-openjdk-javadoc.noarch 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-javadoc-debug.noarch 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-javadoc-zip.noarch 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-javadoc-zip-debug.noarch
java-1.8.0-openjdk-src.x86_64 1:1.8.0.141-1.b16.el7_3 updates
java-1.8.0-openjdk-src-debug.x86_64 1:1.8.0.141-1.b16.el7_3 updates
ldapjdk-javadoc.noarch 4.18-16.el7_3 updates
[root@much ~]# yum install java-1.8.0-openjdk.x86_64 java-1.8.0-openjdk-devel.x86_64 java-1.8.0-openjdk-headless.x86_64
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.yun-idc.com
* c7-media:
* epel: mirrors.ustc.edu.cn
* extras: mirrors.163.com
* updates: centos.ustc.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.131-3.b12.el7_3 will be updated
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3 will be an update
---> Package java-1.8.0-openjdk-devel.x86_64 1:1.8.0.141-1.b16.el7_3 will be installed
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.131-3.b12.el7_3 will be updated
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================
Package Arch Version Repository Size
========================================================================================
Installing:
java-1.8.0-openjdk-devel x86_64 1:1.8.0.141-1.b16.el7_3 updates 9.7 M
Updating:
java-1.8.0-openjdk x86_64 1:1.8.0.141-1.b16.el7_3 updates 234 k
java-1.8.0-openjdk-headless x86_64 1:1.8.0.141-1.b16.el7_3 updates 31 M
Transaction Summary
========================================================================================
Install 1 Package
Upgrade 2 Packages
Total download size: 41 M
Is this ok [y/d/N]: y
Downloading packages:
updates/7/x86_64/prestodelta | 954 kB 00:00:00
Delta RPMs reduced 234 k of updates to 58 k (75% saved)
(1/3): java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3_1.8.0.141-1.b16. | 58 kB 00:00:00
(2/3): java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64.rpm | 9.7 MB 00:00:03
(3/3): java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64. | 31 MB 00:00:03
----------------------------------------------------------------------------------------
Total 10 MB/s | 41 MB 00:04
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : 1:java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64 1/5
warning: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/lib/security/java.security created as /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/lib/security/java.security.rpmnew
Updating : 1:java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64 2/5
Installing : 1:java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64 3/5
Cleanup : 1:java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64 4/5
Cleanup : 1:java-1.8.0-openjdk-headless-1.8.0.131-3.b12.el7_3.x86_64 5/5
Verifying : 1:java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64 1/5
Verifying : 1:java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64 2/5
Verifying : 1:java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64 3/5
Verifying : 1:java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64 4/5
Verifying : 1:java-1.8.0-openjdk-headless-1.8.0.131-3.b12.el7_3.x86_64 5/5
Installed:
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.141-1.b16.el7_3
Updated:
java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3
Complete!
[root@much ~]# 检查 JAVA_HOME
代码语言：javascript复制[root@much ~]# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
[root@much ~]# which java
/usr/bin/java
[root@much ~]# ll /usr/bin/java
lrwxrwxrwx. 1 root root 22 8月 6 23:55 /usr/bin/java -> /etc/alternatives/java
[root@much ~]# ll /etc/alternatives/java
lrwxrwxrwx. 1 root root 73 8月 6 23:55 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
-rwxr-xr-x. 1 root root 7336 7月 21 06:37 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/
total 4
drwxr-xr-x. 2 root root 4096 8月 6 23:55 bin
drwxr-xr-x. 3 root root 132 8月 6 23:55 include
drwxr-xr-x. 4 root root 28 8月 6 23:55 jre
drwxr-xr-x. 3 root root 144 8月 6 23:55 lib
drwxr-xr-x. 2 root root 204 8月 6 23:55 tapset
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/bin/
total 420
-rwxr-xr-x. 1 root root 7400 7月 21 06:37 appletviewer
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 extcheck
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 idlj
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jar
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 jarsigner
-rwxr-xr-x. 1 root root 7336 7月 21 06:37 java
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 javac
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 javadoc
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 javah
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 javap
-rwxr-xr-x. 1 root root 2806 7月 21 06:01 java-rmi.cgi
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jcmd
-rwxr-xr-x. 1 root root 7416 7月 21 06:37 jconsole
-rwxr-xr-x. 1 root root 7400 7月 21 06:37 jdb
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jdeps
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jhat
-rwxr-xr-x. 1 root root 7448 7月 21 06:37 jinfo
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jjs
-rwxr-xr-x. 1 root root 7448 7月 21 06:37 jmap
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jps
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 jrunscript
-rwxr-xr-x. 1 root root 7408 7月 21 06:37 jsadebugd
-rwxr-xr-x. 1 root root 7456 7月 21 06:37 jstack
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 jstat
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 jstatd
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 keytool
-rwxr-xr-x. 1 root root 7400 7月 21 06:37 native2ascii
-rwxr-xr-x. 1 root root 7456 7月 21 06:37 orbd
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 pack200
-rwxr-xr-x. 1 root root 7400 7月 21 06:37 policytool
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 rmic
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 rmid
-rwxr-xr-x. 1 root root 7400 7月 21 06:37 rmiregistry
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 schemagen
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 serialver
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 servertool
-rwxr-xr-x. 1 root root 7464 7月 21 06:37 tnameserv
-rwxr-xr-x. 1 root root 103424 7月 21 06:37 unpack200
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 wsgen
-rwxr-xr-x. 1 root root 7392 7月 21 06:37 wsimport
-rwxr-xr-x. 1 root root 7384 7月 21 06:37 xjc
[root@much ~]#确保 ssh rsync 已经安装
代码语言：javascript复制[root@much ~]# yum install openssh rsync
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.yun-idc.com
* c7-media:
* epel: mirrors.ustc.edu.cn
* extras: mirrors.163.com
* updates: centos.ustc.edu.cn
Package openssh-6.6.1p1-35.el7_3.x86_64 already installed and latest version
Package rsync-3.0.9-17.el7.x86_64 already installed and latest version
Nothing to do
[root@much ~]#关闭 selinux
代码语言：javascript复制[root@much ~]# getenforce
Enforcing
[root@much ~]# setenforce 0
[root@much ~]# getenforce
Permissive
[root@much ~]#创建 hadoop 用户
代码语言：javascript复制[root@much ~]# useradd hadoop
[root@much ~]# passwd hadoop
Changing password for user hadoop.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
[root@much ~]# 配置用户免密码登录
代码语言：javascript复制[root@much ~]# su - hadoop
[hadoop@much ~]$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1 hadoop@much
The key's randomart image is:
--[ RSA 2048]----
| |
| . |
| . . |
| . |
| o o E |
| o o |
| = * o |
| @.. . |
| . X |
-----------------
[hadoop@much ~]$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
[hadoop@much ~]$ chmod 600 $HOME/.ssh/authorized_keys
[hadoop@much ~]$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is 0c:52:20:0a:00:e3:1a:5d:c6:fc:79:b3:e8:6e:d6:f1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Last login: Mon Aug 7 00:04:13 2017
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much ~]$ ssh localhost
Last login: Mon Aug 7 00:05:48 2017 from localhost
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much ~]$ 代码语言：javascript复制Tip: 配置证书登录的过程中，要注意 authorized_keys 文件的权限，如果没有达到预期效果可以通过
ssh -vvv localhost来进行 debug
[hadoop@much ~]$ ssh -vvv localhost
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/hadoop/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/hadoop/.ssh/id_rsa type 1
debug1: identity file /home/hadoop/.ssh/id_rsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_dsa type -1
debug1: identity file /home/hadoop/.ssh/id_dsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ed25519 type -1
debug1: identity file /home/hadoop/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "localhost" from file "/home/hadoop/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/hadoop/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 0c:52:20:0a:00:e3:1a:5d:c6:fc:79:b3:e8:6e:d6:f1
debug3: load_hostkeys: loading entries for host "localhost" from file "/home/hadoop/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/hadoop/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/hadoop/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/hadoop/.ssh/id_rsa (0x7f284841f2c0),
debug2: key: /home/hadoop/.ssh/id_dsa ((nil)),
debug2: key: /home/hadoop/.ssh/id_ecdsa ((nil)),
debug2: key: /home/hadoop/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1001)
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1001)
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/hadoop/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1
debug3: sign_and_send_pubkey: RSA cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([::1]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IPV6_TCLASS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env HOSTNAME
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env HISTSIZE
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env LESSOPEN
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Aug 7 00:05:55 2017 from localhost
[hadoop@much ~]$ 关闭 IPv6
代码语言：javascript复制[root@much ~]# sysctl -a | grep -i net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
[root@much ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
[root@much ~]#
[root@much ~]# vim /etc/sysctl.conf
[root@much ~]# sysctl -p
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
[root@much ~]# sysctl -a | grep -i net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
[root@much ~]#
[root@much ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
[root@much ~]#
[root@much ~]# cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1
[root@much ~]# Note: 如果不关闭 IPv６ 会在后面的 MapReduce 任务调度过程中出错，因为
:8032端口会绑定到 IPv6 的地址上面，从而使本地的 IPv4 的访问方式访问不到服务，而不断重试
下载 Hadoop 包
Hadoop 包的下载地址可以参考 Apache Hadoop Releases Download
这里我们下载最新的稳定版 hadoop-2.8.1.tar.gz
