hadoop 单机伪分布式安装步骤

2021-09-06 10:09:18 浏览数 (1)

文章目录

  • ​
    • 1. 安装 Java
    • 2. 配置SSH无密码登录
    • 3. 下载 hadoop
    • 4. 配置环境变量
    • 5. 报错处理参考

环境 Centos7

参考：

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

https://blog.csdn.net/bingduanlbd/article/details/51874195

1. 安装 Java

注意版本要求：

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop Java Versions

Java 安装：https://cloud.tencent.com/developer/article/1872757

2. 配置SSH无密码登录

代码语言：javascript复制
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost

第一条输入后，一直回车，无密码

3. 下载 hadoop

下载地址：https://hadoop.apache.org/releases.html，选择 binary downloads

解压到路径，我的是 /opt/hadoop-3.3.0/

代码语言：javascript复制
tar -zxvf 包名

4. 配置环境变量

代码语言：javascript复制
vim ~/.bashrc

贴下我的内容：

代码语言：javascript复制
export PATH="~/anaconda3/bin:$PATH"

export JAVA_HOME=/opt/jdk1.8.0_281/
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

export HADOOP_HOME=/opt/hadoop-3.3.0/
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

export PATH=$PATH:/home/dnn/pycharm-2020.2.3/bin
export PATH=${PATH}:${HADOOP_HOME}/bin
export PATH=${PATH}:${JAVA_HOME}/bin
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
#export HADOOP_ROOT_LOGGER=DEBUG,console
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native
代码语言：javascript复制
source ~/.bashrc
  • cd /opt/hadoop-3.3.0/，进入hadoop解压目录
  • 接下来按照 官网的操作 Pseudo-Distributed Operation（伪分布式）

5. 报错处理参考

  • ERROR: Attempting to operate on hdfs namenode as root， 上面已经在bashrc添加过了 https://blog.csdn.net/weixin_49736959/article/details/108897129
  • exitCode=255，注意 hostname 不能有下划线，hostnamectl set-hostname ***，reboot重启 https://blog.csdn.net/wshl1234567/article/details/98346583 https://blog.csdn.net/ntuxiaolei/article/details/81130866
  • export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native，上面已经在bashrc添加过了 https://blog.csdn.net/qq_41541801/article/details/82413776
  • bin/hadoop fs -mkdir /user/root，bin/hadoop fs -mkdir /user/root/input， https://blog.csdn.net/itworkermk/article/details/52352909 在以下这步之前运行上面命令
https 网络安全 java node.js hadoop

0 人点赞