文章目录
- hadoop集群
- 下载
- 环境配置
- 集群配置
- 测试
hadoop集群
参考使用docker部署hadoop集群-手把手复现
下载
- 首先查看hadoop版本
hadoop version
- 下载spark http://spark.apache.org/downloads.html
cd /usr/local
#yum -y install wget
wget https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz
tar -zxvf spark-3.1.1-bin-hadoop3.2.tgz
mv spark-3.1.1-bin-hadoop3.2 spark
环境配置
代码语言：javascript复制vi /etc/profile
export SPARK_HOME=/usr/local/spark
export PATH=...:$SPARK_HOME/bin:$SPARK_HOME/sbin
source /etc/profile
spark-env.sh
代码语言：javascript复制cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.shexport JAVA_HOME=/usr/local/jdk1.8
export SPARK_MASTER_IP=172.18.0.2
export SPARK_WORKER_MEMORY=2g
export SPARK_WORKER_CORES=4
export SPARK_EXECUTOR_MEMORY=1g
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/jdk1.8/jre/lib/aarch64（
插播反爬信息）博主CSDN地址：https://wzlodq.blog.csdn.net/
集群配置
代码语言：javascript复制cd /usr/local/spark/conf
cp workers.template workers
vi workers
#添加从节点：
hadoop2
hadoop3
将主节点配置同步给从节点
代码语言：javascript复制scp -r /usr/local/spark/ hadoop2:/usr/local/
scp -r /usr/local/spark/ hadoop3:/usr/local/测试
代码语言：javascript复制#记得先启动hadoop集群
#/usr/local/hadoop/sbin/./start-all.sh
cd /usr/local/spark/sbin #切到spark目录/sbin下
./start-all.sh
cd /usr/local/spark/bin #如果内存不够可以关闭一个从节点再试试
./run-example SparkPi 2>&1 | grep "Pi is"
cd /usr/local/spark/bin
./spark-shell
开启防火墙端口8080并映射转发到docker容器
代码语言：javascript复制#exit #退出docker容器
systemctl start firewalld
firewall-cmd --add-port=8080/tcp --permanent #开启8080端口
firewall-cmd --add-forward-port=port=8080:proto=tcp:toaddr=172.18.0.2:toport=8080 --permanent
firewall-cmd --add-masquerade --permanent #端口映射
firewall-cmd --reload #重新加载
firewall-cmd --list-ports
# firewall-cmd --zone=public --remove-port=80/tcp --permanent 关闭8080端口公网ip:8080
