hadoop+hbase+zookeeper+hive

hadoop、hbase、zookeeper、hive是大数据里面四个重要的产品，关于这四个产品的介绍，各种书籍和网站介绍得很多，在这里不再赘述，仅介绍在ubunt下如何安装和配置这三种产品。

第一步：配置虚拟机为桥接模式

第二步：安装openssh-server，并且登录不需要

/home/jerry/>apt-get install openssh-server #安装openssh-server

/home/jerry/>ssh localhost #登录本机，需要密码

/home/jerry/>exit# 退出刚才的 ssh localhost

/home/jerry/>cd ~/.ssh/# 若没有该目录，请先执行一次ssh localhost

/home/jerry/>ssh-keygen -t rsa# 会有提示，都按回车就可以

/home/jerry/>cat ./id_rsa.pub >> ./authorized_keys # 加入授权

/home/jerry/>ssh localhost#不需要密码

第三步：安装JDK 1.7以上

/home/jerry/>tar -xzf jdk-8u161-linux-x64.tar.gz -C /usr/local

/home/jerry/>cd /usr/local

/usr/local/>mv jdk1.8.0_161/ java

/usr/local/>gedit ~/.bashrc

代码语言：javascript复制

export JAVA_HOME=/usr/local/java
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

/usr/local/>source ~/.bashrc

/usr/local/>java -version

出现版本信息，说明配置正确。

第四步：安装hadoop

/home/jerry/>tar -xzf hadoop-2.6.4.tar.gz -C /usr/local

/home/jerry/>cd /usr/local

/usr/local/>mv hadoop-2.6.4/ hadoop

/usr/local/>chown -R root ./hadoop

/usr/local/>gedit ~/.bashrc

代码语言：javascript复制

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export JAVA_LIBRAY_PATH=$HADOOP_HOME/lib/native
export LD_LIBRARY_PATH=$JAVA_LIBRARY_PATH

/usr/local/>source ~/.bashrc

/usr/local/>hadoop version

#查看hsddop版本有内容，说明配置正确

/usr/local/>gedit /usr/local/hadoop/etc/hadoop/core-site.xml

代码语言：javascript复制

<configuration>
　　<property>
　　　　<name>fs.defualtFS</name>
　　　　<value>hdfs://localhost:9000</value>
　　</property>
　　<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
</configuration>

/usr/local/>gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

代码语言：javascript复制

<configuration>
　　    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
　　<property>
　　　　<name>dfs.namenode.http-address</name>
　　　　<value>0.0.0.0:50070</value>
　　</property>
</configuration>

dfs.namenode.name.dir：namenode位置
dfs.namenode.data.dir：datanode位置
dfs.namenode.http-address：hadoop WEB端口

gedit /usr/local/hadoop/etc/hadoop/hdoop-env.sh

将语句

代码语言：javascript复制

export JAVA_HOME=$JAVA_HOME

修改为

代码语言：javascript复制

export JAVA_HOME=/usr/local/java

hadoop WEB 页面：http://localhost:50070

/usr/local/>cd hadoop

/usr/local/hadoop/>hdfs namenode -format

/usr/local/hadoop/>sbin/start-dfs.sh

/usr/local/hadoop/>cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml

/usr/local/hadoop/>gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

代码语言：javascript复制

<configuration>
　　<property>
　　　　<name>yarn.nodemanager.aux-server</name>
　　　　<value>mapreduce_shuffle</value>
　　</property>
</configuration>

/usr/local/hadoop/>sbin/start-dfs.sh(./sbin/stop-dfs.sh)

/usr/local/hadoop/>sbin/start-yarn.sh(./sbin/stop-yarn.sh)

/usr/local/hadoop/>sbin/mr-jobhistory-daemon.sh start historyserver #查看历史任务( ./sbin/mr-jobhistory-daemon.sh stop historyserver)

hadoop安全模式，比如hadoop非正常退出，会进入安全模式

$hdfs dfsadmin -safemode get|wait|entry|leave|wait

get：获得当前是否进入安全模式
wait：等待进入安全模式
entry：强制进入安全模式
leave：强制离开安全模式

第五步：安装HBase

/home/jerry/>tar -xzf hbase-2.2.4-bin.tar.gz -C /usr/local

/home/jerry/>cd /usr/local

/usr/local/>mv hbase-2.2.4/ hbase

/usr/local/>gedit ~/.bashrc

代码语言：javascript复制

export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:$HBASE_HOME/bin

/usr/local/>source ~/.bashrc

/usr/local/>chown -R root ./habase

/usr/local/>gedit hbase/conf/hbase-site.xml

代码语言：javascript复制

<configuration>
    <property>
　　　　<name>hbase.rootdir</name>
　　　　<value>hdfs://localhost:9000/hbase</value>
　　</property>
　　<property>
　　　　<name>hbase.cluster.distributed</name>
　　　　<value>true</value>
　　</property>
   <property> 
      <name>hbase.unsafe.stream.capability.enforce</name> 
      <value>false</value> 
   </property>
  <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/usr/local/zookeeper</value>
    </property>
</configuration>

hbase.rootdir： hbase在HDFS的位置，端口号一定要与/usr/local/hadoop/etc/hadoop/core-site.xml保持一致。
hbase.cluster.distributed：是否为分布式，在这里为伪分布式
hbase.unsafe.stream.capability.enforce：在分布式情况下, 一定设置为false（hbase的一个bug）
hbase.zookeeper.property.dataDir： zookeeper的位置。

/usr/local/>hbase version #有结果说明配置正确

/usr/local/>hbase-daemon.sh start maste

/usr/local/>start-hbase.sh #用于启动HBase，关闭HBase用stop-hbase.sh 速度相当慢，./bin/hdfs namenode -format，但是数据全部丢失

/usr/local/>hbase shell#用于打开shell命令行模式

HBase WEB 页面http://localhost:16010

第六步：安装zookeeperz文件

/home/jerry/>tar -zxvf zookeeper-3.4.11.tar.gz

/home/jerry/>mv zookeeper-3.4.11 /usr/local/zookeepe

/home/jerry/>cd /usr/local/zookeeper/conf //切换到该目录

/usr/local/zookeeper/conf >cp zoo_sample.cfg zoo.cfg//复制一份取名为zoo.cfg

/usr/local/zookeeper/conf >gedit zoo.cfg

代码语言：javascript复制

...
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/tmp/zookeeper
dataLogDir=/tmp/zookeeper/logs
# the port at which the clients will connect
clientPort=2181
...

确保/tmp/zookeeper，/tmp/zookeeper/logs存在

/usr/local/>source ~/.bashrc

/usr/local/>chown -R root ./habase

/usr/local/>hbase version

/usr/local/zookeeper/conf >gedit ~/.bashrc

代码语言：javascript复制

export ZOOKEEPER_HOME=/usr/local/zookeeper
export PATH=$PATH:$ ZOOKEEPER_HOME/bin

/usr/local/zookeeper/conf >source ~/.bashrc

cd ../bin/

/usr/local/zookeeper/bin>zkServer.sh start

在实际工作中，建议启动服务以后确定启动没有问题再关闭，最后统一启动，启动顺序为：

haddoop
hbase
zookeep

启动完毕

jps

114865 NameNode

112467 JobHistoryServe

116114 HRegionServe

111637 ResourceManage

115271 SecondaryNameNode

43386 Jps

115994 HMaste

115932 HQuorumPee

115023 DataNode

111774 NodeManager

第七步安装hive

1 安装hive

/home/jerry/>tar -xzf apache-hive-2.3.4-bin.tar.gz -C /usr/local

/home/jerry/>cd /usr/local

/usr/local/>mv apache-hive-2.3.4-bin/ hive

/usr/local/>gedit ~/.bashrc

代码语言：javascript复制

export HIVE_HOME=/usr/local/hive
export HCAT_HOME=$HIVE_HOME/hcatalog
export HIVE_CONF=$HIVE_HOME/conf
export PATH=$PATH:$HIVE_HOME/bin

/usr/local/>source ~/.bashrc

/usr/local/hive/>cd hive

/usr/local/hive/>cp conf/hive-default.xml.template conf/hive-site.xml

/usr/local/hive/>gedit conf/hive-site.xml

代码语言：javascript复制

<property>
       <name>javax.jdo.option.ConnectionURL</name>
       <value>jdbc:mysql://localhost:3306/hive?characterEncoding=UTF-8&amp;useSSL=false</value>
    </property>
    <property>
       <name>javax.jdo.option.ConnectionDriverName</name>
       <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
       <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
    </property>
    <property>
       <name>javax.jdo.option.ConnectionPassword</name>
        <value>hive</value>
    </property>
       <property>
      <name>hive.metastore.schema.verification</name>
            <value>false</value>
       </property>

2 安装mysql

/usr/local/hive/>sudo apt-get install mysql-serve

/usr/local/hive/>service mysql start

下载mysql-jdbc包

/usr/local/hive/>tar -xzf mysql-connector-java-5.1.48.tar.gz

/usr/local/hive/>mv /home/jerry/mysql-connector-java-5.1.48/mysql-connector-java-5.1.48-bin.jar lib/

下载disruptor-3.4.2.jar拷贝到/usr/local/hive/lib/中替换原来的disruptor包

/usr/local/hive/>mysql -u root -p

mysql> create user 'hive' identified by 'hive';

mysql> grant all privileges on *.* to 'hive'@'localhost' identified by 'hive';

mysql> exit

/usr/local/hive/>hive --service metastore &

/usr/local/hive/>-uhive -phive

mysql> create database hive;

mysql> exit

/usr/local/hive/>start-dfs.sh

/usr/local/hive/>hive

…

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive>

几种错误现象。

1，NoNode for /hbase/master

hbase(main):005:0> status

ERROR: KeeperErrorCode = NoNode for /hbase/master

For usage try 'help "status"'

Took 8.1764 seconds

原因

HMaster进程没有打开

2，启动hadoopmei没有datanode

cat /usr/local/hadoop/tmp/dfs/name/current/VERSION

记住clusterID

edit /usr/local/hadoop/tmp/dfs/data/current/VERSION

修改clusterID为刚才记录的name里面的clusterID

重新启动hadoop

3，启动hbase后不久HMaster就关闭

如果使用后hbase自带的zookeeper，建议单独安装zookeeper，否则到zookeeper主目录下删除hbase下所有内容

hadoop hive hbase TDSQLMySQL版 zookeeper

0 人点赞