1 Cloudera 文档总地址
https://www.cloudera.com/documentation/enterprise/6/6.0.html
2 Cloudera Manager 和CDH下载地址
Cloudera Manager下载地址: https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
下载上线的rpm,然后把这些文件放到/root目录下
CDH下载地址:
代码语言:javascript复制https://archive.cloudera.com/cdh6/6.0.1/parcels/,需要下载以下文件:
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256
manifest.json
1
2
3
4
以上三个文件放到/opt/cloudera/parcel-repo/目录下,如果没有此目录,可以自己手动创建。
此外在下面的官网介绍中: https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cm_6_version_download.html
里面有 Cloudera Manager Release Notes Cloudera Installation Guide Gloudera Enterprise Upgrade Guide
代码语言:javascript复制Repositories
Type Location (baseurl) Repo File
RHEL 7 Compatible https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/ cloudera-manager.repo
1
2
3
3 阅读Introduction中的CDH
阅读以下章节
4 安装CDH 6.0.1集群
以下安装过程分别参考:http://www.cnblogs.com/piperck/p/9944469.html 和 官网
4.1 软件环境
操作系统
代码语言:javascript复制[root@hadoop15 ~]# uname -a
Linux hadoop15 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@hadoop15 ~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)
- 1
- 2
- 3
- 4
- 5
机器配置 要注意的是,不同角色的机器,内存可以配置不一样,一般做Master节点的机器所需要的内存要多一些
代码语言:javascript复制IP 内存 操作系统 角色
192.168.106.155 12G Centos7.4 Server agent
192.168.106.156 9G Centos7.4 Agent
192.168.106.157 8G Centos7.4 Agent
192.168.106.151 1.5G Centos7.4 mysql
1
2
3
4
5
4.2 安装CDH 的准备工作
4.2.1 安装wget
代码语言:javascript复制yum -y install wget
1
4.2.2 更新安装vim,安装vim增强包
代码语言:javascript复制yum -y install vim-enhanced
1
4.2.3 配置host和hostname
官网介绍:
配置域名相关,因为只有三台机器组集群,所以直接使用了hosts的方法,参考官网地址可以是:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html 1.配置在集群中的每个节点的host
代码语言:javascript复制A:105机器
[root@hadoop15 ~]# hostnamectl set-hostname cdh1
B:106机器
[root@localhost ~]# hostnamectl set-hostname cdh2
C:107机器
[root@localhost ~]# hostnamectl set-hostname cdh3
1
2
3
4
5
6
检查hostname
代码语言:javascript复制[root@hadoop15 ~]# hostname
cdh1
[root@localhost ~]# hostname cdh2
[root@localhost ~]# hostname cdh2
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
2、编辑/etc/hosts 参考网址:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html,三台机器配置成下面相同的配置:
代码语言:javascript复制[root@localhost etc]# cat /etc/hosts
192.168.106.155 cdh1
192.168.106.156 cdh2
192.168.106.157 cdh3
1
2
3
4
3、编辑/etc/sysconfig/network 参考地址:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html
代码语言:javascript复制[root@localhost ~]# vim /etc/sysconfig/network
[root@localhost ~]# cat /etc/sysconfig/network
HOSTNAME=cdh1
[root@localhost ~]# vim /etc/sysconfig/network [root@localhost ~]# cat /etc/sysconfig/network HOSTNAME=cdh2
[root@localhost ~]# vim /etc/sysconfig/network [root@localhost ~]# cat /etc/sysconfig/network HOSTNAME=cdh3
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
验证配置 uname –a 需要和 hostname 得到一致的域名 修改前:
修改后:
4.2.4 设置防火墙
参考文章:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cdh_disable_iptables.html
1.保存存在的iptables规则设置(略) 2.禁用iptables (centos7发现没有),替代的是firewalled
代码语言:javascript复制停止: systemctl disable firewalld
禁用: systemctl stop firewalld
1
2
4.2.5 设置SELinux
代码语言:javascript复制vim /etc/sysconfig/selinux
SELINUX=enforcing 改为 SELINUX=disabled
1
2
或使用如下配置:
1.检查SELinux的状态
代码语言:javascript复制[root@localhost etc]# getenforce
Disabled
[root@localhost etc]#
1
2
3
2.如果输出是Permissive或者Disabled,你可以跳过此任务,然后开始下面的操作 3.打开/etc/selinux/config文件,在一些系统中,是/etc/sysconfig/selinux 文件 4.修改文件中的SELINUX=enforcing为SELINUX=permissive 笔者配置如下:
5.保存和关闭这个文件 6.重启系统或者使用下面的命令立即停止SELinux
代码语言:javascript复制$ setenforce 0
1
当你已经安装部署了CDH之后,你可以在/etc/selinux/config(或/etc/sysconfig/selinux 文件)通过改变SELINUX=permissive为SELINUX=enforcing重新启用SELINUX,接着运行下面的命令快速切换到enforcing模式
代码语言:javascript复制setenforce 1
1
4.2.6 启用NTP
文档地址:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cdh_enable_ntp.html
CDH需要你在集群中的每个机器中配置Network Time Protocal(NTP)服务,在开机的时候自动重启NTP。在你的集群中执行下面的每个步骤。 1. 安装NTP 兼容RHEL的命令:
代码语言:javascript复制yum install ntp
1
2. 配置NTP 官网此处的配置,笔者没有配置
A:配置NTP
代码语言:javascript复制vim /etc/ntp.conf
1
master配置:(选用复旦大学ntp服务器)
slave配置:(同步master)
B:开启NTP服务
代码语言:javascript复制service ntpd start
chkconfig ntpd on
1
2
C:查看同步效果 命令:ntpstat
4.2.7 安装Hue所需的Python2.7
1.安装Software Collections Library (3个节点都执行以下命令)
代码语言:javascript复制yum install centos-release-scl
1
2.安装Software Collections utilities (3个节点都执行以下命令)
代码语言:javascript复制yum install scl-utils
1
3.安装Python 2.7 (3个节点都执行以下命令)
代码语言:javascript复制yum install python27
1
4.验证python2.7已经被安装了
代码语言:javascript复制[root@cdh3 ~]# source /opt/rh/python27/enable
[root@cdh3 ~]# python --version
Python 2.7.13
1
2
3
4.2.8 3台机器间设置SSH免密
在cdh1,cdh2,cdh3下执行
代码语言:javascript复制ssh-keygen -t rsa (这个过程一直打enter)
ssh-copy-id cdh1
ssh-copy-id cdh2
ssh-copy-id cdh3
1
2
3
4
执行完成之后,分别执行
代码语言:javascript复制ssh cdh1
ssh cdh2
ssh cdh3
1
2
3
如果不需要再进行密码设置,说明免密设置已经完成了
4.3 安装Cloudera Manager和CDH
4.3.1 安装JDK
关于JDK要求:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/cdh_ig_jdk_installation.html
这里使用oracle-j2sdk1.8,此步骤略,按照官网进行安装
Jdk要求: A:JDK必须是64位的,不要使用32位的 B:每个集群节点都要安装 C:JDK必须安装在/usr/java/jdk-version.
代码语言:javascript复制yum install oracle-j2sdk1.8
1
如果下载了这个jdk,直接使用:
代码语言:javascript复制rpm -ivh oracle-j2sdk1.8-1.8.0 update141-1.x86_64.rpm
1
4.3.2 为Clouder Manager配置一个Repository (如果已经下载了Cloudera Manager,此步骤可以忽略)
代码语言:javascript复制参考文章:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_cm_repo.html
http://www.cnblogs.com/piperck/p/9944469.html
首先Cloudera为用户准备好了专用的程序仓库,需要下载下来 wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
Import the repository signing GPG key
rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
4.3.3 安装Cloudera Manager Package
参考地址:https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cm_server.html
在下面这些步骤中,在Cloudera Manager机器(cdh1)上安装Cloudera Manager Packages,选择启用 自动-TLS
4.3.3.1 Install Cloudera Manager Packages
A:在Cloudera Manager服务器上,键入下面的命令安装Cloudera Manager
代码语言:javascript复制OS 命令
RHEL, CentOS, Oracle Linux : sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
SLES : sudo zypper install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
Ubuntu : sudo apt-get install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
1
2
3
4
B:如果你为Cloudera Manager使用的是Oracle database,在Cloudera Manager服务器上编辑/etc/default/cloudera-scm-server文件。定位到以export CMF_JAVA_OPTS的一行,改变-Xmx2G为-Xmx4G
如果网速不行,可以先下好Cloudera Manager和CDH。下载地址是: Cloudera Manager: https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ CDH下载地址:https://archive.cloudera.com/cdh6/6.0.1/parcels/ 即:
https://archive.cloudera.com/cdh6/6.0.1/parcels/
需要下载的内容是:
代码语言:javascript复制cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
jdk oracle-j2sdk1.8-1.8.0 update141-1.x86_64.rpm
- 1
- 2
- 3
- 4
- 5
- 6
将以上安装文件可以放到/root目录下
关于安装:
代码语言:javascript复制Master节点(cdh1):
yum -y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server
Node节点(cdh2,cdh3): yum -y install cloudera-manager-agent cloudera-manager-daemons
本地安装: (网络好的话,可以忽略此步骤。) cloudera-manager-agent-6.0.1-610811.el7.x8664.rpm cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm cloudera-manager-server-6.0.1-610811.el7.x8664.rpm 先安装 cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm 再安装其他的。
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
先安装下面的内容,因为cloudera-manager-agent依赖这些
代码语言:javascript复制yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions
1
cdh1上执行以下命令:
代码语言:javascript复制rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm
1
2
3
4
cdh2,cdh3上执行以下命令:
代码语言:javascript复制rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
1
2
4.3.3.2 启用Auto-TLS以自动创建证书
代码语言:javascript复制JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services
1
4.3.3.3 安装mysql
参考:https://blog.csdn.net/Sky_WANGJ/article/details/84030458 参考:http://www.cnblogs.com/piperck/p/9944469.html
代码语言:javascript复制wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
yum -y install mysql-community-release-el7-5.noarch.rpm
yum -y update
yum -y install mysql-server
*.Move old InnoDB log files /var/lib/mysql/iblogfile0 and /var/lib/mysql/iblogfile1 out of /var/lib/mysql/ to a backup location. rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1
启动mysql systemctl enable mysqld systemctl start mysqld
初始化Mysql /usr/bin/mysql_secure_installation
[…] Enter current password for root (enter for none): OK, successfully used password, moving on… […] Set root password? [Y/n] Y New password:root Re-enter new password:root Remove anonymous users? [Y/n] Y […] Disallow root login remotely? [Y/n] N […] Remove test database and access to it [Y/n] Y […] Reload privilege tables now? [Y/n] Y All done!
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
4.3.3.4 安装MySQL JDBC驱动程序
代码语言:javascript复制wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
#要注意的是后面的名字必须是mysql-connector-java.jar(可以将下面的jar包放到3台机器上) cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
- 1
- 2
- 3
- 4
- 5
- 6
4.3.3.5 必须配置数据库支持MySQL UTF8字符集编码
代码语言:javascript复制mysql -u root -p root
create database scm default character set utf8 default collate utf8_general_ci; grant all on scm.* to ‘scm’@’%’ identified by ‘root’; create database amon default character set utf8 default collate utf8_general_ci; grant all on amon.* to ‘amon’@’%’ identified by ‘root’; create database rman default character set utf8 default collate utf8_general_ci; grant all on rman.* to ‘rman’@’%’ identified by ‘root’; create database hue default character set utf8 default collate utf8_general_ci; grant all on hue.* to ‘hue’@’%’ identified by ‘root’; create database metastore default character set utf8 default collate utf8_general_ci; grant all on metastore.* to ‘hive’@’%’ identified by ‘root’; create database sentry default character set utf8 default collate utf8_general_ci; grant all on sentry.* to ‘sentry’@’%’ identified by ‘root’; create database nav default character set utf8 default collate utf8_general_ci; grant all on nav.* to ‘nav’@’%’ identified by ‘root’; create database oozie default character set utf8 default collate utf8_general_ci; grant all on oozie.* to ‘oozie’@’%’ identified by ‘root’;
CREATE DATABASE activity_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON activity_monitor.* TO ‘activity_monitor’@’%’ IDENTIFIED BY ‘root’;
flush privileges;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
笔者的如下:
创建完成之后查看数据库:
禁用透明大页面压缩(cdh1,cdh2,cdh3上都执行)
代码语言:javascript复制echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
并将上面的两条命令写入开机自启动 vim /etc/rc.local
- 1
- 2
- 3
- 4
- 5
优化交换分区
代码语言:javascript复制vim /etc/sysctl.conf
vm.swappiness = 10
sysctl -p /etc/sysctl.conf
- 1
- 2
- 3
- 4
- 5
其它
代码语言:javascript复制echo never>/sys/kernel/mm/transparent_hugepage/defrag
echo never>/sys/kernel/mm/transparent_hugepage/enabled
1
2
4.3.3.6 设置Cloudera Manager数据库
代码语言:javascript复制mysql在本地时执行:
rm -rf /etc/cloudera-scm-server/db.mgmt.properties
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
如果不在一台机器上,执行类似如下命令(cdh2为mysql所在位置,cdh1为cloudera manager server所在位置): [root@cdh1 cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scm
[root@cdh1 cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scm Enter SCM password: JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/…/lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! [root@cdh1 cloudera-scm-server]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
上面过程中需要输入scm的密码,在上面的数据库创建语句中scm的密码被设置成了root,所以在此处也同样设置scm的密码为root
启动Cloudera Manager Server
代码语言:javascript复制将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中,如果没有此目录,可以自己创建。(如果是在线安装,那么此步骤可以忽略。)
注意: 最后将CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256,重命名为CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha,这点必须注意,否则系统会重新下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel。
存放了CDH和修改了HASH之后,设置Cloudera Manager数据库,启动Cloudera Manager Server,然后登录Cloudera Manager Admin Console:
启动Cloudera Manager Server: systemctl start cloudera-scm-server
可以查看启动日志 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
看到此条信息,说明启动完成 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
要注意的是,在上传完上面的CDH之后,要在manifest.json 中找到CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel ,然后找到上面的hash值,即:
将2e650f1f1ea020a3efc98a231b85c2df1a50b030 替换掉CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha 中的内容
登录
代码语言:javascript复制在Web浏览器中,转到 http://cdh1:7180/cmf/login,其中域名是运行Cloudera Manager Server的主机的FQDN或IP地址。
如果有启用自动TLS,则会重定向到https: cdh1:7183,并显示安全警告。你可能需要表明你信任该证书,或单击以继续执行Cloudera Manager Server主机。
登录Cloudera Manager Admin Console。默认凭据为: Username: admin Password: admin
- 1
- 2
- 3
- 4
- 5
- 6
- 7
进入之后
点击第一个免费的即可
如果出现下面的错误:
网络上说的解决办法,参考:https://yq.aliyun.com/articles/578871。然后笔者使用了之后并不成功。最后定位到是上面各章节的配置没有配置好导致的,需要验证:网络是否通,各机器之间是否能够ssh免密,hostname,hosts等是否设置的正确,网络防火墙是否按照博文叙述进行配置。(为此,笔者重新虚拟过3遍3个集群,熬了3个到凌晨2,3点的夜晚)。所以在配置过程中一定要细心又细心。
最后:
5 卸载Cloudera Manager
代码语言:javascript复制systemctl stop cloudera-scm-server
systemctl stop cloudera-scm-agent
yum -y remove ‘cloudera-manager-*’
yum clean all
umount cm_processes umount /var/run/cloudera-scm-agent/process
rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera* rm -rf /tmp/.scmpreparenode.lock rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper rm -Rf datadrivepath/dfs datadrivepath/mapred datadrivepath/yarn
systemctl stop mysqld
yum -y remove mysql rm -rf /var/lib/mysql rm -rf /var/log/mysqld.log rm -rf /var/lib/mysql/mysql rm -rf /usr/lib64/mysql rm -rf /usr/share/mysql rm -rf /opt/cloudera
rpm -qa | grep -i mysql 卸载MySQL相关的文件。
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
6 Cloudera Manager重启,停止等涉及到的相关问题
发现如果是通过rpm -ivh *.rpm方式安装的软件,在机器重启之后,CM和CDH自动重启了。
如果发现没有自动重启或为了保险起见,启动Cloudera Manager和cdh的方式如下:
代码语言:javascript复制以下,第一行命令在cdh1上执行。第二行在所有的节点上执行
systemctl stop cloudera-scm-server
systemctl stop cloudera-scm-agent
(cdh1上执行) systemctl start cloudera-scm-server (cdh1,cdh2,cdh3上都执行) systemctl start cloudera-scm-agent
[root@cdh1 init.d]# ps -ef | grep cloudera-scm-server (在CDH上执行) 查看CM的日志: [root@cdh1 init.d]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
查看日志,然后一直等待到其启动完毕,大约要等待3~5分钟左右,然后在浏览器中输入:http://cdh1:7180/cmf/login
原文:https://blog.csdn.net/tototuzuoquan/article/details/85111018