编译Ambari
第一步：Ambari编译前环境准备
参考网址：https://cwiki.apache.org/confluence/display/AMBARI/Ambari Development
1. JDK
1.1 卸载原有JDK
[root@node~]# rpm -qa|grep jdk java-1.6.0-openjdk-devel-1.6.0.35-1.13.7.1.el6_6.x86_64 java-1.6.0-openjdk-1.6.0.35-1.13.7.1.el6_6.x86_64 [root@node~]# yum remove -y java-1.6.0-openjdk [root@node~]# rpm -qa|grep java tzdata-java-2015e-1.el6.noarch java_cup-0.10k-5.el6.x86_64 java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 [root@node~]# yum remove -y java
1.2 安装Oracle官方jdk7
[root@node~]# scp jdk-7u67-linux-x64.gz 172.16.54.166:/opt
1.3 配置环境JDK变量
2.安装Maven
2.1 下载http://maven.apache.org/
[root@node~]# tar -zxvf apache-maven-3.0.5-bin.tar.gz
2.2 配置环境变量
[root@node apache-maven-3.0.5]# vi /etc/profile export JAVA_HOME=/opt/jdk1.7.0_67 export MAVEN_HOME=/opt/apache-maven-3.0.5 export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true" [root@node apache-maven-3.0.5]# source /etc/profile
2.3 测试Maven
[root@node~ apache-maven-3.0.5]# mvn -v Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28 0800) Maven home: /opt/apache-maven-3.0.5 Java version: 1.7.0_67, vendor: Oracle Corporation Java home: /opt/jdk1.7.0_67/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-573.el6.x86_64", arch: "amd64", family: "unix"
2.4 更换国内源
[root@node apache-maven-3.0.5]# cd conf [root@node conf]# vi settings.xml <!-- 阿里云仓库 --> <mirror> <id>alimaven</id> <mirrorOf>central</mirrorOf> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/repositories/central/</url> </mirror>
3.Python
3.1 检测python版本
[root@cos6 opt]# python Python 2.6.6 (r266:84292, Jul 23 2015, 15:22:56) [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> exit();
3.2 setuptools
[root@east81 Downloads]# scp setuptools-0.6c11-py2.7.egg 172.16.54.166:/opt [root@vnode1 opt]# sh setuptools-0.6c11-py2.7.egg Processing setuptools-0.6c11-py2.7.egg Removing /usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg Copying setuptools-0.6c11-py2.7.egg to /usr/lib/python2.7/site-packages setuptools 0.6c11 is already the active version in easy-install.pth Installing easy_install script to /usr/bin Installing easy_install-2.7 script to /usr/bin Installed /usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg Processing dependencies for setuptools==0.6c11 Finished processing dependencies for setuptools==0.6c11
4.安装rpmbuild
4.1检测rpmbuild
[root@vnode1 opt]# rpm -qa | grep rpm-build
4.2安装rpmbuild
[root@vnode1 opt]# yum install -y rpm-build
5.安装gcc-c
[root@vnode1 app]# yum install -y gcc-c [root@vnode0 ~]# yum install -y gcc make gcc-c openssl-devel wget
6.安装nodejs
6.1 下载nodejs
https://nodejs.org/download/release/v0.10.44/node-v0.10.44-linux-x64.tar.gz [root@cos6 opt]# tar -zxvf node-v0.10.44-linux-x64.tar.gz
6.2 配置nodejs环境变量
[root@cos6 opt]# cd node-v0.10.44-linux-x64 [root@cos6 node-v0.10.44-linux-x64]# ls bin ChangeLog include lib LICENSE README.md share [root@cos6 node-v0.10.44-linux-x64]# pwd /opt/node-v0.10.44-linux-x64 [root@cos6 node-v0.10.44-linux-x64]# vi /etc/profile #nodejs export NODE_HOME=/opt/node-v0.10.44-linux-x64 export PATH=$PATH:$NODE_HOME/bin 注意NODE_HOME前不要少了$ [root@cos6 node-v0.10.44-linux-x64]# source /etc/profile
6.3 检测nodejs命令
[root@vnode1 ~]# node -v v0.10.44 [root@vnode1 ~]# npm -v 2.15.0
7 安装brunch
7.1 查找node根目录
[root@vnode1 opt]# npm root -g /opt/node-v0.10.44-linux-x64/lib/node_modules
[root@vnode1 opt]# cd /opt/node-v0.10.44-linux-x64/lib/node_modules
7.2 更改taobao源
[root@vnode1 node_modules]# npm config set registry https://registry.npm.taobao.org
[root@vnode1 node_modules]# npm info underscore
7.3 安装1.7.20版本的brunch
[root@vnode1 node_modules]# npm install -g brunch@1.7.20 npm WARN deprecated minimatch@1.0.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue npm WARN optional dep failed, continuing fsevents@0.3.8 npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree. /opt/node-v0.10.44-linux-x64/bin/brunch -> /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch/bin/brunch > brunch@1.7.20 postinstall /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch > node setup.js postinstall brunch@1.7.20 /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch ├── debug@0.7.4 ├── async-waterfall@0.1.5 ├── commonjs-require-definition@0.1.2 ├── async-each@0.1.6 ├── commander@2.0.0 ├── ncp@0.4.2 ├── mkdirp@0.3.5 ├── init-skeleton@0.2.4 (rimraf@2.2.8) ├── anymatch@1.0.0 (minimatch@1.0.0) ├── read-components@0.6.1 (component-builder@0.10.1) ├── source-map@0.1.43 (amdefine@1.0.1) ├── coffee-script@1.8.0 ├── chokidar@0.12.6 (readdirp@1.3.0) ├── anysort@1.0.1 (anymatch@1.3.0) ├── pushserve@0.1.6 (connect-slashes@0.0.11, express@3.3.8) └── loggy@0.2.2 (growl@1.8.1, ansi-color@0.2.1, date-utils@1.2.21) [root@vnode1 node_modules]#
7.4 可能存在的问题
安装失败时，卸载brunch [root@cos6 conf]# npm remove -g brunch [root@vnode1 apache-ambari-2.4.2-src]# npm install node-gyp node-gyp rebuilt卡住 > contextify@0.1.15 install /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/contextify > node-gyp rebuild
8 海量文件设置
可能报错：too many files are opened
[root@cos6 node_modules]# echo fs.inotify.max_user_watches=524288 |tee -a /etc/sysctl.conf && sysctl -p
fs.inotify.max_user_watches=524288 net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 fs.inotify.max_user_watches = 524288 将每个进程可以打开的文件数目加大到10000，缺省为1024 [root@vnode1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimite pending signals (-i) 7217 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimite open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7217 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@vnode1 ~]# ulimit -n 10000
9 安装git
[root@cos6 ~]# git –version -bash: git: command not found [root@cos6 setuptools-32.1.2]# cd
[root@cos6 ~]# yum install -y git
10 安装ant
[root@vnode1 apache-ambari-2.4.2-src]# yum install -y ant
第二步：编译Ambari源码
1、下载ambari源码
wget http://www.apache.org/dist/mbari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz [root@vnode1 opt]# tar -zxvf apache-ambari-2.4.2-src.tar.gz
2、给Ambari打上版本号
[root@vnode1 apache-ambari-2.4.2-src]# mvn versions:set -DnewVersion=2.4.2.0 [root@vnode1 apache-ambari-2.4.2-src]# pushd ambari-metrics /opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src 注意，返回上一级目录 [root@vnode1 ambari-metrics]# cd .. [root@vnode1 apache-ambari-2.4.2-src]# mvn versions:set -DnewVersion=2.4.2.0 [root@vnode1 apache-ambari-2.4.2-src]# popd /opt/apache-ambari-2.4.2-src
3、开始编译
[root@vnode1 apache-ambari-2.4.2-src]# mvn -B clean install package rpm:rpm -DnewVersion=2.4.2.0 -DskipTests -Dpython.ver="python >= 2.6"
.................
.................
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [4.304s] [INFO] Apache Ambari Project POM ......................... SUCCESS [0.149s] [INFO] Ambari Web ........................................ SUCCESS [1:21.863s] [INFO] Ambari Views ...................................... SUCCESS [1.396s] [INFO] Ambari Admin View ................................. SUCCESS [6:00.715s] [INFO] ambari-metrics .................................... SUCCESS [0.454s] [INFO] Ambari Metrics Common ............................. SUCCESS [1.182s] [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [1.961s] [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [1.042s] [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.705s] [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [2.377s] [INFO] Ambari Metrics Storm Sink (Legacy) ................ SUCCESS [1.887s] [INFO] Ambari Metrics Collector .......................... SUCCESS [1:12:45.829s] [INFO] Ambari Metrics Monitor ............................ SUCCESS [2.403s] [INFO] Ambari Metrics Grafana ............................ SUCCESS [28.278s] [INFO] Ambari Metrics Assembly ........................... SUCCESS [35:59.608s] [INFO] Ambari Server ..................................... SUCCESS [4:29.713s] [INFO] Ambari Functional Tests ........................... SUCCESS [15.422s] [INFO] Ambari Agent ...................................... SUCCESS [52.274s] [INFO] Ambari Client ..................................... SUCCESS [0.175s] [INFO] Ambari Python Client .............................. SUCCESS [1.595s] [INFO] Ambari Groovy Client .............................. SUCCESS [17.700s] [INFO] Ambari Shell ...................................... SUCCESS [0.233s] [INFO] Ambari Python Shell ............................... SUCCESS [1.408s] [INFO] Ambari Groovy Shell ............................... SUCCESS [15.462s] [INFO] ambari-logsearch .................................. SUCCESS [0.218s] [INFO] Ambari Logsearch Appender ......................... SUCCESS [0.431s] [INFO] Ambari Logsearch Solr Client ...................... SUCCESS [1.946s] [INFO] Ambari Logsearch Portal ........................... SUCCESS [2:14.732s] [INFO] Ambari Logsearch Log Feeder ....................... SUCCESS [8.750s] [INFO] Ambari Logsearch Assembly ......................... SUCCESS [0.328s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:05:35.686s [INFO] Finished at: Fri Dec 23 07:38:57 EST 2016 [INFO] Final Memory: 365M/1295M [INFO] ------------------------------------------------------------------------ [root@vnode1 apache-ambari-2.4.2-src]#
4、编译成功后，安装rpm包
Ambari安装需要ambari-server安装到管理机上，ambari-agent安装到集群节点上。 Ambari-server的rpm包 ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm /opt/apache-ambari-2.4.2-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-2.4.2.0-0.x86_64.rpm Ambari-agent的rpm包 ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm /opt/apache-ambari-2.4.2-src/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.4.2.0-0.x86_64.rpm
第三、存在问题
1、NodeJS版本不对 [root@vnode1 opt]# node -v v6.9.1 可能通过rpm包安装过高版本的node 尝试卸载： [root@vnode1 opt]# yum remove -y nodejs Loaded plugins: fastestmirror Resolving Dependencies --> Running transaction check ---> Package nodejs.x86_64 1:6.9.1-4.el7 will be erased --> Processing Dependency: nodejs = 1:6.9.1-4.el7 for package: 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 --> Running transaction check ---> Package npm.x86_64 1:3.10.8-1.6.9.1.4.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================= Removing: nodejs x86_64 1:6.9.1-4.el7 @epel 16 M Removing for dependencies: npm x86_64 1:3.10.8-1.6.9.1.4.el7 @epel 9.7 M Transaction Summary ================================================================================================================================================= Remove 1 Package ( 1 Dependent package) Installed size: 26 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Erasing : 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 1/2 Erasing : 1:nodejs-6.9.1-4.el7.x86_64 2/2 Verifying : 1:nodejs-6.9.1-4.el7.x86_64 1/2 Verifying : 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 2/2 Removed: nodejs.x86_64 1:6.9.1-4.el7 Dependency Removed: npm.x86_64 1:3.10.8-1.6.9.1.4.el7 Complete! 看来之前真的安装过高版本的node 再次解压缩，配置环境变量 [root@vnode1 opt]# source /etc/profile [root@vnode1 opt]# node -v v0.10.44 2、下载phantomjs-1.9.8-linux-x86_64.tar.bz2卡住 > phantomjs@1.9.20 install /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs > node install.js PhantomJS not found on PATH Downloading https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2 Saving to /opt/apache-ambari-2.2.2-src/ambari-web/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2 Receiving... 单独下载https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2 [root@east81 Public]# scp phantomjs-1.9.8-linux-x86_64.tar.bz2 172.16.54.131:/opt [root@vnode1 opt]# mkdir -p /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs/ [root@vnode1 opt]# cp phantomjs-1.9.8-linux-x86_64.tar.bz2 /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2 再次确认该文件已经复制过去： [root@vnode1 opt]# ls /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs phantomjs-1.9.8-linux-x86_64.tar.bz2 //npm install phantomjs -g 去掉clean，继续编译： [root@vnode1 apache-ambari-2.4.2-src]# mvn -B install package rpm:rpm -DnewVersion=2.4.2.0 -DskipTests -Dpython.ver="python >= 2.6" -X 3、下载node-v0.10.44-linux-x64.tar.gz出错 [INFO] Downloading Node.js from http://nodejs.org/dist/v0.10.44/node-v0.10.44-linux-x64.tar.gz to /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz [INFO] No proxy was configured, downloading directly [INFO] Extracting Node.js files in node_tmp [INFO] Unpacking /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz into /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [4:04.773s] [INFO] Apache Ambari Project POM ......................... SUCCESS [7.184s] [INFO] Ambari Web ........................................ SUCCESS [2:33.179s] [INFO] Ambari Views ...................................... SUCCESS [4:35.686s] [INFO] Ambari Admin View ................................. FAILURE [9:32.324s] [INFO] ambari-metrics .................................... SKIPPED [INFO] Ambari Metrics Common ............................. SKIPPED [INFO] Ambari Metrics Hadoop Sink ........................ SKIPPED [INFO] Ambari Metrics Flume Sink ......................... SKIPPED [INFO] Ambari Metrics Kafka Sink ......................... SKIPPED [INFO] Ambari Metrics Storm Sink ......................... SKIPPED [INFO] Ambari Metrics Collector .......................... SKIPPED [INFO] Ambari Metrics Monitor ............................ SKIPPED [INFO] Ambari Metrics Grafana ............................ SKIPPED [INFO] Ambari Metrics Assembly ........................... SKIPPED [INFO] Ambari Server ..................................... SKIPPED [INFO] Ambari Agent ...................................... SKIPPED [INFO] Ambari Client ..................................... SKIPPED [INFO] Ambari Python Client .............................. SKIPPED [INFO] Ambari Groovy Client .............................. SKIPPED [INFO] Ambari Shell ...................................... SKIPPED [INFO] Ambari Python Shell ............................... SKIPPED [INFO] Ambari Groovy Shell ............................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:54.669s [INFO] Finished at: Wed Dec 21 14:20:25 CST 2016 [INFO] Final Memory: 24M/61M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) on project ambari-admin: Could not extract the Node archive: Could not extract archive: '/opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz': EOFException -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :ambari-admin [root@cos6 apache-ambari-2.2.2-src]# 确认一下 [root@cos6 apache-ambari-2.2.2-src]# tar -zxvf /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz node-v0.10.44-linux-x64/ node-v0.10.44-linux-x64/bin/ node-v0.10.44-linux-x64/bin/npm node-v0.10.44-linux-x64/bin/node gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now 重新下载，通过迅雷下载http://nodejs.org/dist/v0.10.44/node-v0.10.44-linux-x64.tar.gz 然后复制到/opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz （node-v0.10.44-linux-x64.tar.gz更名为node.tar.gz） [root@cos6 opt]# cp node-v0.10.44-linux-x64.tar.gz /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz [root@cos6 opt]# tar -zxvf /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz
