1、一般地,从Windows本地的Eclipse提交程序到yarn,会报如下错误:
代码语言:javascript复制Diagnostics: Exception from container-launch.
Container id: container_1526537597068_0006_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 1
2、原因:
客户端提交任务时候,在YARNRunner.java
中调用了 createApplicationSubmissionContext
方法,该方法负责初始化启动MRAppMaster
的脚本,该脚本的生成依赖于本地系统,而目前由于是在Windows
系统下,所以生成的脚本提交到Linux集群
上运行必然出错。
createApplicationSubmissionContext
3、手工修改YARNRunner
适配Linux系统
,
// Construct necessary information to start the MR AM
ApplicationSubmissionContext appContext = createApplicationSubmissionContext(conf, jobSubmitDir, ts);
4、修改完再次运行debug看到appContext
的值为:
application_id { id: 7 cluster_timestamp: 1526537597068 } application_name: "wc-fat.jar" queue: "default" am_container_spec { localResources { key: "job.jar" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.jar" } size: 73165925 timestamp: 1526810933633 type: PATTERN visibility: APPLICATION pattern: "(?:classes/|lib/).*" } } localResources { key: "jobSubmitDir/job.splitmetainfo" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.splitmetainfo" } size: 15 timestamp: 1526810935006 type: FILE visibility: APPLICATION } } localResources { key: "jobSubmitDir/job.split" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.split" } size: 355 timestamp: 1526810934874 type: FILE visibility: APPLICATION } } localResources { key: "job.xml" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.xml" } size: 93151 timestamp: 1526810944696 type: FILE visibility: APPLICATION } } tokens: "HDTS 00 00 01 25MapReduceShuffleTokenb342214TYX 31303 33" environment { key: "SHELL" value: "/bin/bash" } environment { key: "CLASSPATH" value: "$PWD:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:$PWD/*" } environment { key: "LD_LIBRARY_PATH" value: "$PWD" } command: "$JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr " application_ACLs { accessType: APPACCESS_VIEW_APP acl: " " } application_ACLs { accessType: APPACCESS_MODIFY_APP acl: " " } } cancel_tokens_when_complete: true maxAppAttempts: 2 resource { memory: 1536 virtual_cores: 1 } applicationType: "MAPREDUCE"
【另外】 规划文件提交在hdfs上的佐证:
image.png
image.png