spark on yarn提交任务时一直显示ACCEPTED，过一个小时后就会出现任务失败，但在提交时shell终端显示的日志并没有报错，logs文件夹中也没有日志产生。注：spark on yarn是不需要启动spark集群的，只需要在提交任务的机器配置spark就可以了，因为任务是由hadoop来执行的，spark只负责任务的提交。
任务提交命令为
bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master yarn-client --num-executors 25 --executor-memory 1g --executor-cores 1 lib/spark-examples-1.0.1-hadoop2.2.0.jar hdfs://192.168.1.11:9000/test
spark-env.sh配置如下：
export SPARK_JAR=hdfs://192.168.1.11:9000/test1/spark-assembly-1.0.1-hadoop2.2.0.jar export SPARK_YARN_APP_JAR=hdfs://192.168.1.11:9000/test1/spark-examples-1.0.1-hadoop2.2.0.jar export HADOOP_CONF_DIR=/hadoop/etc/hadoop export SPARK_EXECUTOR_INSTANCES=1000 export SPARK_EXECUTOR_CORES=1 export SPARK_EXECUTOR_MEMORY=10000M export SPARK_DRIVER_MEMORY=20000M export SPARK_YARN_APP_NAME=spark
查看日志发现报错：
14/09/04 17:10:44 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/09/04 17:10:45 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/09/04 17:10:46 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/09/04 17:10:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
这个问题很明显没有找到resourcemanager，我们可以通过以下方法来解决：
1 在spark-env.sh中添加
export SPARK_YARN_USER_ENV="CLASSPATH=hadoop-2.3.0/etc/hadoop"（修改成自己的hadoop路径）
2 自己编写JavaWordCount程序，在程序中配置集群信息
代码语言：javascript复制 SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
sparkConf.set("mapreduce.framework.name", "yarn");
sparkConf.set("mapreduce.jobtracker.address", "192.168.1.10:9001");
sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");
sparkConf.set("yarn.resourcemanager.admin.address", "192.168.1.10:8033");
sparkConf.set("yarn.resourcemanager.address", "192.168.1.10:8032");
sparkConf.set("yarn.resourcemanager.resource-tracker.address", "192.168.1.10:8031");
sparkConf.set("yarn.resourcemanager.scheduler.address", "192.168.1.10:8030");
sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");然后打包，在提交spark任务时使用自己打包的jar包即可。
