在利用spark sql on hive的过程中,访问Mysql总是报错,其报错的日志总是显示:
15/09/21 11:12:20 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/09/21 11:12:20 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/09/21 11:12:20 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/09/21 11:12:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/09/21 11:12:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/09/21 11:12:21 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
15/09/21 11:12:21 INFO ObjectStore: Initialized ObjectStore
15/09/21 11:12:21 INFO HiveMetaStore: Added admin role in metastore
15/09/21 11:12:21 INFO HiveMetaStore: Added public role in metastore
15/09/21 11:12:21 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/09/21 11:12:21 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO Driver: Concurrency mode is disabled, not creating a lock manager
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING)
15/09/21 11:12:21 INFO ParseDriver: Parse Completed
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO SemanticAnalyzer: Starting Semantic Analysis
15/09/21 11:12:21 INFO SemanticAnalyzer: Creating table src position=27
15/09/21 11:12:21 INFO HiveMetaStore: 0: get_table : db=default tbl=src
15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=get_table : db=default tbl=src
15/09/21 11:12:21 INFO HiveMetaStore: 0: get_database: default
15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=get_database: default
15/09/21 11:12:21 INFO Driver: Semantic Analysis Completed
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO Driver: Starting command: CREATE TABLE IF NOT EXISTS src (key INT, value STRING)
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO PerfLogger:
15/09/21 11:12:21 INFO DDLTask: Default to LazySimpleSerDe for table src
15/09/21 11:12:21 INFO HiveMetaStore: 0: create_table: Table(tableName:src, dbName:default, owner:ndscbigdata, createTime:1442805141, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:key, type:int, comment:null), FieldSchema(name:value, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputF ormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMa ps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
15/09/21 11:12:21 INFO audit: ugi=ndscbigdata ip=unknown-ip-addr cmd=create_table: Table(tableName:src, dbName:default, owner:ndscbigdata, createTime:1442805141, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:key, type:int, comment:null), FieldSchema(name:value, type:string, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputF ormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMa ps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
15/09/21 11:12:21 ERROR RetryingHMSHandler: MetaException(message:file:/user/hive/warehouse/src is not a directory or unable to create one)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1239)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1294)
at sun.reflect.NativeMethodAccessorImpl .invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl .invoke(NativeMethodAccessorImpl .java:62)
at sun.reflect.DelegatingMethodAccessor Impl.invoke(DelegatingMethodAccessor Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy21.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:547)
at sun.reflect.NativeMethodAccessorImpl .invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl .invoke(NativeMethodAccessorImpl .java:62)
at sun.reflect.DelegatingMethodAccessor Impl.invoke(DelegatingMethodAccessor Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy22.createTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:613)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:329)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:310)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:139)
at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:310)
at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:300)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:472)
at org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:939)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:939)
at org.apache.spark.sql.DataFrame.(DataFrame.scala:144)
at org.apache.spark.sql.DataFrame.(DataFrame.scala:128)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:744)
at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:50)
at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
由于网上对这一块的东西介绍得总是很少,按照操作也总是无解,于是自己想着先把HIVE源码编译一下,花了半天时间,终于搞定。
编译环境:Eclipse