Hive on Tez 典型日志内容
代码语言：javascript复制INFO : Compiling command(queryId=hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d):
select count(*) as cnt, new_uv from dm_db. dm_browser_app_retain_d where p_dt = '2019-07-13' GROUP BY new_uv
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:cnt, type:bigint, comment:null), FieldSchema(name:new_uv, type:int, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d); Time taken: 0.54 seconds
INFO : Executing command(queryId=hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d):
select count(*) as cnt, new_uv from dm_db. dm_browser_app_retain_d where p_dt = '2019-07-13' GROUP BY new_uv
INFO : Query ID = hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId: hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d
INFO : Session is already open
INFO : Dag name: select count(*) as cnt, new_uv from...new_uv (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id application_1602484539124_90806)
INFO : Status: DAG finished successfully in 9.90 seconds
INFO :
INFO : Query Execution Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : OPERATION DURATION
INFO : ----------------------------------------------------------------------------------------------
INFO : Compile Query 0.54s
INFO : Prepare Plan 0.22s
INFO : Get Query Coordinator (AM) 0.00s
INFO : Submit Plan 0.27s
INFO : Start DAG 0.55s
INFO : Run DAG 9.90s
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : Task Execution Summary
INFO : ----------------------------------------------------------------------------------------------
INFO : VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDS
INFO : ----------------------------------------------------------------------------------------------
INFO : Map 1 2522.00 7,980 49 31 2
INFO : Reducer 2 4352.00 9,920 99 2 0
INFO : ----------------------------------------------------------------------------------------------
INFO :
INFO : org.apache.tez.common.counters.DAGCounter:
INFO : NUM_SUCCEEDED_TASKS: 3
INFO : TOTAL_LAUNCHED_TASKS: 3
INFO : RACK_LOCAL_TASKS: 1
INFO : AM_CPU_MILLISECONDS: 990
INFO : AM_GC_TIME_MILLIS: 0
INFO : File System Counters:
INFO : FILE_BYTES_READ: 250
INFO : FILE_BYTES_WRITTEN: 138
INFO : HDFS_BYTES_READ: 2780
INFO : HDFS_BYTES_WRITTEN: 212
INFO : HDFS_READ_OPS: 7
INFO : HDFS_WRITE_OPS: 4
INFO : HDFS_OP_CREATE: 2
INFO : HDFS_OP_GET_FILE_STATUS: 6
INFO : HDFS_OP_OPEN: 1
INFO : HDFS_OP_RENAME: 2
INFO : org.apache.tez.common.counters.TaskCounter:
INFO : REDUCE_INPUT_GROUPS: 2
INFO : REDUCE_INPUT_RECORDS: 2
INFO : COMBINE_INPUT_RECORDS: 0
INFO : SPILLED_RECORDS: 4
INFO : NUM_SHUFFLED_INPUTS: 2
INFO : NUM_SKIPPED_INPUTS: 0
INFO : NUM_FAILED_SHUFFLE_INPUTS: 0
INFO : MERGED_MAP_OUTPUTS: 2
INFO : GC_TIME_MILLIS: 148
INFO : TASK_DURATION_MILLIS: 5316
INFO : CPU_MILLISECONDS: 17900
INFO : PHYSICAL_MEMORY_BYTES: 10412359680
INFO : VIRTUAL_MEMORY_BYTES: 25110016000
INFO : COMMITTED_HEAP_BYTES: 10412359680
INFO : INPUT_RECORDS_PROCESSED: 1
INFO : INPUT_SPLIT_LENGTH_BYTES: 1687
INFO : OUTPUT_RECORDS: 2
INFO : OUTPUT_LARGE_RECORDS: 0
INFO : OUTPUT_BYTES: 12
INFO : OUTPUT_BYTES_WITH_OVERHEAD: 28
INFO : OUTPUT_BYTES_PHYSICAL: 56
INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 26
INFO : ADDITIONAL_SPILLS_BYTES_READ: 56
INFO : ADDITIONAL_SPILL_COUNT: 0
INFO : SHUFFLE_CHUNK_COUNT: 1
INFO : SHUFFLE_BYTES: 56
INFO : SHUFFLE_BYTES_DECOMPRESSED: 28
INFO : SHUFFLE_BYTES_TO_MEM: 26
INFO : SHUFFLE_BYTES_TO_DISK: 0
INFO : SHUFFLE_BYTES_DISK_DIRECT: 30
INFO : NUM_MEM_TO_DISK_MERGES: 0
INFO : NUM_DISK_TO_DISK_MERGES: 0
INFO : SHUFFLE_PHASE_TIME: 391
INFO : MERGE_PHASE_TIME: 447
INFO : FIRST_EVENT_RECEIVED: 211
INFO : LAST_EVENT_RECEIVED: 211
INFO : HIVE:
INFO : CREATED_FILES: 2
INFO : DESERIALIZE_ERRORS: 0
INFO : RECORDS_IN_Map_1: 31
INFO : RECORDS_OUT_0: 2
INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 2
INFO : RECORDS_OUT_INTERMEDIATE_Reducer_2: 0
INFO : RECORDS_OUT_OPERATOR_FS_14: 2
INFO : RECORDS_OUT_OPERATOR_GBY_10: 2
INFO : RECORDS_OUT_OPERATOR_GBY_12: 2
INFO : RECORDS_OUT_OPERATOR_MAP_0: 0
INFO : RECORDS_OUT_OPERATOR_RS_11: 2
INFO : RECORDS_OUT_OPERATOR_SEL_13: 2
INFO : RECORDS_OUT_OPERATOR_SEL_9: 31
INFO : RECORDS_OUT_OPERATOR_TS_0: 31
INFO : Shuffle Errors:
INFO : BAD_ID: 0
INFO : CONNECTION: 0
INFO : IO_ERROR: 0
INFO : WRONG_LENGTH: 0
INFO : WRONG_MAP: 0
INFO : WRONG_REDUCE: 0
INFO : Shuffle Errors_Reducer_2_INPUT_Map_1:
INFO : BAD_ID: 0
INFO : CONNECTION: 0
INFO : IO_ERROR: 0
INFO : WRONG_LENGTH: 0
INFO : WRONG_MAP: 0
INFO : WRONG_REDUCE: 0
INFO : TaskCounter_Map_1_INPUT_dm_browser_app_retain_d:
INFO : INPUT_RECORDS_PROCESSED: 1
INFO : INPUT_SPLIT_LENGTH_BYTES: 1687
INFO : TaskCounter_Map_1_OUTPUT_Reducer_2:
INFO : ADDITIONAL_SPILLS_BYTES_READ: 0
INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0
INFO : ADDITIONAL_SPILL_COUNT: 0
INFO : OUTPUT_BYTES: 12
INFO : OUTPUT_BYTES_PHYSICAL: 56
INFO : OUTPUT_BYTES_WITH_OVERHEAD: 28
INFO : OUTPUT_LARGE_RECORDS: 0
INFO : OUTPUT_RECORDS: 2
INFO : SHUFFLE_CHUNK_COUNT: 1
INFO : SPILLED_RECORDS: 2
INFO : TaskCounter_Reducer_2_INPUT_Map_1:
INFO : ADDITIONAL_SPILLS_BYTES_READ: 56
INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 26
INFO : COMBINE_INPUT_RECORDS: 0
INFO : FIRST_EVENT_RECEIVED: 211
INFO : LAST_EVENT_RECEIVED: 211
INFO : MERGED_MAP_OUTPUTS: 2
INFO : MERGE_PHASE_TIME: 447
INFO : NUM_DISK_TO_DISK_MERGES: 0
INFO : NUM_FAILED_SHUFFLE_INPUTS: 0
INFO : NUM_MEM_TO_DISK_MERGES: 0
INFO : NUM_SHUFFLED_INPUTS: 2
INFO : NUM_SKIPPED_INPUTS: 0
INFO : REDUCE_INPUT_GROUPS: 2
INFO : REDUCE_INPUT_RECORDS: 2
INFO : SHUFFLE_BYTES: 56
INFO : SHUFFLE_BYTES_DECOMPRESSED: 28
INFO : SHUFFLE_BYTES_DISK_DIRECT: 30
INFO : SHUFFLE_BYTES_TO_DISK: 0
INFO : SHUFFLE_BYTES_TO_MEM: 26
INFO : SHUFFLE_PHASE_TIME: 391
INFO : SPILLED_RECORDS: 2
INFO : TaskCounter_Reducer_2_OUTPUT_out_Reducer_2:
INFO : OUTPUT_RECORDS: 0
INFO : org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters:
INFO : GROUPED_INPUT_SPLITS_Map_1: 1
INFO : INPUT_DIRECTORIES_Map_1: 1
INFO : INPUT_FILES_Map_1: 1
INFO : RAW_INPUT_SPLITS_Map_1: 1
INFO : Completed executing command(queryId=hive_20201014093918_9393118a-0614-4c9e-a6da-865afb5df98d); Time taken: 11.0 seconds
INFO : OK
