Hive自定义函数的使用——useragent解析

想要从日志数据中分析一下操作系统、浏览器、版本使用情况，但是hive中的函数不能直接解析useragent,于是可以写一个UDF来解析。useragent用于表示用户的当前操作系统，浏览器版本信息，形如：

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 180.173.196.29

其中解析ua可以用一个开源的工具包,叫做useragentutils.jar来处理，但是不能直接引入这个包，因为Hadoop和hive都不支持直接引用第三方的包，要导入源码。项目结构应该如下图

下面的代码用来打印出操作系统、浏览器版本信息：

import org.apache.hadoop.hive.ql.exec.UDF; import org.apache.hadoop.io.Text;

import eu.bitwalker.useragentutils.UserAgent;

public class ParseUserAgent_UDF extends UDF{  public Text evaluate(final Text userAgent){   StringBuilder builder = new StringBuilder();   UserAgent ua = new UserAgent(userAgent.toString());   builder.append(ua.getOperatingSystem() "t" ua.getBrowser() "t" ua.getBrowserVersion());   return new Text(builder.toString());  } }

使用：打成jar包，hive中add jar xx.jar;

create temporary function ua_parse as 'com.xx.ParseUserAgent_UDF';

select ua_parse(ua) from table_name limit 3;

结果：

WINDOWS_7      CHROME21        21.0.1180.89 WINDOWS_7      CHROME33        33.0.1750.146 WINDOWS_7      CHROME21        21.0.1180.89

此种方式只能处理一行，生成一行，无法进行统计分析。

下面使用UDTF(User Defined Table Generating Function),处理一行，生成多列。

import java.util.ArrayList;

import org.apache.hadoop.hive.ql.exec.UDFArgumentException; import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException; import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;

import eu.bitwalker.useragentutils.UserAgent;

public class ParseUserAgent_UDTF extends GenericUDTF{  @Override  public StructObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {   if (args.length != 1) {    throw new UDFArgumentLengthException("ExplodeMap takes only one argument");   }   if (args[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {    throw new UDFArgumentException("ExplodeMap takes string as a parameter");   }   ArrayList<String> fieldNames = new ArrayList<String>();   ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();   fieldNames.add("system");   fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);   fieldNames.add("browser");   fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);   fieldNames.add("version");   fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);   return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);  }  @Override  public void process(Object[] arg){   try {    if(arg == null || arg.length == 0)     return;    String input = arg[0].toString();    String result[] = ua_parse(input).split("t");    forward(result);   } catch (Exception e) {    e.printStackTrace();   }  }  @Override  public void close() throws HiveException {  }  public String ua_parse(String userAgent){   StringBuilder builder = new StringBuilder();   UserAgent ua = new UserAgent(userAgent.toString());   builder.append(ua.getOperatingSystem() "t" ua.getBrowser() "t" ua.getBrowserVersion());   return builder.toString();  } }

select t.browser,count(*) c from (select ua_parse(ua) as (system,browser,version) from table_name) t group by t.browser order by c desc;

前十名：

CHROME31        987220571 UNKNOWN 708890045 IE8    420021677 IE7    411500373 MOBILE_SAFARI  291920740 IE6    217574865 IE11    179582201 IE9    165160040 CHROME30        158623163 CHROME21        155192489

未识别的还是很多！