CTR学习笔记&代码实现3-深度ctr模型 FNN->PNN->DeepFM

2020-05-17 15:45:55 浏览数 (1)
  • IPNN是两两特征做内积
((1*k)*(k*1))

得到的scaler拼接成p，维度是

(O(N^2))
代码语言：txt复制
    - OPNN是两两特征做外积
((k*1)*(1*k))

得到

(K^2)

的矩阵拼接成p，维度是

(O(K^2N^2))
代码语言：txt复制
    - PNN就是[IPNN,OPNN]

之后跟全连接层。可以发现去掉全联接层把权重都设为1，把线性部分对接到最初的离散输入那IPNN就退化成了FM。

Product层的优化

以上IPNN和OPNN的计算都有维度过高，计算复杂度过高的问题，作者进行了相应的优化。

  • OPNN:
(O(N^2K^2) to O(K^2))

原始的OPNN，p中的每个元素都是向量外积的矩阵

(p_{i,j} in R^{k*k})

,优化后作者对所有K*K矩阵进行了sum_pooling，也等同于先对隐向量求和

(R^{N*K} to R^{1*K})

再做外积

(p = sum_{i=1}^Nsum_{j=1}^N f_if_j^T=f_{sum}(f_{sum})^T , , f_{sum}=sum_{i=1}^Nf_i)
  • IPNN：
(O(N^2) to O(N*K))

IPNN的优化又一次用到了FM的思想，内积产生的

(p in R^{N^2})

是对称矩阵,因此在之后全联接层的权重

(w_p)

也一定是对称矩阵。所以用矩阵分解来进行降维，假定每个神经元对应的权重

(w_p^n = theta^n{theta^n}^T text{where } theta^n in R^N)
(w_p^n odot p = sum_{i=1}^Nsum_{j=1}^Ntheta^n_itheta^n_j<f_i,f_j>=<sum_{i=1}^Ndelta_i^n,sum_{i=1}^Ndelta_i^n>)

PNN的几个可能可以吐槽的地方

  • 和FNN一样对低阶特征的提炼比较有限
  • OPNN的部分如果不优化维度太高，在我尝试的训练集上基本是没啥用。优化后这个sum_pooling究竟还保留了什么信息，我是没太琢磨明白

代码实现

代码语言：javascript复制
@tf_estimator_model
def model_fn(features, labels, mode, params):
    dense_feature= build_features()
    dense = tf.feature_column.input_layer(features, dense_feature) # lz linear concat of embedding

    feature_size = len( dense_feature )
    embedding_size = dense_feature[0].variable_shape.as_list()[-1]
    embedding_matrix = tf.reshape( dense, [-1, feature_size, embedding_size] )  # batch * feature_size *emb_size

    with tf.variable_scope('IPNN'):
        # use matrix multiplication to perform inner product of embedding
        inner_product = tf.matmul(embedding_matrix, tf.transpose(embedding_matrix, perm=[0,2,1])) # batch * feature_size * feature_size
        inner_product = tf.reshape(inner_product, [-1, feature_size * feature_size ])# batch * (feature_size * feature_size)
        add_layer_summary(inner_product.name, inner_product)

    with tf.variable_scope('OPNN'):
        outer_collection = []
        for i in range(feature_size):
            for j in range(i 1, feature_size):
                vi = tf.gather(embedding_matrix, indices = i, axis=1, batch_dims=0, name = 'vi') # batch * embedding_size
                vj = tf.gather(embedding_matrix, indices = j, axis=1, batch_dims= 0, name='vj') # batch * embedding_size
                outer_collection.append(tf.reshape(tf.einsum('ai,aj->aij',vi,vj), [-1, embedding_size * embedding_size])) # batch * (emb * emb)

            outer_product = tf.concat(outer_collection, axis=1)
        add_layer_summary( outer_product.name, outer_product )

    with tf.variable_scope('fc1'):
        if params['model_type'] == 'IPNN':
            dense = tf.concat([dense, inner_product], axis=1)
        elif params['model_type'] == 'OPNN':
            dense = tf.concat([dense, outer_product], axis=1)
        elif params['model_type'] == 'PNN':
            dense = tf.concat([dense, inner_product, outer_product], axis=1)
        add_layer_summary( dense.name, dense )

    with tf.variable_scope('Dense'):
        for i, unit in enumerate( params['hidden_units'] ):
            dense = tf.layers.dense( dense, units=unit, activation='relu', name='dense{}'.format( i ) )
            dense = tf.layers.batch_normalization( dense, center=True, scale=True, trainable=True,
                                                   training=(mode == tf.estimator.ModeKeys.TRAIN) )
            dense = tf.layers.dropout( dense, rate=params['dropout_rate'],
                                       training=(mode == tf.estimator.ModeKeys.TRAIN) )
            add_layer_summary( dense.name, dense)

    with tf.variable_scope('output'):
        y = tf.layers.dense(dense, units=1, name = 'output')
        add_layer_summary( 'output', y )

    return y

DeepFM

DeepFM是对Wide&Deep的Wide侧进行了改进。之前的Wide是一个LR，输入是离散特征和交互特征，交互特征会依赖人工特征工程来做cross。DeepFM则是用FM来代替了交互特征的部分，和Wide&Deep相比不再依赖特征工程，同时cross-column的剔除可以降低输入的维度。

和PNN/FNN相比，DeepFM能更多提取到到低阶特征。而且上述这些模型间直接并不互斥，比如把DeepFM的FMLayer共享到Deep部分其实就是IPNN。

模型

Wide部分就是一个FM，输入是N个one-hot的离散特征，每个离散特征对应到等长的低维(k)embedding上，最终输出的就是之前FM模型的output。并且因为这里不需要像IPNN一样输出隐向量，因此可以使用FM降低复杂度的trick。

Deep部分和Wide部分共享N*K的Embedding输入层，然后跟两个全联接层

Deep和Wide联合训练，模型最终的输出是FM部分和Deep部分权重为1的简单加和。联合训练共享Embedding也保证了二阶特征交互学到的Embedding会和高阶信息学到的Embedding的一致性。

代码实现

代码语言：javascript复制
@tf_estimator_model
def model_fn(features, labels, mode, params):
    dense_feature, sparse_feature = build_features()
    dense = tf.feature_column.input_layer(features, dense_feature)
    sparse = tf.feature_column.input_layer(features, sparse_feature)

    with tf.variable_scope('FM_component'):
        with tf.variable_scope( 'Linear' ):
            linear_output = tf.layers.dense(sparse, units=1)
            add_layer_summary( 'linear_output', linear_output )

        with tf.variable_scope('second_order'):
            # reshape (batch_size, n_feature * emb_size) -> (batch_size, n_feature, emb_size)
            emb_size = dense_feature[0].variable_shape.as_list()[0] # all feature has same emb dimension
            embedding_matrix = tf.reshape(dense, (-1, len(dense_feature), emb_size))
            add_layer_summary( 'embedding_matrix', embedding_matrix )
            # Compared to FM embedding here is flatten(x * v) not v
            sum_square = tf.pow( tf.reduce_sum( embedding_matrix, axis=1 ), 2 )
            square_sum = tf.reduce_sum( tf.pow(embedding_matrix,2), axis=1 )

            fm_output = tf.reduce_sum(tf.subtract( sum_square, square_sum) * 0.5, axis=1, keepdims=True)
            add_layer_summary('fm_output', fm_output)

    with tf.variable_scope('Deep_component'):
        for i, unit in enumerate(params['hidden_units']):
            dense = tf.layers.dense(dense, units = unit, activation ='relu', name = 'dense{}'.format(i))
            dense = tf.layers.batch_normalization(dense, center=True, scale = True, trainable=True,
                                                  training=(mode ==tf.estimator.ModeKeys.TRAIN))
            dense = tf.layers.dropout( dense, rate=params['dropout_rate'], training = (mode==tf.estimator.ModeKeys.TRAIN))
            add_layer_summary( dense.name, dense )

    with tf.variable_scope('output'):
        y = dense   fm_output   linear_output
        add_layer_summary( 'output', y )

    return y

等处理好另一份样本, 会把代码更新成匹配高维稀疏特征feat_id:feat_val输入格式的。完整代码在这里 https://github.com/DSXiangLi/CTR

CTR学习笔记&代码实现系列?

CTR学习笔记&代码实现1-深度学习的前奏LR->FFM

CTR学习笔记&代码实现2-深度ctr模型 MLP->Wide&Deep

资料

  1. Huifeng Guo et all. "DeepFM: A Factorization-Machine based Neural Network for CTR Prediction," In IJCAI,2017.
  2. Qu Y, Cai H, Ren K, et al. Product-based neural networks for user response prediction,2016 IEEE
  3. Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data - - A case study on user response
  4. https://daiwk.github.io/posts/dl-dl-ctr-models.html
  5. https://zhuanlan.zhihu.com/p/86181485
推荐系统 机器学习 深度学习

0 人点赞