GCC图神经网络预训练概述

Paper

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Conference: KDD

Keywords: Contrastive Learning, Instance Discrimination, Transferability, Pre-training

Recent Work

However, most representation learning work on graphs has thus far focused on learning representations for one single graph or a fixed set of graphs and very limited work can be transferred to out-of-domain data and tasks.

Challenge

How to design the pre-training task such that the universal structural patterns in and across networks can be captured and further transferred?

Main idea

The idea of pre-training is to use the pre-trained model as a good initialization for fine-tuning over (different) tasks on unseen datasets. Contrastive learning is another idea. Here are three critical steps. (1) Define instances in graphs (2) Define similar instance pairs in and across graphs (3) Choose the proper graph encoders.

Contributions

(1) Formalize the problem of GNN pre-training

(2) Regard pre-training as instance discrimination to capture the universal and transferable structural patterns from multiple input graphs

(3) GCC framework is proposed

(4) Experiments

Requirements

(1) Structural Similarity, it maps vertices with similar local network topologies close to each other in the vector space.

(2) Transferability, it is compatible with vertices and graphs unseen by the pre-training algorithm.

Steps

Ideally, given a (diverse) set of input graphs, such as the Facebook social graph and the DBLP co-author graph, we aim to pre-train a GNN on them with a self-supervised task, and then fine-tune it on different graphs for different graph learning tasks, such as node classification on the US-Airport graph.

Compared Algorithms

RolX

Panther

GraphWave

Some thoughts

Combine attributes

BERT-GRAPH

Remove nodes with noise

Aggregate nodes as a new instance

More Papers About Pre-Training

Strategies for Pre-training Graph Neural Networks. 2020. ICLR

GPT-GNN: Generative Pre-Training of Graph Neural Networks. 2020. KDD

Pre-TrainingGraphNeuralNetworksfor Generic Structural Feature Extraction. 2020

Self-supervised Learning: Generative or Contrastive. 2020.

Gaining insight into SARS-CoV-2 infection and COVID-19 severity using self-supervised edge features and Graph Neural Networks. 2020. ICML

When Does Self-Supervision Help Graph Convolutional Networks? 2020. ICML

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes. 2020. AAAI

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. 2020. KDD

Self-Supervised Graph Representation Learning via Global Context Prediction. 2020.

Contrastive Multi-View Representation Learning on Graphs. 2020.

Self-supervised Training of Graph Convolutional Networks. 2020.

Self-supervised Learning on Graphs: Deep Insights and New Directions. 2020.

GRAPH-BERT: Only Attention is Needed for Learning Graph Representations. 2020.

Graph Neural Distance Metric Learning with GRAPH-BERT. 2020.

Segmented GRAPH-BERT for Graph Instance Modeling. 2020.

Reference

https://zhuanlan.zhihu.com/p/150456349

关键词: 对比学习，实例辨析，可转移性，预训练

图神经网络预训练的图对比编码