【论文阅读】Deep Graph Contrastive Representation Learning

本文介绍: 本文为《Deep Graph Contrastive Representation Learning》的阅读笔记，从研究动机，创新点出发，到方法论，最后还有代码实现。

k=1∑N1[k=i]eθ(ui,uk)/τeθ(ui,vi)/τ

其中，

[

≠

]

∈

{

}

mathbb{1}_{[kneq i]}in{0,1}

$1_{[k \neq = i]} \in {0, 1}$ 是一个指示函数，当且仅当

≠

k neq i

$k \neq = i$ 时定于1。两个视图是对称的，另一个视图定义类似

ℓ

(

)

ell(boldsymbol{v}_i,boldsymbol{u}_i)

$ℓ (v_{i}, u_{i})$ ，最后，要最大化的总体目标被定义为：

∑

[

ℓ

(

)

ℓ

(

)

]

mathcal{J}=dfrac{1}{2N}sum_{i=1}^Nleft[ell(boldsymbol{u}_i,boldsymbol{v}_i)+ell(boldsymbol{v}_i,boldsymbol{u}_i)right]

$J = \frac{1}{2 N} i = 1 \sum N [ℓ (u_{i}, v_{i}) + ℓ (v_{i}, u_{i})]$

3.5、评估

类似于DGI中的线性评估方案，模型首先以无监督的方式训练，得到的嵌入被用来训练逻辑回归分类器并做测试。

3.6、理论动机

3.6.1、最大化目标函数等价于最大化互信息的下界

定理1说明了目标函数

mathcal{J}

$J$ 是InfoNCE目标函数的一个下界，而InfoNCE评估器是MI（即互信息）的下界，所以

≤

(

;

)

mathcal{J} le I(X;U,V)

$J \leq I (X; U, V)$ 。
所以，最大化目标函数

mathcal{J}

$J$ 等价于最大化输入节点特征和学习节点表示之间的互信息

(

;

)

I(X;U,V)

$I (X; U, V)$ 的下界

3.6.2、三重损失

定理2说明了最小化目标函数与最大化三重损失一致。更详细的证明请看原文。

triplet Loss是深度学习中的一种损失函数，用于训练差异性较小的样本，如人脸等。在人脸识别领域，triplet loss常被用来提取人脸的embedding。输入数据是一个三元组，包括锚（Anchor）例、正（Positive）例、负（Negative）例，通过优化锚示例与正示例的距离小于锚示例与负示例的距离，实现样本的相似性计算。

3.7、实验参数设置

Dataset	$p_{m,1} pm,1$	$p_{m,2} pm,2$	$p_{r,1} pr,1$	$p_{r,2} pr,2$	lr	wd	epoch	hidfeat	activation
Cora	0.3	0.4	0.2	0.4	0.005	1e-5	200	128	ReLU
Citeseer	0.3	0.2	0.2	0.0	0.001	1e-5	200	256	PReLU
Pubmed	0.0	0.2	0.4	0.1	0.001	1e-5	1500	256	ReLU

4、代码实现

完整代码见
链接：https://pan.baidu.com/s/1g9Rhe1EjxBZ0dFgOfy3CSg
提取码：6666

4.1、RE and MF

from dgl.transforms import DropEdge
#RE
#随机删除边——使用dgl内建库DropEdge
#MF
#随机掩盖特征
def drop_feature(x, drop_prob):
    drop_masks=[]
    for i in range(x.shape[0]):
        drop_mask = torch.empty(
            size= (x.size(1),) ,
            dtype=torch.float32,
            device=x.device).uniform_(0, 1) < drop_prob
        drop_masks.append(drop_mask)
    x = x.clone()
    for i,e in enumerate(drop_masks):
        x[i,e] = 0
    return x

4.2、encorder

import dgl
import torch.nn as nn
from dgl.nn.pytorch import GraphConv
from model.GCNLayer import GCNLayer

class Encoder(nn.Module):
    def __init__(self, infeat: int, outfeat: int, act_func,base_model=GraphConv, k: int = 2):

        super(Encoder, self).__init__()

        self.base_model = base_model
        assert k >= 2
        self.k = k
        self.convs = nn.ModuleList()
        self.convs.append(base_model(infeat, 2 * outfeat))
        for _ in range(1, k-1):
            self.convs.append(base_model(2 * outfeat, 2 * outfeat))
        self.convs.append(base_model(2 * outfeat, outfeat))
        self.act_func = act_func
    def forward(self, g, x ):
        #g = dgl.add_self_loop(g)
        for i in range(self.k):
            x = self.act_func(self.convs[i](g,x))
        return x

4.3、GRACE

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from dgl.nn.pytorch import GraphConv
from model.encoder import Encoder
class GRACE(nn.Module):
   def __init__(self,infeat,hidfeat,act_func,k=2) -> None:
      super(GRACE,self).__init__()
      self.encoder = Encoder(infeat,hidfeat,act_func,base_model=GraphConv,k=k)
   def forward(self,g,x):
      z =self.encoder(g,x)
      return z

4.4、loss

import torch
import torch.nn as nn
import torch.nn.functional as F
class LossFunc(nn.Module):
    def __init__(self, infeat,hidfeat,outfeat,tau) -> None:
        super(LossFunc,self).__init__()
        self.tau = tau
        self.layer1 = nn.Linear(infeat,hidfeat)
        self.layer2 = nn.Linear(hidfeat,outfeat)
    def projection(self,x):
        x = F.elu(self.layer1(x))
        x = self.layer2(x)
        return x
    def sim(self,x,y):
        x = F.normalize(x)
        y = F.normalize(y)
        return torch.mm(x, y.t())
    def sim_loss(self,h1,h2):
        f = lambda x : torch.exp(x/self.tau)
        #exp(theta(u_i,u_j)/tau)
        intra_sim = f(self.sim(h1,h1))
        #exp(theta(u_i,v_j)/tau)
        inter_sim = f(self.sim(h1,h2))
        return -torch.log(
            inter_sim.diag() / (intra_sim.sum(1) + inter_sim.sum(1) - intra_sim.diag())
            )
    def forward(self,u,v):
        h1 = self.projection(u)
        h2 = self.projection(v)
        loss1 = self.sim_loss(h1,h2)
        loss2 = self.sim_loss(h2,h1)
        loss_sum = (loss1 + loss2) * 0.5
        res = loss_sum.mean()
        return res