Graph Attention Networks (GAT)

This video provides a tutorial on Graph Attention Networks (GAT), a type of graph neural network that uses attention mechanisms to assign different importance weights to neighbors during message passing. The presenter begins by contrasting GAT with previous architectures: GCN uses degree-normalized averaging, and GraphSAGE uses sampling and concatenation, but both treat neighbors equally. GAT instead learns dynamic, feature-dependent attention weights. The video explains the GAT update function: a shared linear transformation is applied to node embeddings, then attention scores are computed by concatenating transformed embeddings of node pairs, applying a learnable vector and LeakyReLU, and normalizing with softmax across neighbors. The update is a weighted sum of transformed neighbor embeddings. The video also discusses multi-head attention for stability and expressivity, and provides a conceptual comparison of GCN, GraphSAGE, and GAT in terms of weight computation, transductive vs. inductive learning, scalability, and handling of homophily vs. heterophily. Strengths of GAT include permutation invariance, no need for full adjacency matrix multiplication, and higher expressivity than mean aggregation. Limitations include higher computational cost, potential instability without multi-head attention, and depth constraints common to message-passing networks.