Pooling Architecture Search

• paper

• technical report

The paper presents a novel framework called Pooling Architecture Search (PAS), which is based on Neural Architecture Search (NAS) to search for adaptive architectures for graph classification.

The Design of the Search Space

• 2 layers

• 6 types of Aggregation Module

  • GCN, GAT, GIN, MF, Transformer, and DeeperGCN

• 3 Pooling Module

  • TOPKPOOL, SAGPOOL, and ASAP

  • The pooling layer is slightly redundant for smaller graphs, and the result of adding pooling layer is not good after testing. Therefore, we delete the search of pooling layer here.

• 3 Readout Module

  • GLOBAL_MEAN, GLOBAL_MAX, and GLOBAL_SUM.

• 5 Merge Module

  • M_LSTM, M_CONCAT, M_MAX, M_MEAN, and M_SUM

on dataset * ogbg-molhiv * the small scale -> cancel the search of pooling and merge operations. * use heterogeneous interpolation process as a means of data enhancement * ogbg-molpcba * the small scale -> cancel the search of pooling and merge operations * ogbg-ppa * complete PAS search framework

Differentiable Search Algorithm

The differentiable search algorithm is a key component of the Pooling Architecture Search (PAS) framework. It is designed to enable efficient search on top of the search space. The algorithm relaxes the discrete search space into a continuous one by mixing the output of all candidate operations. This is achieved by a coarsening strategy designed to properly relax the selection of pooling operations in a continuous manner.

The algorithm constructs a supernet representing the search space for a 2-layer architecture backbone consisting of several modules. Each node in the supernet denotes the representations and each edge represents one operation from the corresponding sets. The discrete selection of operation in each module is relaxed by a weighted summation of all possible operations.

The algorithm also includes optimization by gradient descent. Based on the mixed results of each module in the supernet, the prediction can be calculated by a classifier. The cross-entropy loss on training data is calculated and optim

Pros

• Efficiency: The differentiable search algorithm is efficient as it enables the use of gradient descent for optimization. This reduces the search time by two orders of magnitude compared to reinforcement learning-based methods.

• Flexibility: The algorithm is flexible as it relaxes the discrete search space into a continuous one. This allows for a more comprehensive exploration of the search space.

• Scalability: The algorithm is scalable as it constructs a supernet representing the search space. This allows for the efficient handling of large search spaces.

Cons

• Complexity: The differentiable search algorithm can be complex to implement due to the need for a coarsening strategy to relax the selection of operations.

• Dependency on the Search Space: The performance of the algorithm is highly dependent on the design of the search space. A poorly designed search space may lead to suboptimal results.

• Requirement for Sufficient Training Data: The algorithm requires sufficient training data to effectively optimize the cross-entropy loss.ized.