Cyclic Differentiable Architecture Search

Recently, differentiable architecture search has draw great attention due to its high efficiency and competitive performance. It searches the optimal architecture in a shallow network, and then measures its performance in a deep evaluation network. This leads to the optimization of architecture search is independent of the target evaluation network, and the discovered architecture is sub-optimal. To address this issue, we propose a novel cyclic differentiable architecture search framework (CDARTS). Considering the structure difference, CDARTS builds a cyclic feedback mechanism between the search and evaluation networks. First, the search network generates an initial topology for evaluation, so that the weights of the evaluation network can be optimized. Second, the architecture topology in the search network is further optimized by the label supervision in classification, as well as the regularization from the evaluation network through feature distillation. Repeating the above cycle results in a joint optimization of the search and evaluation networks, and thus enables the evolution of the topology to fit the final evaluation network. The experiments and analysis on CIFAR, ImageNet and NAS-Bench- 201 demonstrate the efficacy of the proposed approach.