In this paper, we introduce two algorithms for neural architecture search (NASGD and NASAGD) following the theoretical work by two of the authors [3]. Such work aims to introduce the conceptual basis for new notions of traditional and accelerated gradient descent algorithms for the optimization of a function on a semi-discrete space using ideas from optimal transport theory. Our methods, which use the network morphism framework introduced in [2], can analyze forty times as many architectures as the fastest methods in the literature [2,10] while using the same computational resources and time and achieving comparable levels of accuracy. For example, with NASGD, on CIFAR-10, our method designs and trains networks with an error rate of 4.06 in only 12 hours on a single GPU.