Deep neural networks were shown to misclassify slightly modified input images. Recently, many defenses have been proposed but none have improved consistently the robustness of neural networks. Here, we propose to use attacks as a function evaluation to automatically search for architectures that can resist such attacks. Experiments on neural architecture search algorithms from the literature show that although their accurate results, they are not able to find robust architectures. Most of the reason for this lies in their limited search space. By creating a novel neural architecture search with options for dense layers to connect with convolution layers and vice-versa as well as the addition of multiplication, addition and concatenation layers in the search space, we were able to evolve an architecture that is $58\%$ accurate on adversarial samples. Interestingly, this inherent robustness of the evolved architecture rivals state-of-the-art defenses such as adversarial training while being trained only on the training dataset. Moreover, the evolved architecture makes use of some peculiar traits which might be useful for developing even more robust ones. Thus, the results here demonstrate that more robust architectures exist as well as opens up a new range of possibilities for the development and exploration of deep neural networks using automatic architecture search. Code available at http://bit.ly/RobustArchitectureSearch.