Neural network (NN) models that are solely trained to maximize the likelihood of an observed dataset are often vulnerable to adversarial attacks. Even though several methods have been proposed to enhance NN models' adversarial robustness, they often require re-training from scratch. This leads to redundant computation, especially in the NLP domain where current state-of-the-art models, such as BERT and ROBERTA, require great time and space resources. By borrowing ideas from Software Engineering, we, therefore, first introduce the Neural Patching mechanism to improve adversarial robustness by "patching" only parts of a NN model. Then, we propose a novel neural patching algorithm, SIENA, that transforms a textual NN model into a stochastic ensemble of multi-expert predictors by upgrading and re-training its last layer only. SIENA forces adversaries to attack not only one but multiple models that are specialized in diverse sub-sets of features, labels, and instances so that the ensemble model becomes more robust to adversarial attacks. By conducting comprehensive experiments, we demonstrate that all of CNN, RNN, BERT, and ROBERTA-based textual models, once patched by SIENA, witness an absolute increase of as much as 20% in accuracy on average under 5 different white and black-box attacks, outperforming 6 defensive baselines across 4 public NLP datasets.