Randomized smoothing has achieved state-of-the-art certified robustness against $l_2$-norm adversarial attacks. However, it also leads to accuracy drop compared to the normally trained models. In this work, we employ a Smoothed WEighted ENsembling (SWEEN) scheme to improve the performance of randomized smoothed classifiers. We characterize the optimal certified robustness attainable by SWEEN models. We show the accessibility of SWEEN models attaining the lowest risk w.r.t. a surrogate loss function. We also develop an adaptive prediction algorithm to reduce the prediction and certification cost of SWEEN models. Extensive experiments show that SWEEN models outperform the upper envelope of their corresponding base models by a large margin. Moreover, SWEEN models constructed using a few small models are able to achieve comparable performance to a single large model with notably reduced training time.