We propose a new method for distributed estimation of a linear model by a network of local learners with heterogeneously distributed datasets. Unlike other ensemble learning methods, in the proposed method, model averaging is done continuously over time in a distributed and asynchronous manner. To ensure robust estimation, a network regularization term which penalizes models with high local variability is used. We provide a finite-time characterization of convergence of the weighted ensemble average and compare this result to centralized estimation. We illustrate the general applicability of the method in two examples: estimation of a Markov random field using wireless sensor networks and modeling prey escape behavior of birds based on a real-world dataset.