* Some of the results have appeared in AISTATS 2018. This is an
extended version with additional results, in particular, an adaptive
synchronicity strategy called AdaSync. arXiv admin note: substantial text
overlap with arXiv:1803.01113 Access Paper or Ask Questions