Electroencephalography (EEG) signals are often contaminated with artifacts. It is imperative to develop a practical and reliable artifact removal method to prevent misinterpretations of neural signals and underperformance of brain-computer interfaces. This study developed a new artifact removal method, IC-U-Net, which is based on the U-Net architecture for removing pervasive EEG artifacts and reconstructing brain sources. The IC-U-Net was trained using mixtures of brain and non-brain sources decomposed by independent component analysis and employed an ensemble of loss functions to model complex signal fluctuations in EEG recordings. The effectiveness of the proposed method in recovering brain sources and removing various artifacts (e.g., eye blinks/movements, muscle activities, and line/channel noises) was demonstrated in a simulation study and three real-world EEG datasets collected at rest and while driving and walking. IC-U-Net is user-friendly and publicly available, does not require parameter tuning or artifact type designations, and has no limitations on channel numbers. Given the increasing need to image natural brain dynamics in a mobile setting, IC-U-Net offers a promising end-to-end solution for automatically removing artifacts from EEG recordings.