The task-oriented semantic communication systems have achieved significant performance gain, however, the paradigm that employs a model for a specific task might be limited, since the system has to be updated once the task is changed or multiple models are stored for serving various tasks. To address this issue, we firstly propose a unified deep learning enabled semantic communication system (U-DeepSC), where a unified model is developed to serve various transmission tasks. To jointly serve these tasks in one model with fixed parameters, we employ domain adaptation in the training procedure to specify the task-specific features for each task. Thus, the system only needs to transmit the task-specific features, rather than all the features, to reduce the transmission overhead. Moreover, since each task is of different difficulty and requires different number of layers to achieve satisfactory performance, we develop the multi-exit architecture to provide early-exit results for relatively simple tasks. In the experiments, we employ a proposed U-DeepSC to serve five tasks with multi-modalities. Simulation results demonstrate that our proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task with significant transmission overhead reduction and much less number of model parameters.