Although deep reinforcement learning has recently been very successful at learning complex behaviors, it requires tremendous amount of data to learn a task, let alone being able to adapt to new tasks. One of the fundamental reasons causing this limitation lies in the nature of the trial-and-error learning paradigm of reinforcement learning, where the agent communicates with the task and progresses the learning only relying on the reward signal, which is implicit and insufficient to learn a task well. On the contrary, human beings mainly learn new skills via semantic representations or natural language instructions. However, utilizing language instructions towards robotic motion control to improve the adaptability is a newly emerged topic and challenging as well. In this paper, we present an meta-RL algorithm that addresses the challenge of learning skills with language instructions in multiple manipulation tasks. On the one hand, our algorithm utilizes the language instructions to shape its interpretation of the task, on the other hand, it still learns to solve task in a trial-and-error process. We evaluate our algorithm on the robotic manipulation benchmark (Meta-World) and significantly outperforms state-of-the-arts method in terms of training and testing success rates. The code is available at \url{https://tumi6robot.wixsite.com/million}.