Most of existing gradient-based meta-learning approaches to few-shot learning assume that all tasks have the same input feature space. However, in the real world scenarios, there are many cases that the input structures of tasks can be different, that is, different tasks may vary in the number of input modalities or the data structure of each modality. Existing gradient-based approaches cannot handle such heterogeneous task distribution (HTD) as different types of tasks only share partial meta-parameters. In this paper, we propose HetMAML, a task-heterogeneous meta-agnostic meta-learning framework that can generalize not only common meta-parameters shared across different types of tasks but also type-specific meta-parameters. Specifically, we design a multi-channel backbone module that encodes the input of each type of tasks into the same length sequence of modality-specific embeddings. Then, we propose a task-aware multimodal encoder which can automatically take into account the context of task-specific input structures and adaptively project the heterogeneous input spaces to the same lower-dimensional concept space. The extensive experiments on five task-heterogeneous datasets demonstrate that our HetMAML successfully captures both type-specific and shared meta-parameters across heterogeneous tasks which fast adapt to all types of new tasks.