Personalization lies at the core of boosting the product search system performance. Prior studies mainly resorted to the semantic matching between textual queries and user/product related documents, leaving the user collaborative behaviors untapped. In fact, the collaborative filtering signals between users intuitively offer a complementary information for the semantic matching. To close the gap between collaborative filtering and product search, we propose a Hierarchical Heterogeneous Graph Neural Network (HHGNN) approach in this paper. Specifically, we organize HHGNN with a hierarchical graph structure according to the three edge types. The sequence edge accounts for the syntax formulation from word nodes to sentence nodes; the composition edge aggregates the semantic features to the user and product nodes; and the interaction edge on the top performs graph convolutional operation between user and product nodes. At last, we integrate the higher-order neighboring collaborative features and the semantic features for better representation learning. We conduct extensive experiments on six Amazon review datasets. The results show that our proposed method can outperform the state-of-the-art baselines with a large margin. In addition, we empirically prove that collaborative filtering and semantic matching are complementary to each other in product search performance enhancement.