Code search aims to retrieve the relevant code fragments based on a natural language query to improve the software productivity and quality. However, automatic code search is challenging due to the semantic gap between the source code and the query. Most existing approaches mainly consider the sequential information for embedding, where the structure information behind the text is not fully considered. In this paper, we design a novel neural network framework, named GraphSearchNet, to enable an effective and accurate source code search by jointly learning rich semantics of both source code and queries. Specifically, we propose to encode both source code and queries into two graphs with Bidirectional GGNN to capture the local structure information of the graphs. Furthermore, we enhance BiGGNN by utilizing the effective multi-head attention to supplement the global dependency that BiGGNN missed. The extensive experiments on both Java and Python datasets illustrate that GraphSearchNet outperforms current state-of-the-art works by a significant margin.