The evaluation of the BBB penetrating ability of drug molecules is a critical step in brain drug development. Computational prediction based on machine learning has proved to be an efficient way to conduct the evaluation. However, performance of the established models has been limited by their incapability of dealing with the interactions between drugs and proteins, which play an important role in the mechanism behind BBB penetrating behaviors. To address this issue, we employed the relational graph convolutional network (RGCN) to handle the drug-protein (denoted by the encoding gene) relations as well as the features of each individual drug. In addition, drug-drug similarity was also introduced to connect structurally similar drugs in the graph. The RGCN model was initially trained without input of any drug features. And the performance was already promising, demonstrating the significant role of the drug-protein/drug-drug relations in the prediction of BBB permeability. Moreover, molecular embeddings from a pre-trained knowledge graph were used as the drug features to further enhance the predictive ability of the model. Finally, the best performing RGCN model was built with a large number of unlabeled drugs integrated into the graph.