The task of table structure recognition is to recognize the internal structure of a table, which is a key step to make machines understand tables. However, tabular data in unstructured digital documents, e.g. Portable Document Format (PDF) and images, are difficult to parse into structured machine-readable format, due to complexity and diversity in their structure and style, especially for complex tables. In this paper, we introduce Split, Embed and Merge (SEM), an accurate table structure recognizer. In the first stage, we use the FCN to predict the potential regions of the table row (column) separators, so as to obtain the bounding boxes of the basic grids in the table. In the second stage, we not only extract the visual features corresponding to each grid through RoIAlign, but also use the off-the-shelf recognizer and the BERT to extract the semantic features. The fused features of both are used to characterize each table grid. We find that by adding additional semantic features to each grid, the ambiguity problem of the table structure from the visual perspective can be solved to a certain extent and achieve higher precision. Finally, we process the merging of these basic grids in a self-regression manner. The correspondent merging results is learned by the attention maps in attention mechanism. With the proposed method, we can recognize the structure of tables well, even for complex tables. SEM can achieve an average F-Measure of $96.9\%$ on the SciTSR dataset which outperforms other methods by a large margin. Extensive experiments on other publicly available table structure recognition datasets show that our model achieves state-of-the-art.