The segregation of brain fiber tractography data into distinct and anatomically meaningful clusters can help to comprehend the complex brain structure and early investigation and management of various neural disorders. We propose a novel stacked bidirectional long short-term memory(LSTM) based segmentation network, (BrainSegNet) for human brain fiber tractography data classification. We perform a two-level hierarchical classification a) White vs Grey matter (Macro) and b) White matter clusters (Micro). BrainSegNet is trained over three brain tractography data having over 250,000 fibers each. Our experimental evaluation shows that our model achieves state-of-the-art results. We have performed inter as well as intra class testing over three patient's brain tractography data and achieved a high classification accuracy for both macro and micro levels both under intra as well as inter brain testing scenario.