In this paper, we describe a fast and light-weight portrait segmentation method based on a new {\em extremely light-weight backbone} (ELB) architecture. The core element of ELB is a {\em bottleneck-based factorized block} (BFB) that has much fewer parameters than existing alternatives while keeping good learning capacity. Consequently, the ELB-based portrait segmentation method can run faster (263.2FPS) than the existing methods yet retaining the competitive accuracy performance with state-of-the-arts. Experiments conducted on two benchmark datasets demonstrate the effectiveness and efficiency of our method.