Cardiovascular diseases are the leading cause of death and require a spectrum of diagnostic procedures as well as invasive interventions. Medical imaging is a vital part of the healthcare system, facilitating both diagnosis and guidance for intervention. Intravascular ultrasound and optical coherence tomography are widely available for characterizing coronary stenoses and provide critical vessel parameters to optimize percutaneous intervention. Intravascular polarization-sensitive optical coherence tomography (PS-OCT) can simultaneously provide high-resolution cross-sectional images of vascular structures while also revealing preponderant tissue components such as collagen and smooth muscle and thereby enhance plaque characterization. Automated interpretation of these features would facilitate the objective clinical investigation of the natural history and significance of coronary atheromas. Here, we propose a convolutional neural network model and optimize its performance using a new multi-term loss function to classify the lumen, intima, and media layers in addition to the guidewire and plaque artifacts. Our multi-class classification model outperforms the state-of-the-art methods in detecting the anatomical layers based on accuracy, Dice coefficient, and average boundary error. Furthermore, the proposed model segments two classes of major artifacts and detects the anatomical layers within the thickened vessel wall regions, which were excluded from analysis by other studies. The source code and the trained model are publicly available at https://github.com/mhaft/OCTseg .