Automated Computer Aided diagnostic tools can be used for the early detection of glaucoma to prevent irreversible vision loss. In this work, we present a Multi-task Convolutional Neural Network (CNN) that jointly segments the Optic Disc (OD), Optic Cup (OC) and predicts the presence of glaucoma in color fundus images. The CNN utilizes a combination of image appearance features and structural features obtained from the OD-OC segmentation to obtain a robust prediction. The use of fewer network parameters and the sharing of the CNN features for multiple related tasks ensures the good generalizability of the architecture, allowing it to be trained on small training sets. The cross-testing performance of the proposed method on an independent validation set acquired using a different camera and image resolution was found to be good with an average dice score of 0.92 for OD, 0.84 for OC and AUC of 0.95 on the task of glaucoma classification illustrating its potential as a mass screening tool for the early detection of glaucoma.