Objective: This paper suggests a novel deep learning architecture for multi-class classification and localization of abnormalities in mammograms. We first assume a weakly supervised setting and present a new approach with data driven decisions. Next, we extend this method to a semi-supervised setting that engages a small set of local annotations. Methods: This novel network combines two learning branches with region-level classification and region ranking, explicitly handling the common normal images without any finding. Our method enables detection of abnormalities in full mammogram resolution for both weakly and semi-supervised settings. A novel objective function allows engagement of local annotations into the model. Results: We present the impact of our schemes with several performance measures for classification and localization, to evaluate the cost effectiveness of lesion annotation effort. Our evaluation is mainly made over a large multi-center mammography dataset of $\sim$3,000 mammograms with various findings. The results for weakly supervised learning show improvement of 4% in AUC, 10-17% in partial AUC (AUC at high sensitivity) and 8-15% in specificity at 0.85 sensitivity, compared to a previous approach. Local annotation of only 5% of the mammograms further boosts performance by 8.2% in specificity and 30% in detection. Conclusion: Lack of laborious local annotations for supervised learning can be addressed by a weakly supervised method that can leverage a subset of locally annotated data. Significance: Weakly and semi-supervised methods coupled with detection suggest a cost effective and explainable model to be adopted by radiologists in the field.