Most existing object detectors suffer from class imbalance problems that hinder balanced performance. In particular, anchor free object detectors have to solve the background imbalance problem due to detection in a per-pixel prediction fashion as well as foreground imbalance problem simultaneously. In this work, we propose Balance-oriented focal loss that can induce balanced learning by considering both background and foreground balance comprehensively. This work aims to address imbalance problem in the situation of using a general unbalanced data of non-extreme distribution not including few shot and the focal loss for anchor free object detector. We use a batch-wise alpha-balanced variant of the focal loss to deal with this imbalance problem elaborately. It is a simple and practical solution using only re-weighting for general unbalanced data. It does require neither additional learning cost nor structural change during inference and grouping classes is also unnecessary. Through extensive experiments, we show the performance improvement for each component and analyze the effect of linear scheduling when using re-weighting for the loss. By improving the focal loss in terms of balancing foreground classes, our method achieves AP gains of +1.2 in MS-COCO for the anchor free real-time detector.