One-stage object detectors rely on the point feature to predict the detection results. However, the point feature may lack the information of the whole object and lead to a misalignment between the object and the point feature. Meanwhile, the classification and regression tasks are sensitive to different object regions, but their features are spatially aligned. In this paper, we propose a simple and plug-in operator that could generate aligned and disentangled features for each task, respectively, without breaking the fully convolutional manner. By predicting two task-aware point sets that are located in each sensitive region, this operator could disentangle the two tasks from the spatial dimension, as well as align the point feature with the object. We also reveal an interesting finding of the opposite effect of the long-range skip-connection for classification and regression, respectively. Based on the object-aligned and task-disentangled operator (OAT), we propose OAT-Net, which explicitly exploits point-set features for more accurate detection results. Extensive experiments on the MS-COCO dataset show that OAT can consistently boost different one-stage detectors by $\sim$2 AP. Notably, OAT-Net achieves 53.7 AP with Res2Net-101-DCN backbone and shows promising performance gain for small objects.