Detecting drifts in data is essential for machine learning applications, as changes in the statistics of processed data typically has a profound influence on the performance of trained models. Most of the available drift detection methods require access to true labels during inference time. In a real-world scenario, true labels usually available only during model training. In this work, we propose a novel task-sensitive drift detection framework, which is able to detect drifts without access to true labels during inference. It utilizes metric learning of a constrained low-dimensional embedding representation of the input data, which is best suited for the classification task. It is able to detect real drift, where the drift affects the classification performance, while it properly ignores virtual drift, where the classification performance is not affected by the drift. In the proposed framework, the actual method to detect a change in the statistics of incoming data samples can be chosen freely. We also propose the two change detection methods, which are based on the exponential moving average and a modified $z$-score, respectively. We evaluate the performance of the proposed framework with a novel metric, which accumulates the standard metrics of detection accuracy, false positive rate and detection delay into one value. Experimental evaluation on nine benchmarks datasets, with different types of drift, demonstrates that the proposed framework can reliably detect drifts, and outperforms state-of-the-art unsupervised drift detection approaches.