Fine-Grained Open-Set Fault Diagnosis via Metric-Guided Time-Frequency Configuration Selection and Class-Specific Autoencoders

Reliable fault diagnosis of rotating machinery is essential for the safe and stable operation of industrial systems. Although deep learning methods perform well under closed-set conditions, real machinery may encounter previously unseen fault states. Existing open-set fault diagnosis (OSFD) methods remain limited in fine-grained severity diagnosis because they often rely on coarse type levels, heuristically selected Short-Time Fourier Transform (STFT) settings, and global class boundaries. We propose a fine-grained OSFD method that combines metric-guided data-centric (MGDC) STFT configuration selection with class-specific autoencoder (CSAE)-based anomaly rejection. MGDC screens candidate STFT configurations using the Silhouette score computed from spectrogram representations, identifying promising time-frequency representations before network training. The diagnostic model then uses a bank of CSAEs to learn compact class-specific manifolds for known degradation states. During inference, reconstruction-error-based class affinity identifies known classes, while a dual-criteria mechanism based on latent dimension-wise boundaries and class-specific reconstruction error rejects unknown samples. Experiments on the Case Western Reserve University (CWRU) and Paderborn University (PU) bearing datasets show that the proposed method achieves H-scores of 0.9924 and 0.9509 for fine-grained fault severity diagnosis. MGDC also identifies the best-performing configuration found by exhaustive search while evaluating only 9 of 38 candidates on CWRU and 2 of 39 candidates on PU, reducing the selection cost by factors of 5.69 and 29.87, respectively. These results indicate that the proposed method supports accurate open-set severity diagnosis with substantially lower configuration-selection cost.