Probabilistic matrix factorization (PMF) is a well-known model of recommender systems. With the development of image recognition technology, some PMF recommender systems that combine images have emerged. Some of these systems use the image shape features of the recommended products to achieve better results compared to those of the traditional PMF. However, in the existing methods, no PMF recommender system can combine the image features of products previously purchased by customers and of recommended products. Thus, this study proposes a novel probabilistic model that integrates double convolutional neural networks (CNNs) into PMF. For apparel goods, two trained CNNs from the image shape features of users and items are combined, and the latent variables of users and items are optimized based on the vectorized features of CNNs and ratings. Extensive experiments show that our model predicts outcome more accurately than do other recommender models.