Early prediction of pathological complete response (pCR) following neoadjuvant chemotherapy (NAC) for breast cancer plays a critical role in surgical planning and optimizing treatment strategies. Recently, machine and deep-learning based methods were suggested for early pCR prediction from multi-parametric MRI (mp-MRI) data including dynamic contrast-enhanced MRI and diffusion-weighted MRI (DWI) with moderate success. We introduce PD-DWI, a physiologically decomposed DWI machine-learning model to predict pCR from DWI and clinical data. Our model first decomposes the raw DWI data into the various physiological cues that are influencing the DWI signal and then uses the decomposed data, in addition to clinical variables, as the input features of a radiomics-based XGBoost model. We demonstrated the added-value of our PD-DWI model over conventional machine-learning approaches for pCR prediction from mp-MRI data using the publicly available Breast Multi-parametric MRI for prediction of NAC Response (BMMR2) challenge. Our model substantially improves the area under the curve (AUC), compared to the current best result on the leaderboard (0.8849 vs. 0.8397) for the challenge test set. PD-DWI has the potential to improve prediction of pCR following NAC for breast cancer, reduce overall mp-MRI acquisition times and eliminate the need for contrast-agent injection.