We consider a cell-free massive multiple-input multiple-output (MIMO) system with multiple antennas on the users and access points (APs). In previous works, the downlink spectral efficiency (SE) has been evaluated using the hardening bound that requires no downlink pilots. This approach works well for single-antenna users. In this paper, we show that much higher SEs can be achieved if downlink pilots are sent when having multi-antenna users. The reason is that the effective channel matrix does not harden. We propose a pilot-based downlink estimation scheme, derive a new SE expression, and show numerically that it yields substantially higher performance when having correlated Rayleigh fading channels. In cases with multi-antenna users, the APs can either transmit the same or different data streams. The latter reduces the fronthaul signaling but comes with a SE loss. We propose precoding and combining schemes for these cases and consider whether channel knowledge is shared between the APs. Finally, we show numerically how the number of users, APs, and the number of antennas on users and APs affect the SE.