A digitally-assisted photonic analog domain self-interference cancellation (SIC) and frequency downconversion method is proposed for in-band full-duplex multiple-input multiple-output (MIMO) systems using the least square (LS) algorithm with adaptive order. The SIC and frequency downconversion are achieved in the optical domain via a dual-parallel Mach-Zehnder modulator (DP-MZM), while the downconverted signal is processed by the LS algorithm with adaptive order that is used to track the response of the multipath self-interference (SI) channel and reconstruct the reference signal for SIC. The proposed method can overcome the reconstruction difficulty of the multipath analog reference signal for SIC with high complexity in the MIMO scenario and can also solve the problem that the order of the reference reconstruction algorithm is not optimized when the wireless environment changes. An experiment is carried out to verify the concept. 30.2, 26.9, 23.5, 19.5, and 15.8 dB SIC depths are achieved when the SI signal has a carrier frequency of 10 GHz and baud rates of 0.1, 0.25, 0.5, 1, and 2 Gbaud, respectively. The convergence of the LS algorithm with adaptive order is also verified for different MIMO multipath SI signals.