In this paper, authentication for mobile radio frequency identification (RFID) systems with low-cost tags is studied. Firstly, a diagonal block key matrix (DBKM) encryption algorithm is proposed, which effectively expands the feasible domain of the key space. Subsequently, in order to enhance the security, a self updating encryption order (SUEO) algorithm is conceived. To further weaken the correlation between plaintext and ciphertext, a self updating modulus (SUM) algorithm is constructed. Based on the above three algorithms, a new joint DBKM-SUEO-SUM matrix encryption algorithm is established, which intends to enhance security without the need of additional storage for extra key matrices. Making full use of the advantages of the proposed joint algorithm, a two-way RFID authentication protocol named DBKM-SUEO-SUM-RFID is proposed for mobile RFID systems. In addition, the Burrows-Abadi-Needham (BAN) logic and security analysis indicate that the newly proposed DBKM-SUEO-SUM-RFID protocol can effectively resist various typical attacks, such as replay attacks and de-synchronization. Finally, numerical results demonstrate that the DBKM-SUEO-SUM algorithm can save at least 90.46\% of tag storage compared to traditional algorithms, and thus, is friendly to be employed with low-cost RFID tags.