In this paper, we detailedly describe our solution for the IEEE BigData Cup 2021: RL-based RecSys (Track 1: Item Combination Prediction). We first conduct an exploratory data analysis on the dataset and then utilize the findings to design our framework. Specifically, we use a two-headed transformer-based network to predict user feedback and unlocked sessions, along with the proposed session-aware reweighted loss, multi-tasking with click behavior prediction, and randomness-in-session augmentation. In the final private leaderboard on Kaggle, our method ranked 2nd with a categorization accuracy of 0.39224.