Being able to recognize emotions in human users is considered a highly desirable trait in Human-Robot Interaction (HRI) scenarios. However, most contemporary approaches rarely attempt to apply recognized emotional features in an active manner to modulate robot decision-making and dialogue for the benefit of the user. In this position paper, we propose a method of incorporating recognized emotions into a Reinforcement Learning (RL) based dialogue management module that adapts its dialogue responses in order to attempt to make cognitive training tasks, like the 2048 Puzzle Game, more enjoyable for the users.