We study the stream-based online active learning in a contextual multi-armed bandit framework. In this framework, the reward depends on both the arm and the context. In a stream-based active learning setting, obtaining the ground truth of the reward is costly, and the conventional contextual multi-armed bandit algorithm fails to achieve a sublinear regret due to this cost. Hence, the algorithm needs to determine whether or not to request the ground truth of the reward at current time slot. In our framework, we consider a stream-based active learning setting in which a query request for the ground truth is sent to the annotator, together with some prior information of the ground truth. Depending on the accuracy of the prior information, the query cost varies. Our algorithm mainly carries out two operations: the refinement of the context and arm spaces and the selection of actions. In our algorithm, the partitions of the context space and the arm space are maintained for a certain time slots, and then become finer as more information about the rewards accumulates. We use a strategic way to select the arms and to request the ground truth of the reward, aiming to maximize the total reward. We analytically show that the regret is sublinear and in the same order with that of the conventional contextual multi-armed bandit algorithms, where no query cost