The natural way of obtaining different perspectives on any given topic is by conducting a debate, where participants argue for and against the topic. Here, we propose a novel debate framework for understanding the classifier's reasoning for making a particular prediction by modelling it as a multiplayer sequential zero-sum game. The players aim to maximise their utilities by adjusting their arguments with respect to other players' counterarguments. The contrastive nature of our framework encourages players to put forward diverse arguments, picking up the reasoning trails missed by their opponents. Thus, our framework answers the question: why did the classifier make a certain prediction?, by allowing players to argue for and against the classifier's decision. In the proposed setup, given the question and the classifier's latent knowledge, both agents take turns in proposing arguments to support or contradict the classifier's decision; arguments here correspond to the selection of specific features from the discretised latent space of the continuous classifier. By the end of the debate, we collect sets of supportive and manipulative features, serving as an explanation depicting the internal reasoning of the classifier. We demonstrate our Visual Debates on the geometric SHAPE and MNIST datasets for subjective validation, followed by the high-resolution AFHQ dataset. For further investigation, our framework is available at \url{https://github.com/koriavinash1/VisualDebates}.