The revolution in artificial intelligence (AI) brings up an enormous storage and data processing requirement. Large power consumption and hardware overhead have become the main challenges for building next-generation AI hardware. Therefore, it is imperative to look for a new architecture capable of circumventing these bottlenecks of conventional von Neumann architecture. Since the human brain is the most compact and energy-efficient intelligent device known, it was intuitive to attempt to build an architecture that could mimic our brain, and so the chase for neuromorphic computing began. While relentless research has been underway for years to minimize the power consumption in neuromorphic hardware, we are still a long way off from reaching the energy efficiency of the human brain. Besides, design complexity, process variation, etc. hinder the large-scale implementation of current neuromorphic platforms. Recently, the concept of implementing neuromorphic computing systems in cryogenic temperature has garnered immense attention. Several cryogenic devices can be engineered to work as neuromorphic primitives with ultra-low demand for power. Cryogenic electronics has therefore become a promising exploratory platform for an energy-efficient and bio-realistic neuromorphic system. Here we provide a comprehensive overview of the reported cryogenic neuromorphic hardware. We carefully classify the existing cryogenic neuromorphic hardware into different categories and draw a comparative analysis based on several performance metrics. Finally, we explore the future research prospects to circumvent the challenges associated with the current technologies.