With increasing physical threats in recent years targeted at critical infrastructures, it is crucial to establish a reliable threat monitoring system integrating video surveillance and digital sensors based on cutting-edge technologies. A physical threat monitoring solution unifying the floorplan, cameras, and sensors for smart buildings has been set up in our study. Computer vision and deep learning models are used for video streams analysis. When a threat is detected by a rule engine based on the real-time analysis results combining with feedback from related digital sensors, an alert is sent to the Video Management System so that human operators can take further action. A physical threat monitoring system typically needs to address complex and even destructive incidents, such as fire, which is unrealistic to simulate in real life. Restrictions imposed during the Covid-19 pandemic and privacy concerns have added to the challenges. Our study utilises the Unreal Engine to simulate some typical suspicious and intrusion scenes with photorealistic qualities in the context of a virtual building. Add-on programs are implemented to transfer the video stream from virtual PTZ cameras to the Milestone Video Management System and enable users to control those cameras from the graphic client application. Virtual sensors such as fire alarms, temperature sensors and door access controls are implemented similarly, fulfilling the same programmatic VMS interface as real-life sensors. Thanks to this simulation system's extensibility and repeatability, we have consolidated this unified physical threat monitoring system and verified its effectiveness and user-friendliness. Both the simulated Unreal scenes and the software add-ons developed during this study are highly modulated and thereby are ready for reuse in future projects in this area.