In this paper, we present a way of enforcing safety and passivity properties of robot teleoperation systems, where a human operator interacts with a dynamical system modeling the robot. The approach does so in a holistic fashion, by combining safety and passivity constraints in a single optimization-based controller which effectively filters the desired control input before supplying it to the system. The result is a safety and passivity filter implemented as a convex quadratic program which can be solved efficiently and employed in an online fashion in many robotic teleoperation applications. Simulation results show the benefits of the approach developed in this paper applied to the human teleoperation of a second-order dynamical system.