This paper studies the performance of physical layer security (PLS) in a multi-user hybrid heterogeneous visible light communication (VLC) and radio frequency (RF) wireless communication system with simultaneous lightwave information and power transfer (SLIPT). In the considered system, VLC is used for downlink (DL) while RF is employed for uplink (UL) transmission. In addition, to support multiple users, time division multiple access (TDMA) is assumed for both DL and UL channels. In the DL, each user receives information during its allocated time slot of the TDMA frame and harvests energy from the received signal outside the time slot. The harvested energy is then used for transmitting the signal over the UL channel, which is subject to eavesdropping by an unauthorized user. Therefore, PLS is employed to protect the confidentiality of the UL information. Then, an optimization problem is formulated to solve the optimal DL and UL time slots that maximize the PLS performance given a target sum rate of the DL. We show that the problem can be cast as a difference of convex functions (DC) program, which can be solved efficiently using the DC algorithm (DCA).