With the large number of deep space (DS) missions anticipated by the end of this decade, reliable-high capacity DS communication systems are needed more than ever. Nevertheless, existing DS communication technologies are far from meeting such a goal. Improving current systems does not only demand a system engineering leadership, but more crucially a well investigation in the potentials of emerging technologies in overcoming the challenges of the unique-ultra long DS communication channel. This project starts with a survey that highlights current technologies, trends, and advancements, investigates potentials, and identify challenges, and in essence, provide perspectives and propose solutions. It focuses on free-space optical (FSO) communication as a potential technology that can overcome the shortcomings of current radio frequency (RF)-based communication systems. To the best of our knowledge, in addition, it provides for the very first time a thoughtful discussion about implementing orbital angular momentum (OAM) for DS, identifies major related challenges, and proposes some novel solutions. Furthermore, we discuss DS modulations and coding schemes, as well as emerging receiver technologies and communication protocols. We also elaborate on how all of these technologies guarantee reliability, improve efficiency, offer capacity boosts, and enhance security in the unique DS environment. In addition to that, an extended study on the design and performance analysis of deep space optical communication (DSOC) is included, with the most suggested modulation for such a link being pulse position modulation (PPM) and a focus on the communication between Earth and the planet Mars, which is an important destination for space exploration.