Lower leg prostheses could improve the lives of amputees by increasing comfort and reducing energy to locomote, but currently the control methods make it difficult to modulate behaviors based upon the human's experience. This paper describes the first steps toward learning complex controllers for dynamic robotic assistive devices. We provide the first example of behavioral cloning to control a powered transfemoral knee and ankle prostheses using a Gated Recurrent Unit (GRU) based recurrent neural network (RNN) running on a custom hardware accelerator that exploits temporal sparsity. The RNN is trained on data collected from the original prosthesis controller. The RNN inference is realized by a novel EdgeDRNN accelerator in real-time. Experimental results show that the RNN can model the dynamic system with impacts and replace the nominal PD controller to realize end-to-end control of the AMPRO3 prosthetic leg walking on flat ground and unforeseen slopes with comparable tracking accuracy. EdgeRNN computes the RNN about 240 times faster than real time, opening the possibility of more complex future optimizations. Implementing an RNN on this real-time dynamic system with impacts sets the ground work to incorporate other learned elements of the human-prosthesis system into prosthesis control.