Deep Learning (DL) algorithm is the state-of-the-art algorithm of many computer science fields and applied on many intelligent mobile applications. In this paper, we propose a system called CoINF, a practical, adaptive, and flexible deep learning framework that enables cooperative inference between wearable devices (e.g., smartwatches and smart glasses) and handhelds. Our framework accelerates the processing and saves the energy consumption of generic deep learning models inference on wearables via judiciously offloading the workloads to paired handhelds at fine granularity in considering of the system environment, the application requirements, and user preference. Deployed as a user-space library, CoINF offers developer-friendly APIs that are as simple as those in traditional DL libraries such as TensorFlow, with all complicated offloading details hidden. We have implemented a prototype of CoINF on Android OS, and used real deep learning models to evaluate its performance on commercial off-the-shelf smartphone and smartwatches. The experimental results show that our framework can achieve substantial execution speedup and energy saving compared to wearable-only and handheld-only strategies.