As a user-friendly and straightforward solution for robot trajectory generation, imitation learning has been viewed as a vital direction in the context of robot skill learning. In contrast to unconstrained imitation learning which ignores possible internal and external constraints arising from environments and robot kinematics/dynamics, recent works on constrained imitation learning allow for transferring human skills to unstructured scenarios, further enlarging the application domain of imitation learning. While various constraints have been studied, e.g., joint limits, obstacle avoidance and plane constraints, the problem of nonlinear hard constraints has not been well-addressed. In this paper, we propose extended kernelized movement primitives (EKMP) to cope with most of the key problems in imitation learning, including nonlinear hard constraints. Specifically, EKMP is capable of learning the probabilistic features of multiple demonstrations, adapting the learned skills towards arbitrary desired points in terms of joint position and velocity, avoiding obstacles at the level of robot links, as well as satisfying arbitrary linear and nonlinear, equality and inequality hard constraints. Besides, the connections between EKMP and state-of-the-art motion planning approaches are discussed. Several evaluations including the planning of joint trajectories for a 7-DoF robotic arm are provided to verify the effectiveness of our framework.