In this paper, we investigate the visual tracking problem for robotic systems without image-space velocity measurement, simultaneously taking into account the uncertainties of the camera model and the manipulator kinematics and dynamics. We propose a new image-space observer that exploits the image-space velocity information contained in the unknown kinematics, upon which, we design an adaptive controller without using the image-space velocity signal where the adaptations of the depth-rate-independent kinematic parameter and depth parameter are driven by both the image-space tracking errors and observation errors. The major superiority of the proposed observer-based adaptive controller lies in its simplicity and the separation of the handling of multiple uncertainties in visually servoed robotic systems, thus avoiding the overparametrization problem of the existing work. Using Lyapunov analysis, we demonstrate that the image-space tracking errors converge to zero asymptotically. The performance of the proposed adaptive control scheme is illustrated by a numerical simulation.