Visual feature clustering is one of the cost-effective approaches to segment objects in videos. However, the assumptions made for developing the existing algorithms prevent them from being used in situations like segmenting an unknown number of static and moving objects under heavy camera movements. This paper addresses the problem by introducing a clustering approach based on superpixels and short-term Histogram of Oriented Optical Flow (HOOF). Salient Dither Pattern Feature (SDPF) is used as the visual feature to track the flow and Simple Linear Iterative Clustering (SLIC) is used for obtaining the superpixels. This new clustering approach is based on merging superpixels by comparing short term local HOOF and a color cue to form high-level semantic segments. The new approach was compared with one of the latest feature clustering approaches based on K-Means in eight-dimensional space and the results revealed that the new approach is better by means of consistency, completeness, and spatial accuracy. Further, the new approach completely solved the problem of not knowing the number of objects in a scene.