Crossing or Not-Crossing (C/NC) problem is important to autonomous vehicles (AVs) to safely interact with pedestrians. However, this problem setup ignores pedestrians walking along the direction of vehicles' movement (LONG). To enhance AVs' awareness of pedestrians behavior, we make the first step towards extending C/NC to C/NC/LONG problem and recognize them based on single body pose. In contrast, previous C/NC state classification work depend on multiple poses or contextual information. Our proposed shallow neural network classifier is able to recognize these three states within a very short time. We test our it on JAAD dataset and report average 81.23% accuracy. In order to further improve the classifier's performance, we introduce a computational-efficient method, action momentum optimizer (AMO), which correct prediction based on crossing behavior pattern. And our experiment shows that classifier perform at most 11.39% better on continuous pose test with the help of it. Furthermore, this model can cooperate with different sensors and algorithms that provide 2D pedestrian body pose so that it is able to work across multiple light and weather conditions. In addition, we have created extended annotations of pose for JAAD dataset, which will be publicly released soon