The open radio access network (O-RAN) architecture offers a cost-effective and scalable solution for internet service providers to optimize their networks using machine learning algorithms. The architecture's open interfaces enable network function virtualization, with the O-RAN serving as the primary communication device for users. However, the limited frequency resources and information explosion make it difficult to achieve an optimal network experience without effective traffic control or resource allocation. To address this, we consider mobility-aware load balancing to evenly distribute loads across the network, preventing network congestion and user outages caused by excessive load concentration on open radio unit (O-RU) governed by a single open distributed unit (O-DU). We have proposed a multi-agent multi-armed bandit for load balancing and resource allocation (mmLBRA) scheme, designed to both achieve load balancing and improve the effective sum-rate performance of the O-RAN network. We also present the mmLBRA-LB and mmLBRA-RA sub-schemes that can operate independently in non-realtime RAN intelligent controller (Non-RT RIC) and near-RT RIC, respectively, providing a solution with moderate loads and high-rate in O-RUs. Simulation results show that the proposed mmLBRA scheme significantly increases the effective network sum-rate while achieving better load balancing across O-RUs compared to rule-based and other existing heuristic methods in open literature.