This paper presents a novel approach to resource allocation in Open Radio Access Networks (O-RAN), leveraging a Generative AI technique with network slicing to address the diverse demands of 5G and 6G service types such as Enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), and Massive Machine-Type Communications (mMTC). Additionally, we provide a comprehensive analysis and comparison of machine learning (ML) techniques for resource allocation within O-RAN, evaluating their effectiveness in optimizing network performance. We introduce a diffusion-based reinforcement learning (Diffusion-RL) algorithm designed to optimize the allocation of physical resource blocks (PRBs) and power consumption, thereby maximizing weighted throughput and minimizing the delay for user equipment (UE). The Diffusion-RL model incorporates controlled noise and perturbations to explore optimal resource distribution while meeting each service type's Quality of Service (QoS) requirements. We evaluate the performance of our proposed method against several benchmarks, including an exhaustive search algorithm, deep Q-networks (DQN), and the Semi-Supervised Variational Autoencoder (SS-VAE). Comprehensive metrics, such as throughput and latency, are presented for each service type. Experimental results demonstrate that the Diffusion-based RL approach outperforms existing methods in efficiency, scalability, and robustness, offering a promising solution for resource allocation in dynamic and heterogeneous O-RAN environments with significant implications for future 6G networks.