Adaptive token processing has emerged as a promising approach for improving the efficiency of semantic communication systems. However, existing semantic communication frameworks largely overlook token-level multiple access and the impact of semantic interference among simultaneously transmitted semantic tokens. In this paper, we propose Adaptive Token Selection and Token-Domain Multiple Access (ATS-ToDMA), a novel cross-modal semantic communication framework that jointly performs semantic token selection, interference-aware scheduling, and semantic-aware power allocation. The proposed framework introduces a Semantic Signal-to-Interference-plus-Noise Ratio (SSINR) metric that captures the combined effects of channel impairments and semantic interference arising from token similarity. A transformer-based scheduler is developed to allocate selected semantic tokens across token-domain transmission slots while mitigating both intra-modal and cross-modal semantic interference. To characterize the behavior of the proposed system, analytical bounds on semantic interference and feasible token occupancy are derived, together with a closed-form approximation for semantic-aware power allocation. Simulation results demonstrate significant gains in semantic throughput and semantic decoding accuracy while reducing aggregate semantic interference and transmit power compared with OMA, Semantic NOMA, Random-TS, and Greedy ATS benchmarks.