The field of autonomous physical science - where machine learning guides and learns from experiments in a closed-loop - is rapidly growing in importance. Autonomous systems allow scientists to fail smarter, learn faster, and spend less resources in their studies. The field promises improved performance for various facilities such as labs, research and development pipelines, and warehouses. As autonomous systems grow in number, capability, and complexity, a new challenge arises - how will these systems work together across large facilities? We explore one solution to this question - a multi-agent framework. We demonstrate a framework with 1) a simulated facility with realistic resource limits such as equipment use limits, 2) machine learning agents with diverse learning capabilities and goals, control over lab instruments, and the ability to run research campaigns, and 3) a network over which these agents can share knowledge and work together to achieve individual or collective goals. The framework is dubbed the MULTI-agent auTonomous fAcilities - a Scalable frameworK aka MULTITASK. MULTITASK allows facility-wide simulations including agent-instrument and agent-agent interactions. Framework modularity allows real-world autonomous spaces to come on-line in phases, with simulated instruments gradually replaced by real-world instruments. Here we demonstrate the framework with a real-world materials science challenge of materials exploration and optimization in a simulated materials lab. We hope the framework opens new areas of research in agent-based facility control scenarios such as agent-to-agent markets and economies, management and decision-making structures, communication and data-sharing structures, and optimization strategies for agents and facilities including those based on game theory.