GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learning

Pre-training over mixtured multi-task, multi-domain, and multi-modal data remains an open challenge in vision perception pre-training. In this paper, we propose GPPF, a General Perception Pre-training Framework, that pre-trains a task-level dynamic network, which is composed by knowledge "legos" in each layers, on labeled multi-task and multi-domain datasets. By inspecting humans' innate ability to learn in complex environment, we recognize and transfer three critical elements to deep networks: (1) simultaneous exposure to diverse cross-task and cross-domain information in each batch. (2) partitioned knowledge storage in separate lego units driven by knowledge sharing. (3) sparse activation of a subset of lego units for both pre-training and downstream tasks. Noteworthy, the joint training of disparate vision tasks is non-trivial due to their differences in input shapes, loss functions, output formats, data distributions, etc. Therefore, we innovatively develop a plug-and-play multi-task training algorithm, which supports Single Iteration Multiple Tasks (SIMT) concurrently training. SIMT lays the foundation of pre-training with large-scale multi-task multi-domain datasets and is proved essential for stable training in our GPPF experiments. Excitingly, the exhaustive experiments show that, our GPPF-R50 model achieves significant improvements of 2.5-5.8 over a strong baseline of the 8 pre-training tasks in GPPF-15M and harvests a range of SOTAs over the 22 downstream tasks with similar computation budgets. We also validate the generalization ability of GPPF to SOTA vision transformers with consistent improvements. These solid experimental results fully prove the effective knowledge learning, storing, sharing, and transfer provided by our novel GPPF framework.

Distributed Communication-aware Motion Planning for Multi-agent Systems from STL and SpaTeL Specifications

In future intelligent transportation systems, networked vehicles coordinate with each other to achieve safe operations based on an assumption that communications among vehicles and infrastructure are reliable. Traditional methods usually deal with the design of control systems and communication networks in a separated manner. However, control and communication systems are tightly coupled as the motions of vehicles will affect the overall communication quality. Hence, we are motivated to study the co-design of both control and communication systems. In particular, we propose a control theoretical framework for distributed motion planning for multi-agent systems which satisfies complex and high-level spatial and temporal specifications while accounting for communication quality at the same time. Towards this end, desired motion specifications and communication performances are formulated as signal temporal logic (STL) and spatial-temporal logic (SpaTeL) formulas, respectively. The specifications are encoded as constraints on system and environment state variables of mixed integer linear programs (MILP), and upon which control strategies satisfying both STL and SpaTeL specifications are generated for each agent by employing a distributed model predictive control (MPC) framework. Effectiveness of the proposed framework is validated by a simulation of distributed communication-aware motion planning for multi-agent systems.

Combined Top-Down and Bottom-Up Approaches to Performance-guaranteed Integrated Task and Motion Planning of Cooperative Multi-agent Systems

We propose a hierarchical design framework to automatically synthesize coordination schemes and control policies for cooperative multi-agent systems to fulfill formal performance requirements, by associating a bottom-up reactive motion controller with a top-down mission plan. On one hand, starting from a global mission that is specified as a regular language over all the agents' mission capabilities, a mission planning layer sits on the top of the proposed framework, decomposing the global mission into local tasks that are in consistency with each agent's individual capabilities, and compositionally justifying whether the achievement of local tasks implies the satisfaction of the global mission via an assume-guarantee paradigm. On the other hand, bottom-up motion plans associated with each agent are synthesized corresponding to the obtained local missions by composing basic motion primitives, which are verified safe by differential dynamic logic (d$\mathcal{L}$), through a Satisfiability Modulo Theories (SMT) solver that searches feasible solutions in face of constraints imposed by local task requirements and the environment description. It is shown that the proposed framework can handle dynamical environments as the motion primitives possess reactive features, making the motion plans adaptive to local environmental changes. Furthermore, on-line mission reconfiguration can be triggered by the motion planning layer once no feasible solutions can be found through the SMT solver. The effectiveness of the overall design framework is validated by an automated warehouse case study.