Multi: Multimodal Understanding Leaderboard with Text and Images

Rapid progress in multimodal large language models (MLLMs) highlights the need to introduce challenging yet realistic benchmarks to the academic community. Existing benchmarks primarily focus on simple natural image understanding, but Multi emerges as a cutting-edge benchmark for MLLMs, offering a comprehensive dataset for evaluating MLLMs against understanding complex figures and tables, and scientific questions. This benchmark, reflecting current realistic examination styles, provides multimodal inputs and requires responses that are either precise or open-ended, similar to real-life school tests. It challenges MLLMs with a variety of tasks, ranging from formula derivation to image detail analysis, and cross-modality reasoning. Multi includes over 18,000 questions, with a focus on science-based QA in diverse formats. We also introduce Multi-Elite, a 500-question subset for testing the extremities of MLLMs, and Multi-Extend, which enhances In-Context Learning research with more than 4,500 knowledge pieces. Our evaluation indicates significant potential for MLLM advancement, with GPT-4V achieving a 63.7% accuracy rate on Multi, in contrast to other MLLMs scoring between 31.3% and 53.7%. Multi serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.

Supervisory Control of Probabilistic Discrete Event Systems under Partial Observation

The supervisory control of probabilistic discrete event systems (PDESs) is investigated under the assumptions that the supervisory controller (supervisor) is probabilistic and has a partial observation. The probabilistic P-supervisor is defined, which specifies a probability distribution on the control patterns for each observation. The notions of the probabilistic controllability and observability are proposed and demonstrated to be a necessary and sufficient conditions for the existence of the probabilistic P-supervisors. Moreover, the polynomial verification algorithms for the probabilistic controllability and observability are put forward. In addition, the infimal probabilistic controllable and observable superlanguage is introduced and computed as the solution of the optimal control problem of PDESs. Several examples are presented to illustrate the results obtained.