Visibility into AI Agents

Increased delegation of commercial, scientific, governmental, and personal activities to AI agents -- systems capable of pursuing complex goals with limited supervision -- may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents.

Black-Box Access is Insufficient for Rigorous AI Audits

External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of system access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to its training and deployment information (e.g., methodology, code, documentation, hyperparameters, data, deployment details, findings from internal evaluations) allows for auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.

SkyWatch: A Passive Multistatic Radar Network for the Measurement of Object Position and Velocity

(Abridged) Quantitative three-dimensional (3D) position and velocity estimates obtained by passive radar will assist the Galileo Project in the detection and classification of aerial objects by providing critical measurements of range, location, and kinematics. These parameters will be combined with those derived from the Project{\textquoteright}s suite of electromagnetic sensors and used to separate known aerial objects from those exhibiting anomalous kinematics. SkyWatch, a passive multistatic radar system based on commercial broadcast FM radio transmitters of opportunity, is a network of receivers spaced at geographical scales that enables estimation of the 3D position and velocity time series of objects at altitudes up to 80km, horizontal distances up to 150km, and at velocities to {\textpm}2{\textpm}2km/s ({\textpm}6{\textpm}6Mach). The receivers are designed to collect useful data in a variety of environments varying by terrain, transmitter power, relative transmitter distance, adjacent channel strength, etc. In some cases, the direct signal from the transmitter may be large enough to be used as the reference with which the echoes are correlated. In other cases, the direct signal may be weak or absent, in which case a reference is communicated to the receiver from another network node via the internet for echo correlation. Various techniques are discussed specific to the two modes of operation and a hybrid mode. Delay and Doppler data are sent via internet to a central server where triangulation is used to deduce time series of 3D positions and velocities. A multiple receiver (multistatic) radar experiment is undergoing Phase 1 testing, with several receivers placed at various distances around the Harvard{\textendash}Smithsonian Center for Astrophysics (CfA), to validate full 3D position and velocity recovery.

The Scientific Investigation of Unidentified Aerial Phenomena (UAP) Using Multimodal Ground-Based Observatories

(Abridged) Unidentified Aerial Phenomena (UAP) have resisted explanation and have received little formal scientific attention for 75 years. A primary objective of the Galileo Project is to build an integrated software and instrumentation system designed to conduct a multimodal census of aerial phenomena and to recognize anomalies. Here we present key motivations for the study of UAP and address historical objections to this research. We describe an approach for highlighting outlier events in the high-dimensional parameter space of our census measurements. We provide a detailed roadmap for deciding measurement requirements, as well as a science traceability matrix (STM) for connecting sought-after physical parameters to observables and instrument requirements. We also discuss potential strategies for deciding where to locate instruments for development, testing, and final deployment. Our instrument package is multimodal and multispectral, consisting of (1) wide-field cameras in multiple bands for targeting and tracking of aerial objects and deriving their positions and kinematics using triangulation; (2) narrow-field instruments including cameras for characterizing morphology, spectra, polarimetry, and photometry; (3) passive multistatic arrays of antennas and receivers for radar-derived range and kinematics; (4) radio spectrum analyzers to measure radio and microwave emissions; (5) microphones for sampling acoustic emissions in the infrasonic through ultrasonic frequency bands; and (6) environmental sensors for characterizing ambient conditions (temperature, pressure, humidity, and wind velocity), as well as quasistatic electric and magnetic fields, and energetic particles. The use of multispectral instruments and multiple sensor modalities will help to ensure that artifacts are recognized and that true detections are corroborated and verifiable.