Reducing Diffusion Model Memorization with Higher Order Langevin Dynamics

Diffusion/score-based models have emerged as powerful generative models, capable of generating high-quality samples that mimic the training data distribution. However, it has been observed that they are prone to reproducing training samples-known as "memorization"-potentially violating copyright and privacy. In this paper, we study the effect of Higher-Order Langevin Dynamics (HOLD) on this phenomenon. HOLD diffusion processes introduce auxiliary variables; if the data variable is interpreted as "position," then the auxiliary variables can be interpreted as "velocity" and "acceleration," depending on the chosen order of the model. They were originally proposed based on the intuition that they regularize the trajectories of the data variable by implicitly imposing additional dynamical constraints. Our work provides, to our knowledge, the first theoretical characterization of the regularization effect of HOLD. Specifically, we show that in HOLD, the dynamics of the data variable are governed by a low-pass-filtered version of the learned score function, with smoothness increasing with the order of HOLD. We then analyze the optimal empirical score and the possibility of distribution collapse. Together, our results explain the mitigation of memorization as the model order increases. Finally, we present an empirical study on real-world data that supports our theory and highlights this distinct advantage of HOLD over standard diffusion in practice.

Defending Diffusion Models Against Membership Inference Attacks via Higher-Order Langevin Dynamics

Recent advances in generative artificial intelligence applications have raised new data security concerns. This paper focuses on defending diffusion models against membership inference attacks. This type of attack occurs when the attacker can determine if a certain data point was used to train the model. Although diffusion models are intrinsically more resistant to membership inference attacks than other generative models, they are still susceptible. The defense proposed here utilizes critically-damped higher-order Langevin dynamics, which introduces several auxiliary variables and a joint diffusion process along these variables. The idea is that the presence of auxiliary variables mixes external randomness that helps to corrupt sensitive input data earlier on in the diffusion process. This concept is theoretically investigated and validated on a toy dataset and a speech dataset using the Area Under the Receiver Operating Characteristic (AUROC) curves and the FID metric.

Fusion of Information in Multiple Particle Filtering in the Presence of Unknown Static Parameters

An important and often overlooked aspect of particle filtering methods is the estimation of unknown static parameters. A simple approach for addressing this problem is to augment the unknown static parameters as auxiliary states that are jointly estimated with the time-varying parameters of interest. This can be impractical, especially when the system of interest is high-dimensional. Multiple particle filtering (MPF) methods were introduced to try to overcome the curse of dimensionality by using a divide and conquer approach, where the vector of unknowns is partitioned into a set of subvectors, each estimated by a separate particle filter. Each particle filter weighs its own particles by using predictions and estimates communicated from the other filters. Currently, there is no principled way to implement MPF methods where the particle filters share unknown parameters or states. In this work, we propose a fusion strategy to allow for the sharing of unknown static parameters in the MPF setting. Specifically, we study the systems which are separable in states and observations. It is proved that optimal Bayesian fusion can be obtained for state-space models with non-interacting states and observations. Simulations are performed to show that MPF with fusion strategy can provide more accurate estimates within fewer time steps comparing to existing algorithms.