Divide and Fuse: Body Part Mesh Recovery from Partially Visible Human Images

We introduce a novel bottom-up approach for human body mesh reconstruction, specifically designed to address the challenges posed by partial visibility and occlusion in input images. Traditional top-down methods, relying on whole-body parametric models like SMPL, falter when only a small part of the human is visible, as they require visibility of most of the human body for accurate mesh reconstruction. To overcome this limitation, our method employs a "Divide and Fuse (D&F)" strategy, reconstructing human body parts independently before fusing them, thereby ensuring robustness against occlusions. We design Human Part Parametric Models (HPPM) that independently reconstruct the mesh from a few shape and global-location parameters, without inter-part dependency. A specially designed fusion module then seamlessly integrates the reconstructed parts, even when only a few are visible. We harness a large volume of ground-truth SMPL data to train our parametric mesh models. To facilitate the training and evaluation of our method, we have established benchmark datasets featuring images of partially visible humans with HPPM annotations. Our experiments, conducted on these benchmark datasets, demonstrate the effectiveness of our D&F method, particularly in scenarios with substantial invisibility, where traditional approaches struggle to maintain reconstruction quality.

Autonomous programmable microscopic electronic lablets optimized with digital control

Lablets are autonomous microscopic particles with programmable CMOS electronics that can control electrokinetic phenomena and electrochemical reactions in solution via actuator and sensor microelectrodes. In this paper, we describe the design and fabrication of optimized singulated lablets (CMOS3) with dimensions 140x140x50 micrometers carrying an integrated coplanar encapsulated supercapacitor as a rechargeable power supply. The lablets are designed to allow docking to one another or to a smart surface for interchange of energy, electronic information, and chemicals. The paper focusses on the digital and analog design of the lablets to allow significant programmable functionality in a microscopic footprint, including the control of autonomous actuation and sensing up to the level of being able to support a complete lablet self-reproduction life cycle, although experimentally this remains to be proven. The potential of lablets in autonomous sensing and control and for evolutionary experimentation are discussed.

Design and fabrication of autonomous electronic lablets for chemical control

Lablets are autonomous microscopic particles with programmable CMOS electronics that canvcontrol electrokinetic phenomena and electrochemical reactions in solution via actuator and sensor microelectrodes. The lablets are designed to be rechargeable using an integrated supercapacitor, and to allow docking to one another or to a smart surface for interchange of energy, electronic information and chemicals. In this paper, we describe the design and fabrication of singulated lablets (CMOS2) at the scale of 100 by 200 {\mu}m, with the supercap adjacent to the functional lablet and occupying half the space. In other works, we have characterized the supercap and described the electronic design and proven functionality using arrays of these lablets. Here we present fabrication details for integrating functional coatings and the supercap and demonstrate electronic functionality of the lablets following singulation.

Spintronic Implementation of UNet for Image Segmentation

Image segmentation plays a crucial role in computer vision applications like self-driving cars, satellite imagery analysis, and medical diagnosis. Implementing these complex deep neural networks on conventional hardware is highly inefficient. In this work, we propose hardware implementation of UNet for segmentation tasks, using spintronic devices. Our approach involves designing hardware for convolution, deconvolution, ReLU, and max pooling layers of the UNet architecture. We demonstrate the synaptic behavior of the domain wall MTJ, and design convolution and deconvolution layers using the domain wall-based crossbar array. We utilize the orthogonal current injected MTJ with its continuous resistance change and showcase the ReLU and max pooling functions. We employ a hybrid simulation setup by coupling micromagnetic simulation, non-equilibrium Green's function, Landau-Lifshitz-Gilbert-Slonczewski equations, and circuit simulation with Python programming to incorporate the diverse physics of spin-transport, magnetization dynamics, and CMOS elements in our proposed designs. We evaluate our UNet design on the CamVid dataset and achieve segmentation accuracies that are comparable to software implementation. During training, our design consumes 43.59pJ of energy for synaptic weight updates.

Re-DiffiNet: Modeling discrepancies loss in tumor segmentation using diffusion models

Identification of tumor margins is essential for surgical decision-making for glioblastoma patients and provides reliable assistance for neurosurgeons. Despite improvements in deep learning architectures for tumor segmentation over the years, creating a fully autonomous system suitable for clinical floors remains a formidable challenge because the model predictions have not yet reached the desired level of accuracy and generalizability for clinical applications. Generative modeling techniques have seen significant improvements in recent times. Specifically, Generative Adversarial Networks (GANs) and Denoising-diffusion-based models (DDPMs) have been used to generate higher-quality images with fewer artifacts and finer attributes. In this work, we introduce a framework called Re-Diffinet for modeling the discrepancy between the outputs of a segmentation model like U-Net and the ground truth, using DDPMs. By explicitly modeling the discrepancy, the results show an average improvement of 0.55\% in the Dice score and 16.28\% in HD95 from cross-validation over 5-folds, compared to the state-of-the-art U-Net segmentation model.

PBADet: A One-Stage Anchor-Free Approach for Part-Body Association

The detection of human parts (e.g., hands, face) and their correct association with individuals is an essential task, e.g., for ubiquitous human-machine interfaces and action recognition. Traditional methods often employ multi-stage processes, rely on cumbersome anchor-based systems, or do not scale well to larger part sets. This paper presents PBADet, a novel one-stage, anchor-free approach for part-body association detection. Building upon the anchor-free object representation across multi-scale feature maps, we introduce a singular part-to-body center offset that effectively encapsulates the relationship between parts and their parent bodies. Our design is inherently versatile and capable of managing multiple parts-to-body associations without compromising on detection accuracy or robustness. Comprehensive experiments on various datasets underscore the efficacy of our approach, which not only outperforms existing state-of-the-art techniques but also offers a more streamlined and efficient solution to the part-body association challenge.

An Optimization Framework for Processing and Transfer Learning for the Brain Tumor Segmentation

Tumor segmentation from multi-modal brain MRI images is a challenging task due to the limited samples, high variance in shapes and uneven distribution of tumor morphology. The performance of automated medical image segmentation has been significant improvement by the recent advances in deep learning. However, the model predictions have not yet reached the desired level for clinical use in terms of accuracy and generalizability. In order to address the distinct problems presented in Challenges 1, 2, and 3 of BraTS 2023, we have constructed an optimization framework based on a 3D U-Net model for brain tumor segmentation. This framework incorporates a range of techniques, including various pre-processing and post-processing techniques, and transfer learning. On the validation datasets, this multi-modality brain tumor segmentation framework achieves an average lesion-wise Dice score of 0.79, 0.72, 0.74 on Challenges 1, 2, 3 respectively.

Gemini: A Family of Highly Capable Multimodal Models

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of Gemini models in cross-modal reasoning and language understanding will enable a wide variety of use cases and we discuss our approach toward deploying them responsibly to users.

Reinforced Self-Training (ReST) for Language Modeling

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Robust Decision-Focused Learning for Reward Transfer

Decision-focused (DF) model-based reinforcement learning has recently been introduced as a powerful algorithm which can focus on learning the MDP dynamics which are most relevant for obtaining high rewards. While this approach increases the performance of agents by focusing the learning towards optimizing for the reward directly, it does so by learning less accurate dynamics (from a MLE standpoint), and may thus be brittle to changes in the reward function. In this work, we develop the robust decision-focused (RDF) algorithm which leverages the non-identifiability of DF solutions to learn models which maximize expected returns while simultaneously learning models which are robust to changes in the reward function. We demonstrate on a variety of toy example and healthcare simulators that RDF significantly increases the robustness of DF to changes in the reward function, without decreasing the overall return the agent obtains.