Learning to Assemble Geometric Shapes

Assembling parts into an object is a combinatorial problem that arises in a variety of contexts in the real world and involves numerous applications in science and engineering. Previous related work tackles limited cases with identical unit parts or jigsaw-style parts of textured shapes, which greatly mitigate combinatorial challenges of the problem. In this work, we introduce the more challenging problem of shape assembly, which involves textureless fragments of arbitrary shapes with indistinctive junctions, and then propose a learning-based approach to solving it. We demonstrate the effectiveness on shape assembly tasks with various scenarios, including the ones with abnormal fragments (e.g., missing and distorted), the different number of fragments, and different rotation discretization.

Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning

Discovering a solution in a combinatorial space is prevalent in many real-world problems but it is also challenging due to diverse complex constraints and the vast number of possible combinations. To address such a problem, we introduce a novel formulation, combinatorial construction, which requires a building agent to assemble unit primitives (i.e., LEGO bricks) sequentially -- every connection between two bricks must follow a fixed rule, while no bricks mutually overlap. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit volumetric information to the agent. This problem requires a comprehensive understanding of partial information and long-term planning to append a brick sequentially, which leads us to employ reinforcement learning. The approach has to consider a variable-sized action space where a large number of invalid actions, which would cause overlap between bricks, exist. To resolve these issues, our model, dubbed Brick-by-Brick, adopts an action validity prediction network that efficiently filters invalid actions for an actor-critic network. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.

Combinatorial 3D Shape Generation via Sequential Assembly

3D shape generation has drawn attention in computer vision and machine learning since it opens an inspiring way to designing or creating new objects. Existing methods, however, do not reflect an important aspect of human generation processes in real life -- we often create a 3D shape by sequentially assembling geometric primitives into a combinatorial configuration. In this work, we propose a new 3D shape generation algorithm that aims to create such a combinatorial configuration from a set of volumetric primitives. To tackle the exponential growth of feasible combinations in terms of the number of primitives, we adopt sequential model-based optimization. Our method sequentially assembles primitives by exploiting and exploring adequate regions that are constrained by the current primitive placements. The evaluation function conveys global structure guidance for the assembling process to follow. Experimental results demonstrate that our method successfully generates combinatorial objects and simulates more realistic generation processes. We also introduce a new dataset for combinatorial 3D shape generation.