Exploring Grokking: Experimental and Mechanistic Investigations

The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero training error and near-random test error. Subsequent prolonged training leads to a sharp transition from no generalization to perfect generalization. Our study comprises extensive experiments and an exploration of the research behind the mechanism of grokking. Through experiments, we gained insights into its behavior concerning the training data fraction, the model, and the optimization. The mechanism of grokking has been a subject of various viewpoints proposed by researchers, and we introduce some of these perspectives.

Imitation Learning for Autonomous Trajectory Learning of Robot Arms in Space

This work adds on to the on-going efforts to provide more autonomy to space robots. Here the concept of programming by demonstration or imitation learning is used for trajectory planning of manipulators mounted on small spacecraft. For greater autonomy in future space missions and minimal human intervention through ground control, a robot arm having 7-Degrees of Freedom (DoF) is envisaged for carrying out multiple tasks like debris removal, on-orbit servicing and assembly. Since actual hardware implementation of microgravity environment is extremely expensive, the demonstration data for trajectory learning is generated using a model predictive controller (MPC) in a physics based simulator. The data is then encoded compactly by Probabilistic Movement Primitives (ProMPs). This offline trajectory learning allows faster reproductions and also avoids any computationally expensive optimizations after deployment in a space environment. It is shown that the probabilistic distribution can be used to generate trajectories to previously unseen situations by conditioning the distribution. The motion of the robot (or manipulator) arm induces reaction forces on the spacecraft hub and hence its attitude changes prompting the Attitude Determination and Control System (ADCS) to take large corrective action that drains energy out of the system. By having a robot arm with redundant DoF helps in finding several possible trajectories from the same start to the same target. This allows the ProMP trajectory generator to sample out the trajectory which is obstacle free as well as having minimal attitudinal disturbances thereby reducing the load on ADCS.

AutoCross: Automatic Feature Crossing for Tabular Data in Real-World Applications

Feature crossing captures interactions among categorical features and is useful to enhance learning from tabular data in real-world businesses. In this paper, we present AutoCross, an automatic feature crossing tool provided by 4Paradigm to its customers, ranging from banks, hospitals, to Internet corporations. By performing beam search in a tree-structured space, AutoCross enables efficient generation of high-order cross features, which is not yet visited by existing works. Additionally, we propose successive mini-batch gradient descent and multi-granularity discretization to further improve efficiency and effectiveness, while ensuring simplicity so that no machine learning expertise or tedious hyper-parameter tuning is required. Furthermore, the algorithms are designed to reduce the computational, transmitting, and storage costs involved in distributed computing. Experimental results on both benchmark and real-world business datasets demonstrate the effectiveness and efficiency of AutoCross. It is shown that AutoCross can significantly enhance the performance of both linear and deep models.