Magnetometer and inertial sensors are widely used for orientation estimation. Magnetometer usage is often troublesome, as it is prone to be interfered by onboard or ambient magnetic disturbance. The onboard soft-iron material distorts not only the magnetic field, but the magnetometer sensor frame coordinate and the cross-sensor misalignment relative to inertial sensors. It is desirable to conveniently put magnetic and inertial sensors information in a common frame. Existing methods either split the problem into successive intrinsic and cross-sensor calibrations, or rely on stationary accelerometer measurements which is infeasible in dynamic conditions. This paper formulates the magnetometer calibration and alignment to inertial sensors as a state estimation problem, and collectively solves the magnetometer intrinsic and cross-sensor calibrations, as well as the gyroscope bias estimation. Sufficient conditions are derived for the problem to be globally observable, even when no accelerometer information is used at all. An extended Kalman filter is designed to implement the state estimation and comprehensive test data results show the superior performance of the proposed approach. It is immune to acceleration disturbance and applicable potentially in any dynamic conditions.