Recently, there has been significant interest in the ability to navigate without GPS using the magnetic anomaly field of the Earth (magnav). One of the key technical bottlenecks to achieving magnav is obtaining an accurate magnetic sensor calibration, taking into account own-ship and sensor effects. The Tolles-Lawson magnetic calibration method was developed when airborne magnetic survey aircraft were first employed 80 years ago. In this paper, we present a magnetic calibration algorithm based on a factor graph optimization using inertial measurements as well as inputs from both a vector and scalar magnetometer. The factor graph is well suited for combining multiple sensor inputs and allows accurate calibrations in the presence of large permanent moments as well as estimating a time-varying external magnetic field during the calibration, two problems that are difficult to solve with previous approaches. The ability to accurately calibrate a magnetic sensor in flight (i.e. with the presence of a large platform field and a varying Earth field due to movement) will allow greater flexibility in sensor mounting locations for magnetic anomaly navigation.