The error sources of a three-axis magnetometer are systematically analyzed, and a uniform error model specially for calibration is established. An in-suit three-axis magnetometer calibration method for micro and mini unmanned aerial vehicle (UAV)based on the two-step estimation algorithm and circle restricted misalignment calibration algorithm is proposed. Taking into full consideration the inclination and declination of the earth’s magnetic vector, the proposed in-suit data sampling strategy can obtain excellent sphere coverage by using the fewest rotations. The simulation results indicate that in spite of all error sources existing, including strong hard iron and soft iron disturbances, the standard deviation of the magnetic heading error obtained by the calibrated magnetic data using the in-suit three-axis magnetometer calibration method is less than 1.2°, which is a mere 8% of the result obtained by using the usual industrial maximum and minimum calibration method. The precision is improved by 92%. The calibration result based on the real measurement data from a three-axis magnetometer also validates the effectiveness of the in-suit three-axis magnetometer calibration method.