In this paper, based on the port-Hamiltonian system theory and sliding mode control method, a passive sliding mode control method for distributed control of Unmanned Aerial Vehicles (UAVs) formation is designed. Firstly, a distributed formation error model was established based on the 3-DOF nonlinear UAVs formation system model and consensus estimation algorithm. Secondly, based on the port Hamiltonian system theory, the UAVs formation system was modeled as a port Hamiltonian system. Then, based on the distributed formation error model, the expected Hamiltonian energy function of the system is designed, and the sliding modulus is introduced into the expected Hamiltonian energy function through the energy integer technique, so that the passive sliding mode control law is designed, and the stability proof and robustness analysis of the passive sliding mode control method designed in this paper are carried out. Finally, the designed passive sliding mode control law is simulated and verified. The simulation results show that the passive sliding mode control law designed in this paper can make the state of the UAV in the swarm converge to the sliding mode surface and then converge along the sliding mode surface to the desired state, while exhibiting certain advantages in performance and disturbance rejection.