High-speed helicopters use technologies such as advancing blade， reduced rotor rotation speed， and auxiliary propeller to achieve high-speed flight. The coupled system formed by coaxial rigid dual-rotor， variable speed drive train system， and high-power output tail propeller brings new challenges to traditional helicopter torsional vibration analysis. Firstly， a new modeling strategy based on the transfer matrix method is innovatively proposed to address the problem of complex multi-mode coupling torsional vibration system of high-speed helicopters. Compared to the modeling strategy of conventional finite element method， the present method does not require the equivalent processing of the drive train system， nor does it require the derivation of the overall governing equations based on Hamilton’s principle. System governing equations can be directly obtained according to the topology structure of the system. In addition， a virtual geared branch element is innovatively introduced to decouple the topology of the drive train system into multiple independent chain systems， further significantly reducing the difficulty of modeling. Finally， the coupled torsional vibration dynamics of high-speed helicopters under different working conditions is studied based on the proposed method.