In order to solve the equilibrium states of hypersonic vehicles with unstable modals, a hybrid optimization technique by the name of GA-SQP is developed. According to the performance index of the integral of time multiplied by the absolute value of error (ITAE), an improved genetic algorithm (GA) is used adopting chaos searching and elimination to calculate the equilibrium states, which are transformed into solutions of the minimum of the cost function. Furthermore, sequential quadratic programming (SQP) is introduced so as to accelerate the local optimum. The elevator deflection, throttle setting, and initial angle of attack are obtained by steps based on the Simulink dynamic model of the vehicle. Simulation results show that the GA-SQP algorithm, with desired steadiness, is able to determine the final solution accurately without being sensitive to the initial point. The algorithm provides a practical and effective method to solve the equilibrium of complex nonlinear systems.