To estimate the angles of multiple radiation sources with unknown numbers of signals, this paper presents an algorithm for angle estimation of the nested array based on Maximum Likelihood Estimation (MLE). Based on the nested array model, the maximum likelihood function and its gradient of the multiple signals intercepted by the nested array are derived. The angles of all radiation sources in the airspace are estimated by the steepest descent method. Using the method of multiple hypothesis testing, the maximum likelihood ratio and the threshold are compared to determine the active radiation source angle of the transmitted signal at a certain time and exclude false source angles. The problem of DOA estimation of multiple radiation sources with unknown number of signals by the nested array is thus solved. The simulation results show that under the conditions of unknown number of the radiation source, existing coherent signal, low Signal to Noise Ratio (SNR), and low sampling number, the proposed algorithm has better performance in angle estimation than traditional MUltiple SIgnal Classification (MUSIC) algorithm. The method of multiple hypothesis testing has more advantages than traditional source number estimation algorithms under the condition of low SNR and in the processing of coherent signals.