A new approach named the relative mode transfer method (RMTM) is developed in this study to solve the vibration problem of continuous systems with piecewise-linear boundary conditions or other nonlinear boundary conditions which can be transformed into piecewise-linear boundary conditions based on the mode transfer principle. The nonlinear vibration of a cantilever with double stops on the tip is investigated. The transfer between the contact states and non-contact states is dealt with using the new method, and its validity is proved by contrasting the results of a case studied both by the new approach and Moon's method which was proposed in 1983. The problem represented by the case is studied through the bifurcation response of the cantilever tip, and a discussion of the effects of the coupling between modes, modal damping, and stop stiffness. It is found that different combinations of the parameters of stop stiffness, damping ratio, driving force, etc., can lead to period-one, period-n and chaotic vibration, and the area of complex nonlinear response in the range of the driving force is obtained.