According to the phenomenon that gear transmission system of shearer cutting unit produced vibrations and noise in operation, a nonlinear dynamics model for gear transmission system of shearer cutting unit was established with the consideration of mesh stiffness, mesh damping, and comprehensive errors. Differential equations were solved by employing variable step size RungeKutta integration method. The influences of mesh stiffnesses,damping ratios and excitation frequencies on gear transmission system were studied through analyzing phase plane and Poincare section. The results show that in a certain range,displacement response of sun gear changes from single periodic motion to multiperiodic motion and then into chaotic motion with the decreasing of damping ratio. Displacement response of sun gear also gradually changes from periodic motion to chaotic motion with the increasing of mesh stiffness. Displacement response of sun gear changes from periodic response to chaotic response and then into quasiperiod response with the increasing of the excitation frequencies.