The gear and rack jacking system is a crucial load transmission device in the Gear and Rack Drilling Rig (GRDR). Vibration induced by stiffness excitation during meshing between the gear and rack is a significant factor that affects the transmission performance. This paper focuses on the jacking mechanism of the GRDR and proposes a gear position design method based on time-varying meshing stiffness. A time-varying mesh stiffness model is established, considering tooth profile by the slice-iteration method. With the White Shark Optimizer (WSO) global search optimization algorithm, the gear position conditions are explored based on the fluctuation of stiffness. The load-bearing performance and dynamic characteristics of the mechanism are effectively improved. Dynamic analysis is conducted before and after the optimization of the scheme, and the improvements of the gear position optimization to the displacement of the transmission mechanism are verified through the meshing process of the rack and gear. The results show that the proposed optimization design may reduce the fluctuation by 89.64% and the maximum displacement by 9% compared to before. The proposed design method can effectively improve the motion performance, which is significant for optimizing the jacking system of the GRDR.