This paper proposes an integral sliding mode observer (ISMO)-based robust tracking control method for quadrotor systems taking into account the motor and propeller dynamics in the presence of system disturbances. First, a quadrotor system is modeled considering the motor and propeller dynamics and disturbances unlike the existing studies. Second, the disturbances of the quadrotor system are estimated and compensated for using the ISMO from the initial moment. Third, pseudo angle commands are designed for position tracking control of a quadrotor system using disturbance estimates. Fourth, to follow the desired states of the quadrotor system, robust controllers using sliding mode control are designed for tracking the altitude, angle, and thrust. Finally, the stability of the overall quadrotor system is analyzed mathematically, and experiments are conducted to validate the proposed method. The quadrotor system model used in the actual experiments is obtained by identifying the quadrotor system using the data acquired during the pre-flight, and the validities of the proposed quadrotor model and quadrotor control method are verified through actual experiments.