Being able to track objects that surround us is key for planning actions in dynamic environments. However, rigorous cognitive models for tracking of one or more objects are currently lacking. In this study, we asked human subjects to judge the time to contact (TTC) a finish line for one or two objects that became invisible shortly after moving. We showed that the pattern of subject responses had an error variance best explained by an inverse Gaussian distribution and consistent with the output of a biased drift-diffusion model. Furthermore, we demonstrated that the pattern of errors made when tracking two objects showed a level of dependence that was consistent with subjects using a single decision variable for reporting the TTC for two objects. This finding reveals a serious limitation in the capacity for tracking multiple objects resulting in error propagation between objects. Apart from explaining our own data, our approach helps interpret previous findings such as asymmetric interference when tracking multiple objects.