The ab initio molecular dynamics (AIMD) based on finite-temperature Kohn-Sham density functional theory (FT-KS-DFT) is a reliable and valuable tool for obtaining quantitative information on warm dense matter (WDM). The requirement to reach to the thermodynamic limit poses a computational challenge in FT-KS-DFT calculations since it scales cubically with system size and with the temperature. In order to address these difficulties, we developed stochastic FT-KS-DFT, a statistical theory in which the electron density is determined from stochastic orbitals in a trace formula, reducing the algorithmic complexity to linear with system size and inverse linear with temperature. The forces produced by stochastic DFT (sDFT) have a noisy component that is used within Langevin molecular dynamics to sample configurations of the system at a given temperature. In order to reduce force fluctuations we use energy windows (Chen et al, J. Chem. Phys. 151, 114116 (2019)). Here we study the efficacy of the energy windows technique and show it reduces the variance by a substantial factor without a significant additional cost, nearly independent of the temperature.