More and more hyper-spectral satellites will be used to estimate total suspended matter (TSM) in waters instead of multi-spectral satellites, such as China's Gaofen-5 and Zhuhai-1. Although they have not been widely used because of the consistency of sampling and image time. Hence, the study based on measured hyper-spectroscopy is important for applying to hyper-spectral satellites. Fractional-order derivatives (FODs) considers more detailed spectral information, and it is a better spectral preprocessing method than conventional integer-order derivatives. The application and analysis of FODs for spectra in waters is rare. If FOD is successfully applied to estimate TSM, the TSM mapping with FOD using hyper-spectral satellites will be meaningful. Based on these points, this study aimed to apply FOD to predict TSM and to prove the prediction feasibility of FOD in waters. Different prediction models and eight FOD transformation processes with increment of 0.25 per step for 392 spectral reflectance data from China were used and compared. The prediction models include the optimum models of the single wavelength, ratio index, difference index and TSM index at each FOD order, and the random forest (RF) model with all wavelengths was also used. Discrete wavelet transform (DWT) was used to reduce noise and improve the model accuracy after using FOD. Our results achieved the followings First, FOD enhanced spectral characteristics at 500-600 nm and 800 nm that were affected by TSM. Second, the correlation between TSM and FOD spectra was enhanced (e.g., the correlation coefficients of 19 wavelengths (789-807 nm) of 0.75 order were higher than 0.8 but the original spectra were not). Third, FOD improved the performance of different prediction models, and the RF model from 0.5-order to 1.25-order derivative spectra all led good results (). Fourth, DWT can reduce the noise and improve the performance, and FOD-DWT model of 1.25-order led the R-2 of 0.84, RMSE of 16.30 and MAPE of 78.62 in validation. Overall, our results suggest that FOD can improve the prediction performance for most models, and the optimum order of some models is not integer. Our results also provide a reference for predicting other water quality parameters and mapping these parameters using hyper spectral satellites. The accurate estimation of TSM is helpful for protecting ecological and social environments.