Better null models for assessing predictive accuracy of disease models

Abstract

Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R2) when evaluating the predictive ability of a model, which is often misleading in conveying the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) and Always Absent null models were the strongest overall, and the majority of null models significantly outperformed the grand mean. Somewhat surprisingly, the length of the training timeseries did not strongly affect the performance of most null models. We argue that a combination of null models is needed to evaluate the forecasting performance of predictive models and the grand mean is the lowest bar.