How are probability forecasts typically validated?
Forecasts are validated by a process called backtesting as well as a process called monitoring. In backtesting, data from the past are divided into a training period (prior data) and a testing period (posterior data). Forecasts are made using prior data to forecast events that occur during the posterior interval. The accuracy of these forecasts are then scored by a variety of statistical tests. Forecasts that achieve a pre-determined level of accuracy are considered to be validated at the observed confidence level. In monitoring, actual real time forecasts are computed, then actual events are observed. The results are scored using the same types of statistical analysis. Many researchers consider monitoring to be a higher level of validation than backtesting, since the “answer” is not known in advance. However, monitoring can take many years to determine the accuracy of a forecast method, whereas backtesting typically leads to an answer within days, weeks, or a few months at most.
- OH Community
- Web Apps
- About Us