Testing for ontological Error

“Science is rooted in the concept that a model can be tested against observations and rejected when necessary. However, the problem of model testing becomes formidable when we consider the probabilistic forecasting of natural systems. We show that testability is facilitated by the definition of an experimental concept, external to the model, that identifies collections of data, observed and not yet observed, that are judged to be exchangeable and can thus be associated with well-defined frequencies. We clarify several conceptual issues regarding the testing of probabilistic forecasting models, including the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, the interplay between Bayesian and frequentist methods, and the scientific pathway for capturing predictability.”
– Warner Marzocchi and Thomas Jordan

H0 =

“Testability derives from a null hypothesis, here called the “ontological hypothesis. H0 = the data-generating hazard curve ϕ ˆ  (the “true” hazard) is a sample from the extended experts’ distribution, ϕ ˆ ∼p(ϕ). If an observational test rejects this null hypothesis, then we can claim to have found an ontological error.”