Assessing the predictive performance of probabilistic caries risk assessment models: the importance of calibration

Probabilistic caries risk assessment models (P-CRA), such as the Cariogram, are promising tools to planning treatments in order to control and prevent caries. The usefulness of these models for informing patients and medical decision-making depends on 2 properties known as discrimination and calibration. Current common assessment of P-CRA models, however, ignores calibration, and this can be misleading. The aim of this paper was to provide tools for a proper assessment of calibration of the P-CRA models and improve calibration when lacking. A combination of standard calibration tools (calibration plot, calibration in-the-large, and calibration slope) and 3 novel measures of calibration (the Calibration Index and 2 related metrics, E50 and E90) are proposed to evaluate if a P-CRA model is well calibrated. Moreover, an approach was proposed and validated using data from a previous follow-up study performed on children evaluated by means of a reduced Cariogram model; Platt scaling and isotonic regression were applied showing a lack of calibration. The use of the Cariogram overestimates the actual risk of new caries for forecast probabilities <0.5 and underestimates the risk for forecast probabilities >0.6. Both Platt scaling and isotonic regression were able to significantly improve the calibration of the reduced Cariogram model, preserving its discrimination properties. The average specificity and sensitivity for both Platt scaling and isotonic regression using the cut-off point p= 0.5 were >83 and their sum well exceeded 160. The benefits of the proposed calibration methods are promising, but further research in this field is required.