Multiple Sclerosis (MS) progresses at an unpredictable rate, but predictions on the disease course in each patient would be extremely useful to tailor therapy to the individual needs. We explore different machine learning (ML) approaches to predict whether a patient will shift from the initial Relapsing-Remitting (RR) to the Secondary Progressive (SP) form of the disease, using only “real world” data available in clinical routine. The clinical records of 1624 outpatients (207 in the SP phase) attending the MS service of Sant'Andrea hospital, Rome, Italy, were used. Predictions at 180, 360 or 720 days from the last visit were obtained considering either the data of the last available visit (Visit-Oriented setting), comparing four classical ML methods (Random Forest, Support Vector Machine, K-Nearest Neighbours and AdaBoost) or the whole clinical history of each patient (History-Oriented setting), using a Recurrent Neural Network model, specifically designed for historical data. Missing values were handled by removing either all clinical records presenting at least one missing parameter (Feature-saving approach) or the 3 clinical parameters which contained missing values (Record-saving approach). The performances of the classifiers were rated using common indicators, such as Recall (or Sensitivity) and Precision (or Positive predictive value). In the visit-oriented setting, the Record-saving approach yielded Recall values from 70% to 100%, but low Precision (5% to 10%), which however increased to 50% when considering only predictions for which the model returned a probability above a given “confidence threshold”. For the History-oriented setting, both indicators increased as prediction time lengthened, reaching values of 67% (Recall) and 42% (Precision) at 720 days. We show how “real world” data can be effectively used to forecast the evolution of MS, leading to high Recall values and propose innovative approaches to improve Precision towards clinically useful values.

Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis / Seccia, R.; Gammelli, D.; Dominici, F.; Romano, S.; Landi, A. C.; Salvetti, M.; Tacchella, A.; Zaccaria, A.; Crisanti, A.; Grassi, F.; Palagi, L.. - In: PLOS ONE. - ISSN 1932-6203. - 15:3(2020). [10.1371/journal.pone.0230219]

Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis

Seccia R.
Primo
Writing – Original Draft Preparation
;
Gammelli D.
Formal Analysis
;
Dominici F.
Formal Analysis
;
Romano S.
Data Curation
;
Landi A. C.
Data Curation
;
Salvetti M.
Supervision
;
Tacchella A.
Conceptualization
;
Crisanti A.
Formal Analysis
;
Grassi F.
Penultimo
Supervision
;
Palagi L.
Ultimo
Supervision
2020

Abstract

Multiple Sclerosis (MS) progresses at an unpredictable rate, but predictions on the disease course in each patient would be extremely useful to tailor therapy to the individual needs. We explore different machine learning (ML) approaches to predict whether a patient will shift from the initial Relapsing-Remitting (RR) to the Secondary Progressive (SP) form of the disease, using only “real world” data available in clinical routine. The clinical records of 1624 outpatients (207 in the SP phase) attending the MS service of Sant'Andrea hospital, Rome, Italy, were used. Predictions at 180, 360 or 720 days from the last visit were obtained considering either the data of the last available visit (Visit-Oriented setting), comparing four classical ML methods (Random Forest, Support Vector Machine, K-Nearest Neighbours and AdaBoost) or the whole clinical history of each patient (History-Oriented setting), using a Recurrent Neural Network model, specifically designed for historical data. Missing values were handled by removing either all clinical records presenting at least one missing parameter (Feature-saving approach) or the 3 clinical parameters which contained missing values (Record-saving approach). The performances of the classifiers were rated using common indicators, such as Recall (or Sensitivity) and Precision (or Positive predictive value). In the visit-oriented setting, the Record-saving approach yielded Recall values from 70% to 100%, but low Precision (5% to 10%), which however increased to 50% when considering only predictions for which the model returned a probability above a given “confidence threshold”. For the History-oriented setting, both indicators increased as prediction time lengthened, reaching values of 67% (Recall) and 42% (Precision) at 720 days. We show how “real world” data can be effectively used to forecast the evolution of MS, leading to high Recall values and propose innovative approaches to improve Precision towards clinically useful values.
2020
Multiple sclerosis; machine learning; prediction
01 Pubblicazione su rivista::01a Articolo in rivista
Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis / Seccia, R.; Gammelli, D.; Dominici, F.; Romano, S.; Landi, A. C.; Salvetti, M.; Tacchella, A.; Zaccaria, A.; Crisanti, A.; Grassi, F.; Palagi, L.. - In: PLOS ONE. - ISSN 1932-6203. - 15:3(2020). [10.1371/journal.pone.0230219]
File allegati a questo prodotto
File Dimensione Formato  
Seccia_Considering-patient-clinical_2020.pdf

accesso aperto

Note: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0230219
Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Creative commons
Dimensione 1.15 MB
Formato Adobe PDF
1.15 MB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1385217
Citazioni
  • ???jsp.display-item.citation.pmc??? 11
  • Scopus 28
  • ???jsp.display-item.citation.isi??? 22
social impact