Chat-GPT, a natural language processing (NLP) tool created by Open-AI, can potentially be used as a quick source for obtaining information related to prostate cancer. This study aims to analyze the quality and appropriateness of Chat-GPT's responses to inquiries related to prostate cancer compared to those of the European Urology Association's (EAU) 2023 prostate cancer guidelines. Overall, 195 questions were prepared according to the recommendations gathered in the prostate cancer section of the EAU 2023 Guideline. All questions were systematically presented to Chat-GPT's August 3 Version, and two expert urologists independently assessed and assigned scores ranging from 1 to 4 to each response (1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect). Sub-analysis per chapter and per grade of recommendation were performed. Overall, 195 recommendations were evaluated. Overall, 50/195 (26%) were completely correct, 51/195 (26%) correct but inadequate, 47/195 (24%) a mix of correct and misleading and 47/195 (24%) incorrect. When looking at different chapters Open AI was particularly accurate in answering questions on follow-up and QoL. Worst performance was recorded for the diagnosis and treatment chapters with respectively 19% and 30% of the answers completely incorrect. When looking at the strength of recommendation, no differences in terms of accuracy were recorded when comparing weak and strong recommendations (p > 0,05). Chat-GPT has a poor accuracy when answering questions on the PCa EAU guidelines recommendations. Future studies should assess its performance after adequate training.

Quality of information and appropriateness of Open AI outputs for prostate cancer / Lombardo, Riccardo; Gallo, Giacomo; Stira, Jordi; Turchi, Beatrice; Santoro, Giuseppe; Riolo, Sara; Romagnoli, Matteo; Cicione, Antonio; Tema, Giorgia; Pastore, Antonio; Al Salhi, Yazan; Fuschi, Andrea; Franco, Giorgio; Nacchia, Antonio; Tubaro, Andrea; De Nunzio, Cosimo. - In: PROSTATE CANCER AND PROSTATIC DISEASES. - ISSN 1365-7852. - (2024). [10.1038/s41391-024-00789-0]

Quality of information and appropriateness of Open AI outputs for prostate cancer

Lombardo, Riccardo
;
Gallo, Giacomo;Stira, Jordi;Turchi, Beatrice;Santoro, Giuseppe;Riolo, Sara;Romagnoli, Matteo;Cicione, Antonio;Tema, Giorgia;Pastore, Antonio;Al Salhi, Yazan;Fuschi, Andrea;Franco, Giorgio;Nacchia, Antonio;Tubaro, Andrea;De Nunzio, Cosimo
2024

Abstract

Chat-GPT, a natural language processing (NLP) tool created by Open-AI, can potentially be used as a quick source for obtaining information related to prostate cancer. This study aims to analyze the quality and appropriateness of Chat-GPT's responses to inquiries related to prostate cancer compared to those of the European Urology Association's (EAU) 2023 prostate cancer guidelines. Overall, 195 questions were prepared according to the recommendations gathered in the prostate cancer section of the EAU 2023 Guideline. All questions were systematically presented to Chat-GPT's August 3 Version, and two expert urologists independently assessed and assigned scores ranging from 1 to 4 to each response (1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect). Sub-analysis per chapter and per grade of recommendation were performed. Overall, 195 recommendations were evaluated. Overall, 50/195 (26%) were completely correct, 51/195 (26%) correct but inadequate, 47/195 (24%) a mix of correct and misleading and 47/195 (24%) incorrect. When looking at different chapters Open AI was particularly accurate in answering questions on follow-up and QoL. Worst performance was recorded for the diagnosis and treatment chapters with respectively 19% and 30% of the answers completely incorrect. When looking at the strength of recommendation, no differences in terms of accuracy were recorded when comparing weak and strong recommendations (p > 0,05). Chat-GPT has a poor accuracy when answering questions on the PCa EAU guidelines recommendations. Future studies should assess its performance after adequate training.
2024
Chat-GPT; prostate cancer; guidelines.
01 Pubblicazione su rivista::01a Articolo in rivista
Quality of information and appropriateness of Open AI outputs for prostate cancer / Lombardo, Riccardo; Gallo, Giacomo; Stira, Jordi; Turchi, Beatrice; Santoro, Giuseppe; Riolo, Sara; Romagnoli, Matteo; Cicione, Antonio; Tema, Giorgia; Pastore, Antonio; Al Salhi, Yazan; Fuschi, Andrea; Franco, Giorgio; Nacchia, Antonio; Tubaro, Andrea; De Nunzio, Cosimo. - In: PROSTATE CANCER AND PROSTATIC DISEASES. - ISSN 1365-7852. - (2024). [10.1038/s41391-024-00789-0]
File allegati a questo prodotto
File Dimensione Formato  
Lombardo_Quality-information_2024.pdf

solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 300.51 kB
Formato Adobe PDF
300.51 kB Adobe PDF   Contatta l'autore

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1705650
Citazioni
  • ???jsp.display-item.citation.pmc??? 0
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact