NLP-Based Management of Large Multiple-Choice Test Item Repositories

Multiple-choice questions (MCQs) are widely used in educational assessments and professional certification exams. Managing large repositories of MCQs, however, poses several challenges due to the high volume of questions andthe need to maintain their quality and relevance over time. One of these challenges is the presence of questions thatduplicate concepts but are formulated differently. Such questions can indeed elude syntactic controls but provide noadded value to the repository. In this paper, we focus on this specific challenge and propose a workflow for the discovery and management ofpotential duplicate questions in large MCQ repositories. Overall, the workflow comprises three main steps: MCQpreprocessing, similarity computation, and finally a graph-based exploration and analysis of the obtained similarity values. For the preprocessing phase, we consider three main strategies: (i) removing the list of candidate answers from each question, (ii) augmenting each question with the correct answer, or (iii) augmenting each question with all candidate answers. Then, we use deep learning–based natural language processing (NLP) techniques, based on the Transformers architecture, to compute similarities between MCQs based on semantics. Finally, we propose a new approach to graph exploration based on graph communities to analyze the similarities and relationships between MCQs in the graph. We illustrate the approach with a case study of the Competenze Digitaliprogram, a large-scale assessment project by the Italian government.