Finite Rank-Biased Overlap (FRBO): A New Measure for Stability in Sequential Recommender Systems (Extended Abstract)

Sequential Recommender Systems (SRSs) are widely used to model user behavior over time but they often face a critical challenge: they can fail when faced with perturbations in their training data. While the conventional Rank-Biased Overlap (RBO) measure is widely used, it does not properly address this issue, especially when dealing with finite rankings. To fill this gap, we introduce the Finite Rank-Biased Overlap (FRBO) measure. We study the impact of removing elements at the beginning, in the middle, and at the end of the sequence: the latter removal has a negative impact on performance of up to 60% in NDCG. Surprisingly, removing elements from the beginning or middle of sequences has minimal impact on performance. These results shed light on the crucial role of element positioning within the training data and highlight the urgent need for improved robustness in SRSs. We make available our code implementation1 for FRBO and invite further exploration and adoption by the research community.