Discrimination Bias Detection through Categorical Association in Pre-trained Language Models

The analysis of the presence of bias, prejudices and unwanted discriminatory behavior in pre-trained neural language models (NLMs), considering the sensitivity of the topic and its public interest, should respect two main criteria: the intuition and the statistical rigor. To the state of the art, there are two main categories of approaches for analyzing bias: those based on the models’ textual output, and those based on the geometric space of the embedded representations calculated by the NLMs. While the first one is intuitive, this kind of analysis is often conducted on simple template sentences, which limit the overall validity of their conclusions in a real-world context. On the contrary, geometric methods are more rigorous but quite more complex to implement and understand for those who are non-experts in Natural Language Processing (NLP). In this paper, we propose a unique method for analyzing bias in pre-trained language models that combines these two aspects. Through a simple classification task, we verify whether the information contained in the embedded representation of words that describes a protected property (such as the religion) can be used to identify a stereotyped property (such as the criminal behavior), requiring only a minimal supervised dataset. We experimentally verify our approach, finding that four widespread Transformer-based models are affected by prejudices of gender, nationality, and religion.