Categorical Foundation of Explainable AI: A Unifying Theory

Explainable AI (XAI) aims to address the human need for safe and reliable AI systems. However, numerous surveys emphasize the absence of a sound mathematical formalization of key XAI notions—remarkably including the term “explanation”, which still lacks a precise definition. To bridge this gap, this paper introduces a unifying mathematical framework allowing the rigorous definition of key XAI notions and processes, using the well-funded formalism of Category theory. In particular, we show that the introduced framework allows us to: (i) model existing learning schemes and architectures in both XAI and AI in general, (ii) formally define the term “explanation”, (iii) establish a theoretical basis for XAI taxonomies, and (iv) analyze commonly overlooked aspects of explaining methods. As a consequence, the proposed categorical framework represents a significant step towards a sound theoretical foundation of explainable AI by providing an unambiguous language to describe and model concepts, algorithms, and systems, thus also promoting research in XAI and collaboration between researchers from diverse fields, such as computer science, cognitive science, and abstract mathematics.