The IlluminAI project. A deep neural network and immersive visualization system to enhance illuminated manuscripts

In museum and archive digital catalogues, illuminated manuscript pages can often be found within heterogeneous groups of reproductions, coexisting with other types of artworks and objects. With them, figurative miniatures share the depicted subjects that are recognizable, regardless of the medium used, for their specific iconography. Thanks to the use of a visual vocabulary still common today, illuminations are also the element that mostly attracts the non-academic public, making the often-incomprehensible content partly accessible despite the language. The paper will present IlluminAI, a project still in progress, which aims at the enhancement of late medieval and Renaissance illuminated codices using artificial intelligence through an immersive visualization system capable of automatically recognizing manuscript sheets, analyzing their content, and relating specimens with similar illustrations or artworks from the same theme. After some brief references to contextualize the work, we will expose the first completed phase of the research focusing on the original dataset composition before outlining the chosen semi-automatic labeling strategy and the interactive machine learning approach. This was used to create with transfer learning a model able to recognize manuscript pages and identify inside of them five characteristic layout elements. We will then switch to the second ongoing part of the project with the design of the immersive Web3D system, based on the open-source ATON framework, that will give users the possibility to explore, inspect, compare and query large amounts of images in a three-dimensional space. The data aggregation criteria and the presentation modes will be described with particular attention to the spatial organization and novel 3D interfaces.