Towards Explainable Image Style Transfer: Metrics for Measuring Style and Content

Image style transfer poses a fascinating dilemma in the very definitions of content and style, as even humans struggle to quantify how much an image has been 'stylized', whether the content has been sufficiently preserved, and how adequately the two have been combined. Essentially, evaluating the quality of the output image is not a trivial task and is prone to subjectivity. We therefore propose two novel metrics to measure the balance between Style and Content in a stylized image: Regional Style Influence (RSI) and Style-Content Attribution Ratio (SCAR). Such measures are designed to highlight the most relevant regions of the image and quantify the trade-off between content and style within them. They are computed through a patch-based analysis of the input and output images, assessing the impact of style embedding and the dominance of style over content within each patch. RSI and SCAR should yield a helpful representation of the reasoning behind the tested model, taking one step forward toward its explainability. This paper provides an insightful description of our proposed metrics and shows their outcomes when applied to state-of-the-art models. The results are presented as heatmaps for the tested images, highlighting the most interesting aspects in terms of content and style definitions, along with a study on the influence of patch size. Our study is motivated by the lack of standard measures for this type of research and by the pursuit of explainability in the style transfer process, which is a crucial part of its evolution. Hopefully, our findings will provide visual support for reasoning about style and content characteristics, motivating future work in this promising area.