Understanding Tool-Tissue Interactions in Surgical Videos

Artificial intelligence (AI) is increasingly adopted in surgery for performance assessment and intraoperative decision support. A key capability in this context is the automatic recognition of tool- tissue interactions (TTIs) from surgical videos. Existing approaches for modelling surgical actions, mainly represented by 〈tool, verb, tissue〉 triplets, do not explicitly model contact. In this work, we propose a formulation of the TTI understanding problem based on contact-aware interaction modelling, which introduces two sequential objectives: (i) spacial detection of physical contact between tool and tissue, and (ii) classification of the interaction type. By explicitly incorporating depth information, our approach moves beyond purely 2D semantic reasoning and enables inference of actual physical contact rather than visual co-occurrence. This formulation enables the use in robotic and safety-critical applications for which semantic understanding only is not sufficient. The proposed architecture integrates YOLO- based instance segmentation, depth estimation, and a Vision Transformer for interaction reasoning. Experimental results show that the proposed method achieves 87.61% end-to-end accuracy on the TTI understanding task, with F1 score of 88.03%, demonstrating the feasibility of reliable, contact-aware TTI inference.