Analyzing, modelling, and specifying visual interaction

This paper discusses the processes underlying human-computer visual interaction, thereby analysing the characteristics of visual interaction; a model and a theory of visual interaction, from which a formal specification of visual interactive systems that are trustable by their users can be derived, are also illustrated. Such a theory is called theory of visual sentences, since each message on the computer screen is described as a visual sentence, i.e., an element of a visual language that specifies the interaction. The concept of relational structure is introduced to take into account different kinds of relations which can exist among the characteristic patterns present in a visual sentence. A formal model of the dynamics of visual interaction is presented, which is specified as transformations of visual sentences, modelled through visual rewriting systems, and transformations of relational structures. Particular attention is given to usability issues so as to satisfy relevant features needed to allow non-ambiguity of interpretation, adequate communication, determinism and system viability. An example of visual interaction is given where an immunologist interacts with a simulation of the human immune system.