Supporting cognitive walkthrough with video data: a mobile learning evaluation study

Although expert-based evaluation techniques such as heuristic evaluation and cognitive walkthrough are often inexpensive and quick to apply, they have not proved to be effective in capturing contextual factors that arise in real-world settings. It is no trivial issue to understand how such evaluation techniques could be modified or differently applied so as to better take into account context, without loosing the advantages inherent in those techniques. This paper explores a possible way of addressing the trade-off between application of cognitive walkthrough and low cost improvements of its contextual validity. In particular, we propose and investigate the benefits of supporting cognitive walkthrough with video data about user interaction with an eLearning course on mobile device. Initial results from this study indicated that video data provided evaluators with a more detailed understanding of user characteristics and interaction contexts, leading to an improvement of their assessments in terms of the total number of system's flaws detected. Video data was regarded by experts as both relevant and useful, especially for tuning the evaluation focus on types of difficulties they would normally not have experienced because of differences in terms of abilities, knowledge and background with those of the target user group. Copyright 2005 ACM.