Keyrtual: A Lightweight Virtual Musical Keyboard Based on RGB-D and Sensors Fusion

The digital world is becoming more and more part of our everyday life. Research groups and industries have been working for years to find innovative and natural interaction ways to improve users’ satisfaction with the approach to the technology. In particular, Human-Computer Interaction (HCI) made a great evolutionary leap with accurate sensors and high-computational devices. Moreover, the miniaturization of circuits made modern systems portable and affordable. Also, advanced algorithmic approaches, e.g., Machine and Deep Learning (ML and DL), strongly support achieving noticeable results at the price of potentially long training times and considerable resource requirements. In this context, we propose a novel system for playing a virtual musical keyboard exploiting an RGB and a dual-Infrared (IR) sensor. It is designed for musicians and enthusiasts for multi-purposes, e.g., training or music production. After a calibration phase, the software can be executed in real-time with responsive feedback. The main focuses of the proposal are the portability and the low-computational resources requirement. Quantitative and qualitative results highlight the system’s overall effectiveness, proving that the proposed pipeline could be promising.