Julian Ramos
← All projects Input techniques · Prototyping

Interaction Techniques

Exploring hand gestures and keyboard-surface input through working prototypes and human evaluation.

Keyboard Surface Interaction

Switching between typing and pointing requires moving a hand from the keyboard to a mouse or trackpad. This adds transition time and can cause discomfort during tasks that require frequent switching.

I developed Fingers, a prototype that turns the keyboard surface into a pointing area. Across three prototype versions, I refined sensing and switching between typing and pointing so users could control the pointer while keeping their hand on the keyboard. A 25-participant study compared Fingers with a mouse and trackpad in mixed typing-and-pointing tasks. With practice, Fingers reduced switching time compared with both devices and perceived discomfort compared with the mouse. Cursor movement was faster than with the trackpad but slower than with the mouse.

Box plots comparing Fingers, trackpad, and mouse on homing time, movement time, total time, and discomfort for novice and practiced users
Fingers compared with a trackpad and mouse: switching (homing) time, pointer movement time, total time, and discomfort. “n” denotes novice users and “e” denotes users with more practice. Lower values are better. Figure 7 from the Keyboard Surface Interaction paper.
Keyboard Surface Interaction demonstration · Watch on YouTube ↗

Serendipity

Small smartwatch screens and buttons can make quick interactions awkward, especially when the other hand is occupied. The challenge was to support subtle finger input using the hardware already available on a commercial watch.

With Hongyi Wen and Anind K. Dey, I developed Serendipity, the first work to demonstrate fine-grained finger-gesture recognition using only the built-in IMU sensors of an off-the-shelf smartwatch. Our machine learning method uses accelerometer and gyroscope signals to recognize gestures such as pinching, tapping, and rubbing fingers, without additional gesture sensors. In a study with 10 participants, it distinguished five gestures with an average F1 score of 87%. Each participant’s model was trained in one session and tested on a different day, across three wrist orientations while the participant remained stationary.

F1 scores for Serendipity’s five-gesture classification for ten participants, with an average of 87 percent
Gesture classification F1 score for each of the 10 participants and the overall mean of 87%. Models were trained in one session and tested on a different day. Figure 4 from the Serendipity paper.
Serendipity demonstration · Watch on YouTube ↗
Full list →