This project focuses on the invention of intuitive gestures to interact with a menu designed in a VR space. The new interaction technique will be used to improve the way in which natural user interaction can be applied to innovate current menu operations in a three-dimensional setting so it can be used in AR and VR situations. We call this innovation – Turn Menu.
Team– Aniruddh Iyer, Advita Adyanthaya, Won-Woo Chung Role– Interaction Design, UX Design, User Research Duration– 2 months
PROBLEM
Current menus used in VR or AR settings suffer from a few issues:
The gestures used to interact with the menu are not natural or intuitive.
The gestures and menu designs are distracting.
The menu designs themselves afford interactions in an unnatural form.
Extended usage of gestures causes fatigue.
Why is this a problem?
Gestures that do not feel natural or are unintuitive cause users to break out from the flow when interacting with an application. It distracts users from the purpose of the application itself.
SOLUTION
We created an intuitive and natural gesture set to be used with a special menu design we came up with. We call it the Turn Menu. The main gesture involved in interacting with this menu is turning your hands clockwise or anticlockwise. This gesture helps in scrolling through menu options or controlling a selected item. To select the highlighted item in the menu, you push your entire hand forward.
MAIN FEATURES
Natural Turning Interaction
The menu was designed in such a way that affords turning interactions (clockwise and anticlockwise). The turning interactions fit the mental model of users turning a dial.
Sub Menu Operations
In addition to the basic menu option, sub menu operations can also be performed. By making use of similar clockwise or anticlockwise gestures in the selection mode, menu options' values can be altered by users.
Ease of Access
The simplicity of opening and closing the menu itself makes it very easy to use. It additionally allows users to make use of the menu anywhere (in the VR environment) they see fit.
PROCESS
Background Research
We looked at a multitude of different research papers and articles on the web from different domains to understand what the current state of the art is for menu design based on gestures.
Many of the menus that we saw followed a circular menu design paradigm. They followed through on their designs by creating ring menus and pie menus. Additionally the creation of a popular menu design paradigm called TULIP brought up many different considerations for designing our menu.
Design Considerations
From our background research we identified different various factors that we wanted to take into consideration while designing the menu and the required gestures for it.
Performance of the menu shouldn’t suffer compared to other menus.
Should not cause the user discomfort.
Should not occlude environment.
Should be appropriate for both novice and expert users.
Expert users should be able to do “eyes-off” interaction with the menu.
Keep the number of gestures at a minimum to avoid cognitive overload.
Menu should have hierarchy, preferably no flat structure.
Non-dominant hands can only do non-precision tasks (high level) while the dominant hand can do precise tasks.
Menus should provide appropriate feedback for various states.
There should be clear indication of which menu is being viewed. (recognition over recall)
Constraint Analysis
Once we had our design considerations, we wanted to understand the extent to which gesture recognition works. To do this, we tested out the limits of the Leap Motion device.
We learnt that finger gestures above the palm area facing upward along with other ambiguous gestures are not detected accurately by the device. This is because the controller fails to detect movements that cannot be seen by the sensor.
Ideation
Now that we had a good understanding on existing gesture menus and various design considerations, we were ready to start creating our own menus and testing them with users in a way to elicit qualitative feedback. We needed a quick way of validating our designs without significant cost. Hence, we decided to create paper prototypes since menus in AR/VR would almost exist as physical objects.
Name - Ring Menu (v1) Navigation - Swipe hand along x-axis of menu Selection - Static selector on middle item, push with index finger
Name - Ring Menu (v2) Navigation - Swipe hand along z-axis of menu, palm facing menu Selection - Static selector on middle item, push with index finger
Name - Semi - circular Menu Navigation - Turn menu like a dial Selection - Static selector at top-middle, push with palm
Name - TULIP Menu (v1) Navigation - Swipe hand along z-axis of menu, palm facing body Selection - Bend finger corresponding to menu item
Name - TULIP Menu (v2) Navigation - “Next” and “Previous” are items on the fingers Selection - Bend finger corresponding to menu item
Name - Linear Horizontal Menu Navigation - Swipe hand along x-axis of menu Selection - Static selector on middle item, push with index finger
Concept Validation and Testing
We conducted semi-structured interviews and think-alouds for each of the prototypes focusing on intuitiveness and fatigue. This would give us good insight into initial impressions and what the expectations are based on various factors of the menu design.
Overall, the semi-circular menu was better received with the following suggestions:
Give the design better affordance of a dial
Make all items at the same level of hierarchy visible
With this feedback, we decided to improve on the semi-circular menu design for our higher fidelity prototype.
Designing the High Fidelity Prototype
Once we decided to implement the semi-circular menu, we researched on a suitable menu design. We looked up similar menu designs that have been previously implemented and were used by a substantial number of users. We believed that we could bring some of those menu design elements into our menu. One of the most impactful menu designs was the Surface Dial menu from Microsoft.
Surface Dial menu design became the basis of our design. Instead of having a semi-circular menu that only showed around 5 menu items and needed to be turned left or right to reveal the next or the previous 5 menu items, we decided to have a fully circular menu that showed around 10 items at once.
In addition, we decided to use the turning gesture to scroll through individual menu items. We made this design decision because we wanted the gesture to be intuitive so that the users can relate to an object in a real world that worked in similar fashion. In our case, it was the dial lock.
We decided to create a hierarchy of menus similar to how a linear menu has a sub menu by hovering or clicking on a menu item. In our menu, by pushing in, the item is selected or a sub menu would appear depending on the type of item that is selected.
For certain tasks such as volume control, the sub menu is different. Once the user selects volume, the user can turn left and right to adjust the volume level and go back to the released state to finalize the selection or the volume level in this case.
Usability Testing
We conducted think-alouds by giving participants specific tasks. At the end of the think-aloud, we conducted interviews with them to understand their feelings about the menu and gestures with respect to ergonomics, fatigue, intuitiveness etc.
Usability testing provided us great insights on the designed menu and gestures. We learnt that the gestures were intuitive, discoverable and easy to learn. There is minimal fatigue while interacting with the menu due to limited number of items.
Future Improvements
The menu, in its current state, only provides visual feedback. We would have liked to integrate auditory feedback. We hope that with prolonged use, expert users could use audio to control the menu without needing to look at it. Additionally, to improve the menu we could potentially make the design look 3-dimensional providing additional affordance to grab and turn it.
OVERALL REFLECTION
Creating a whole new interaction technique was a feat that I never thought I would be able to achieve. The amount of research and prototyping to be done before coming up with an innovative and intuitive solution is extremely high. One very important realization I've had is that at every step of the process, it is quintessential to go back and check if your "new" technique has already been implemented or if the problem has been solved in a better way. Additionally, I've realized that in order to come up with something tangible, innovative and realistic, you HAVE to scope the project appropriately.
