Mario Kart & The Leap Motion
- Alexis Hale
- Nov 12, 2018
- 9 min read
For this particular assignment, my Human-Computer Interaction class was assigned to do the conceptual design and user testing of the Leap Motion. We weren’t allowed to use it as a VR or a sign language input device, as those integrations of Leap Motion had already been discussed in class. The Leap Motion would have to complement the application we were integrating it with.
If you aren’t familiar with the Leap Motion, it is a device that can detect hand and finger movement and can be used as a mouse, but without needing to touch anything. In order to detect the users’ hands position and movement, it shoots light upwards from the device and uses cameras to detect where the light bounces.
As we know, I’m a huge fan of video games: both playing them and making them. How would the Leap Motion improve my gaming experience? I thought. For one, I wouldn’t have to hold a controller! I thought back to the recent past searching for games that could work with the leap motion. My group partner, Allie, and I ended up agreeing on integrating the Leap Motion with Mario Kart.
Mario Kart is a go-kart style racing video game where you can race with many characters from the Mario franchise. One of the most well-known features of this game are the power-ups players can use to their advantage to advance in the race.
Gestures for Functions (1st Iteration)
We first set out by determining what functions the Leap Motion would need to be able to execute when setting up and playing Mario Kart. For all of the following gestures, the Leap Motion is located a foot or two in front of the user so that the user’s hands are just above the device.
For the user interface, the user would need to be able to select options in order to start the game, navigate through selection pages, and change settings in the game. This would all be done through a pointing gesture: swiping the finger to change the selected button and tapping to select it.
In our first iteration of our gestures, navigating through menus was done by pointing and tapping the pointer finger on the selection. To move left or right, the user would swipe their finger in that direction, just as someone would navigate on a smartphone touch screen.
When in game, a player needs to be able to control their vehicle; they need to accelerate, steer, brake, reverse, drift (if driving in manual mode), and perform tricks. Additionally, players need to be able to throw power-ups forward and backwards, pause and unpause the game.
Driving would be controlled with the dominant hand and the steering motion like a rudder. In order to start accelerating forward the player would hold their dominant hand up with their palm facing inward (refer to photo below). To go left and right, the player can direct their hand either to the left or the right. They can choose to move either from the wrist or elbow, although a user tester commented that moving from the elbow was more comfortable. Since the acceleration is a binary function in the game - the user is either accelerating or not accelerating - we determined there was no need for the driving gesture to indicate more than direction. Unlike a car, the player can’t accelerate at their own pace; acceleration is determined by what kind of vehicle they race in.
Admittedly, the driving gesture could be more intuitive - the user could drive by pretending to hold a steering wheel - however, that gesture would be hard for the Leap Motion to detect direction and for it to detect both hands if the hands are located at 12 o’clock and 6 o’clock. Additionally, this games has a lot of functionality for us to cover with gestures, so we needed one hand to be free.
Break and reverse work very similarly to driving. In order to break, the player would hold their hand out flat, palm facing the floor. Braking and reversing are combined to one gesture since it is also combined in-game. When the car is moving and the user hits “brake”, it will apply the hand brake. When the car is no longer moving, it will continue to reverse. When reversing, the player can control the direction, the same way they would with the driving gesture.
In Mario Kart, the player can perform tricks like a wheelie if they are on a bike, or a flip if they are going off a jump. Since this is happening while the player is driving, their dominant hand will start in the driving position. To perform the trick, the user will flick their hand upward so their fingers are pointing towards the ceiling, and then return to the driving position.
Players also have the option to drift corners while driving. If they chose to drive an automatic car, this is done automatically when turning a corner; however, if they chose to drive manual, then they have control over when they want to drift. In order to drift, the user would change their straightened out hand into a fist while driving. In order for the Leap Motion to determine the direction of the drift, we thought it would be able to decide from the orientation of the fist and the wrist, since it is able to distinguish the difference between a fist and wrist. Additionally, the device can detect the direction of motion, so it would be able to tell if the user moved it from the center to the left, from the left to the right, etc.
When using power-ups, users can chose to throw the power-up directly in front of them or throw it behind them. To throw a power-up forward, the player uses their non-dominant hand to make a throwing gesture.
In order to throw the power-up backwards, the user makes the same motion, but downwards, as if they were dropping their power-up on the ground. Although throwing it backwards seems more intuitive, it is more awkward of a gesture. It turns out throwing it downward is still intuitive because when using the banana power-up backwards, it drops the power-up on the ground, just like our gesture does.
Lastly, in order to pause and unpause the game, the user turns both hands into fists. Since this is also a binary gesture - the game is either running or paused - one gesture would suffice.
User TestingUser testing was done by Wizard-of-Ozing since we were not implementing our system. Wizard-of-Ozing is when the user is fooled into thinking that the system is responding to their input. Alternatively, the user can just pretend that they are in control as well, when there is someone behind the scenes controlling the inputs.
For our user testing, we played Mario Kart on the Wii. The users sat in front of the TV screen performing the gestures while Allie, a Mario Kart wizard, sat diagonally behind them, carefully interpreting their gestures on the Wii remote and what was going on in-game and on-screen.
Our first user tester was a good friend, Evan. He generously allowed us to use his Mario Kart game and Wii for our user testing. Before starting the game, we explained our assignment to Evan and what a Leap Motion was. We told him that we wanted to play the game using our hand gestures and Allie would interpret his gestures on the Wii remote. With the Allie’s demonstrations, Evan was able to learn the gestures in just a few minutes.
We started with a practice round in order to get used to Wizard-of-Ozing. The practice round helped us determine where everyone was sitting, especially so Allie could see both of Evan’s hands and the screen as well as adjust to the puppeteer-behind-the-scenes role. During the practice round, we also ended up cutting the manual drifting gesture. It was too hard for Allie to interpret while Wizard-of-Ozing and would likely be just as hard for the Leap Motion. Since we cut this gesture, if players are to play Mario Kart with our Leap Motion gesture set, they will have to drive using the automatic mode, rather than the manual mode.
After the practice round, Evan was already well-versed in the gestures and didn’t have to ask for help remembering them. This suggests that our gestures were both easy to learn and easy to remember.
Our second user, Ed, had a smoother gameplay experience since we made improvements after a few iterations with Evan. He was briefed the same as Evan was and proved to learn and remember the gestures easily.
What We Learned From User Testing
As previously mentioned, both users found the gestures to be easy to learn and didn’t have trouble remembering them while in-game.
Since Evan was our first user tester, we made most if not all of our changes with him. Firstly, we ended up getting rid of the drifting option, thus eliminating manual mode gameplay.
Then, we changed how users navigate through the menus. We found that swiping in the air with pointer finger grew tiring and it was hard to distinguish between a directional gesture and a selection gesture. Rather than using a pointer finger to swipe through, we changed it to an open hand. So now the motion is more like waving through the menu. To select something off of a menu, the user would point with their pointer finger, just as they did before. We ended up keeping pointing for selection since it was the most intuitive and still made it easy to distinguish between swiping and selecting.
Originally, Evan was steering just by turning his hand at the wrist, but this motion became tiring for him after a few rounds. We ended up changing the steering to a motion that stems at the elbow. He commented that the wrist motion is something that people aren’t used to so moving from his elbow felt more natural and less tiring.
Overall, both users found that our motions both made sense and were easy to learn. Evan found that using the power-ups were especially intuitive. They both were able to play well in-game and even placed in the top 3 among the computer competitors. After several rounds, we found that their dominant arms grew tired from holding them up the whole time. There’s not much more to improve upon besides the aforementioned gestures since this is just how the Leap Motion works. You use your hands and arms as a controller above the sensor.
Limitations
Implementing and using this technology does come with its limitations, all of which we had to consider in the design of our gestures and gameplay. For one, the user has to use their hands and arms to control any programs like our game and after a while, it becomes tiring.
Since it’s using light and how the light bounces back to determine the location of the hands and arms, it’s essentially reading a 2 dimensional input in black and white with some depth information. If you were to put your hands on top of each other, it wouldn’t be able to detect the top hand, since the light bounces off the bottom hand, it obscures the top hand. For this reason, we couldn’t use a more intuitive driving motion, as if the user was using a steering wheel.
We are also limited in how many gestures the leap motion can interpret. How many motions can you think of that the Leap Motion can see while remaining within its limited range of sight? A controller is really much easier to use because all the buttons can be close together and there can be many of them. As long as they are labeled or textured, buttons are easy to distinguish, and with enough time and practice, easy to memorize.
This method of user testing, Wizard-of-Ozing, also comes with its limitations. Our wizard, Allie, is not the Leap Motion (in case that wasn’t clear before haha), and certainly isn’t receiving the same data the Leap Motion is receiving. How far our users wanted to turn left or right was determined just through Allie’s interpretation and affected by her perspective/angle, rather than an algorithm that would be able read just how much the user had moved their hand and in which direction. Of course, none of this is Allie’s fault! She is a splendid human being, but an awful Leap Motion.
Concluding Remarks
Through user testing, we were able to determine that our gestures do work. Users could learn them in a short time, remember them, and use them when they needed to in-game (i.e. they weren’t slowed down by gesturing). I think for what we were working with, our gestures did a good job.
As far as the Wizard-of-Oz experience goes, I think it could be somewhat believably done, even with a user didn’t know they weren’t controlling it all along. It looks believable in our using testing videos. When the user steers their hand toward the left, their vehicle drives toward the left. When the user wants to use the power-up, its used correctly in the indicated direction. It helps that in the video, you can’t see Allie sitting behind our testers, playing with the Wii remote.
Did this enhance their experience at all? I think playing Mario Kart with gestures is fun for only a few rounds because the arms get tired. The idea of playing hands-free is pretty cool, but right now, the technology we have has many limitations. If the cameras magically had x-ray abilities without exposing us to harmful radiation, this would be awesome!
Comments