User testing is a strategy we use to evaluate how well we’re meeting our goals. We packed up our tech and visited Keith Middle School in New Bedford, Massachusetts earlier this month, where students tried VR Hoover Dam during their after-school club meeting. Michelle Turk, our partner at the University of Nevada, Las Vegas UNLV) is also testing the game with her undergraduate students!
What is User Testing, and Why is It Important?
In order to gauge how well the game works, we need to do user testing “in the wild.” In March, Prof. Anthony Arrigo, Prof. Scott Ahrens, and graduate student intern Gabriela Calderon visited local Keith Middle School to test the game with sixth and eighth-graders.
Earlier this year, our team reached out to local middle schools seeking opportunities to bring VR Hoover Dam to their students for user testing. We partnered with Keith Middle School in New Bedford, MA, and had a chance to meet with students in an after-school program focused on communication and decorum.
We brought three stations’ worth of equipment to each session. Each set of user test equipment consists of an Alienware gaming laptop, one Oculus headset, two Oculus handheld controllers, a high-speed data cable to connect the headset to the computer, and of course, a charger.
Ms. Jeanine Cambra, a content instructional leader for English Language Arts and Social Studies classes at Keith Middle School, and Dr. Bruce Tench, Assistant Principal, introduced us to the kids and helped facilitate the testing.
We had enough time for a few students to play most of the game, but we were unable to collect survey data because our websites were not pre-cleared through the school district and so they were blocked. Lesson learned!
Still, our ability to observe the students playing the game offered valuable insight that we took back and used to change some of its features. For example, there is a tutorial at the beginning of the game on moving around and interacting with things in the virtual world. But, not surprisingly, most students were either distracted by the excitement of the event or simply clicked through too quickly and out of order, causing the controller tutorial visuals to stack one on top of one another and obstruct the player’s view.
Also, players accumulate items and tools throughout the game which are tucked away in their inventory, but some students thought the items they grabbed disappeared and didn’t know how to access the inventory and items again.
We also learned about other visual and narrative cues that are needed. User testing is so important because, as the developer, you’ve played the game so many times that everything just seems “obvious,” but to other people, it’s not quite so obvious.
We returned the next week for a follow-up usability testing session. Most of the students who joined us during this session were already familiar with the game from the previous week and had specific events they were looking forward to reaching when it was their turn.
“I can’t wait to ride across the dam. Does the water actually look like it’s moving underneath you?” asked one student, who had been an observer in their group the previous week. Other students looked forward to more, well, explosive events of the game that they recalled from the previous session.
The user testing was exciting for everyone—us and the kids—and gave us a lot of important information to make the gameplay work better.
Some things we’re working on now…
● Sound: A feature we hadn’t considered before user testing is sound; students need to be able to hear the dialogue taking place in-game in order to appreciate watching another student play. Right now, sound only plays through the Oculus headset for the player playing the game. So how do we adapt this so that other kids can hear, too?
● Visual cues: Through several play-throughs, we realized students had a hard time finding “the next place” to go to in the game. In other cases, objects were made available to players before they had a purpose for finding those objects, and so we’re figuring out how to use objects in the game in such a way that they aren’t a distraction from the sequence of game events.
● Wint’s dog: Everyone wants to be able to interact with Wint’s dog. We probably should have seen this coming.
Are you surprised by our results from our first user testing sessions? Are you curious about how our next round of user testing will go? Contact us to let us know what you think!