Physical Immersive Video Gaming:
Using Pac-man to Help Fight Obesity
by
Mathew Buckman

The first step of the proposed methodology was to create an immersive 3D game that is both fitting to the 4-screen "Immersatorium" cube setup, as well as the pouch interface. The "Immersatorium" consists of four screens forming a cube, onto which images are projected creating a virtual immersive environment. The pouch interface is a device, worn on the hip, containing various sensors which allow for the movement through virtual environments by walking in place. The game developed would be based on the arcade game Pac-Man as it is a relatively simple environment in terms of 3D modeling, the labyrinth layout fits well with the projection cube setup, and allows for easy comparison between the traditional and immersive versions. Additionally, the need to "watch your back" in the immersive form of this game supports interaction from secondary active players.

The game was to be built in a custom game engine developed using the Microsoft DirectX framework as it is more robust than other 3D environments such as Adobe Director. The decision was made to create several environments, each having a unique layout and high quality models with photorealistic textures. This decision was counterintuitive to the one choosing to use Pac-Man for the simplicity of its environment, and was a large contributing factor to the changes in methodology, which will be explained in a later section.

Given a two quarter time-frame for development, game development was to be completed in one quarter (10-12 weeks) with the second quarter allotted for usability testing and refinement of the game, interface, and immersive medium. The original work schedule for game development was as follows:

• Week 1
  • research into the features and game play of the original Pac-man game
  • development of features that will be incorporated into the new version of the game
• Weeks 2-3
  • storyboarding and mock-ups (either sketches or modeled) of 3D characters and environment
  • design and sketch board layouts and object placement
  • design and sketch virtual user interface
  • obtain/record sounds to be used in the game
• Week 4
  • initial development of 3D models in Maya
  • refinement and texturing of 3D models
  • development of graphics for virtual user interface
• Week 5
  • plan and develop layout of game classes and functions
• Weeks 6-13
  • build/code the game
    • build the game "shell"
    • class layouts/placeholders
    • empty 3D environment
    • setup of the screen display
    • develop splash screens and initial menus
    • add functionality to interface with the pouch
    • start building the 3D environment
    • incorporate 2D virtual user interface elements
    • continue development of remaining game elements including sound
• Weeks 14-20
  • test and debug the game and interface
  • fix issues and continue the testing cycle until satisfied that the game is completed
  • usability testing

The methodology originally laid out in the Capstone proposal eventually proved to be unrealistic. The decision to build an original game engine from the ground up with high quality models and textures was overly ambitious and beyond the scope of what was required and feasible in the given timeframe. This led to the project falling very behind the outlined work schedule. As a result, the project was abandoned for some time due to outside factors, including employment and financial requirements, which wouldn't have been encountered had the project been completed in a more timely manner.

After remaining dormant for a few years, project work picked back up, but with a greatly condensed timeline. In addition to a tighter deadline, institutional changes added the challenge of finding a useable projection cube setup that would meet my needs. In the end, a setup could not be found, and building a setup of my own was added to my growing list of tasks.

While many of the decisions remained from the original proposal, such as moving away from Adobe Director and building a game based on Pac-Man, some changes needed to be made to simplify the project and scope something much more realistic and feasible. The first such change was to scrap the idea of building an original game engine. This was far outside the scope of what the project set out to do, and required way too much time and additional research to complete in a timeframe expected of a Capstone project. If the project was to succeed, a pre-built game engine would have to be used.

The second change made was a reduction in the visual complexity of the game. Many weeks were spent solely working on complex models and textures just for props in the game environments. For a commercially distributed product this level of work would be acceptable. But for a prototype that kind of detail is unnecessary and becomes a hindrance to progress. Therefore, the environment would be altered to something simple, yet still visually appealing. Additionally, models would be simplified, and only required objects would be placed in the environment.

A third change was to that of the complexity of the game itself. Originally, it was intended that there would be at least three different labyrinth layouts with different difficulty settings affecting how ghosts behave. The original Pac-Man game was built using just one labyrinth layout with various properties changing as a player progresses through levels, affecting the difficulty of gameplay in a more programmatic way. While it added to the complexity of the project code, implementing this method of game progression drastically reduced the amount of required time consuming tasks such as designing labyrinth layouts and placing wall models, dot models, and movement waypoints. It also expanded the number of playable game levels from three to an infinite number with over twenty levels of difficulty.

Putting the project off for an extended period of time put me in a tight spot in regards to the time available to complete the project. Instead of the original 20 weeks of development time, I now had 11 weeks, with only 5 of those weeks dedicated solely to software development. Additionally, I now had the requirement of building my own projector cube setup. As a result, the following revised timeline was used to guide the project development:

• Weeks 1-6
  • Review status of previous work
  • Determine remaining work
  • Adjust scope and complexity of the project
  • Investigate game engine options
• Week 7
  • Become familiar with Unity game engine
  • Build & place environment models (walls, dots)
  • Player movement
  • Score management
  • Ghost movement
  • Heads Up Display
• Week 8
  • Bonuses
    • Management
    • Modeling
  • Levels
    • Progression
    • Difficulty
  • Player health
  • Tunnel areas
  • Ghost release from pen
• Week 9
  • Frightened mode
  • Ghost phases (chase, scatter)
  • Additional modeling
    • Ghosts
    • Dot & energizer
  • Texturing
    • Walls
    • Ground plane
    • Ghosts
    • Dot & energizer
    • Bonuses
  • Game audio
  • Menu system
    • Main menu
    • Options
    • Pause menu
    • Game over screen
• Week 10
  • Multiple directional cameras
  • Portal texture & audio
  • Build projector cube
• Week 11
  • Finalize assembly of input device
  • Complete Arduino code for input device
  • Add menus for hardware interface
    • Connect to device
    • Calibrate device
  • Testing, tweaking, & bug fixing

To create a fully immersive gaming experience, the player needs to have a 360 degree view of the virtual environment. This is achieved by surrounding the player on four sides with a translucent material, and projecting reversed images onto that material from outside the cube. Four cameras are used in the game with their fields of view aligned to capture a seamless 360 degree view around the player.

The frame of the cube was built using ½" PVC pipe because it's sturdy, lightweight, and easy to assemble using pipe fittings. The cube is 5x5x6' to allow ample room for movement. Stretched across the sides of the frame are four shower curtain liners, attached to the top horizontal pipes with white duct tape, and the side vertical pipes with several pieces of Velcro. Shower curtain liners are very inexpensive and translucent enough to clearly display projected images to the player inside. Velcro was used on the vertical pipes to allow for adjustments of the liners, ensuring a taut viewing surface which provides the clearest image. Additionally, the Velcro allows for easy entry and exit of the cube.

Accelerometer tracking makes use of sensors to detect player tilt, rotation, and movement in various directions. This data is then used to affect movement of player avatars or other aspects of gameplay. The primary product to make use of this form of tracking is the Nintendo Wii system through the Nintendo Wii Remote. Some fitness games developed for the Nintendo Wii use accelerometer readings from the Wii Remote to detect player steps, treating it like a pedometer. This behavior is similar to that of the input device used for this project, but is lacking in a couple ways.

First, the Wii Remote is meant to be placed in the player's pocket. If the player is wearing clothing without pockets, the remote is instead held in the player's hand. This forces the game to rely on activity detected when the player moves his arms, and may not accurately reflect actual walking or running. This can lead to less accuracy and a less immersive experience.

A second shortcoming stems from the fact that these interactions are intended to happen with only one display. This means that an additional input device must be used to control player movement outside of simple forward momentum. This also means that these interactive games are incapable of offering a fully immersive gaming experience.

One of the most popular methods of detecting player movement is through the use of cameras. These cameras continuously capture images of the player and their environment which are used by the system to interpret movement. Two available products using this technology are Xbox Kinect and PlayStation Camera. Both of these systems make use of image capture to detection motion, but do so in slightly different ways.

The Kinect 2.0 device used with Microsoft's Xbox One system consists of a single 1080p RGB camera used for capturing high definition images. The Kinect's motion detection is augmented by the use of an infrared blaster which sends out particles of light that bounce off the player and their environment. The amount of time taken for these particles of light to return to the camera is used to determine distance, very similar to how sonar imaging works.

Sony's PlayStation Camera device does not have an IR blaster, but does make use of a second camera to improve the accuracy of its motion detection. The two cameras work together to triangulate the player's position. This dual camera setup improves accuracy and allows Sony to ditch the wand controllers required by PlayStation Camera's predecessor, PlayStation Eye.

Both camera tracking devices offer application control and interaction through gesturing, something this project lacks but could make great use of. Gesture control could further remove connections to the world outside the projection cube, offering an even more immersive experience. However, like the Nintendo Wii, these products are intended for use with a single display, again depriving the user of an experience and environment that fully surrounds them.

One immersive experience which has gained a lot of traction as of late is that provided through the use of virtual reality headsets. These devices are worn over the user's eyes, completely covering them to block out all other views and completely immerse the user in the virtual world. Screens inside the device display a view of the 3D virtual environment which is adjusted based on the user's movement. While Sony is working on their device, Morpheus, and Google has released Google Cardboard which uses an Android smartphone and cardboard goggles built from a template to create a VR headset, the most prominent VR headset today is the Oculus Rift.

In addition to the goggles housing the display screens, the Oculus Rift contains a control box attached to the headset with a 6 foot cable, as well as several sensors used to read user movement. These sensors include a gyroscope, accelerometer, and magnetometer, two of which are used in the input device for this project. While the original device only allowed for the tracking of player orientation, newer iterations are including infrared LEDs on the device which are monitored by an external camera. This provides some positional tracking and allows players to lean in for closer views or tilt their heads to look around corners.

Virtual reality headsets definitely have the edge when it comes to full immersion into a virtual environment. The devices are able to completely block out the external world in a way that this project is incapable. What is lacking, though, is the health benefit provided by this project. While the VR headsets could likely be used to track a user's movement in a way that facilitates travel through a virtual world, it's accompanied by a danger to the user. Having no view of the real world around you places the user at risk of walking into objects, tripping, or becoming disoriented and falling down. The cable permanently attached to the headset adds an additional risk of injury. These devices have also been known to cause headaches and motion sickness in some users. So, while it's a more immersive experience, VR headsets do not seem well suited for vigorous physical activity.

That being said, use of a VR headset with additional equipment, namely the Virtuix Omni, could alleviate some of the issues previously stated. The Virtuix Omni is a treadmill system with a support harness attached to and support ring surrounding the user. The user is able to walk or run on the platform, which moves under their feet simulating real world movement. While this doesn't help with the issues of disorientation and motion sickness, it is a great solution to the previously stated safety concerns and is one of, if not the, best way to handle movement through an immersive virtual environment.

The most valuable lesson learned from this project is to set realistic goals. A Capstone project is intended to show your skill and what you learned throughout the course of your graduate studies. It is not, however, a time to learn completely new topics and attempt to reinvent the wheel. Because I decided to tackle something new and go way above and beyond the requirements of a Capstone project, an incredible amount of additional work was imposed and the project strayed greatly from the established work timeline. As a result, a large amount of completed work eventually had to be scrapped and redone in a much smaller timeframe. Additionally, delays resulting from that original derailment imposed extra hurdles due to facility and other changes occurring as time went by and the project stagnated. For these reasons it is imperative to set realistic goals and deadlines at the start of a project, or you will be very likely to fail.

A second lesson learned is that the translation of an application from a traditional format to an immersive one is accompanied by a whole host of obstacles one might not immediately think of. Particularly impacted is the display of textual or other two-dimensional data, like that shown in a heads up display (HUD). Moving the HUD on the screen as the user rotates presented issues because the heading received constantly varied slightly, resulting in a jittery display. Displaying the HUD stretched across all screens spread the information out too much, and showing it on all screens was repetitive and cluttered up the view. The decision was made to only display the HUD on the screen considered forward, that is the one the player faces at the start of the game. This served our application well since the only time a view of the HUD is vital to the player is at the start of the level. Displays with more pertinent information would not be best incorporated using this method, and an alternative solution would likely need to be implemented.

Another impact of the translation to an immersive format was the loss of a top down view of the environment. In the original Pac-Man game, the player had a view of the entire level and could see when ghosts were around an upcoming corner. This is not the case when the player is placed inside of the environment. The loss of that all-encompassing view increased the difficulty of the game, but also made it more exciting. Users must now attempt to listen for the locations of ghosts and need quicker reflexes to alter course if a ghost crosses their path. Nevertheless, the difficulty of the game was increased. To counter this, some features of the original game were left out, such as periodic increases in ghost speeds and "chase mode", where ghosts would relentlessly target the player.

The change in input device from keyboard to human movement also instilled the need to make adjustments to the game. The new method of input if far more physically demanding, which serves the goal of promoting health benefits, but could also exhaust the player before the first level is complete if not handled correctly. For this reason, the speed of the player's movement through the environment had to be tweaked to create an experience that provided aerobic value, but was also enjoyable and sustainable to play.

A third more minor, yet still valuable, lesson learned was to always thoroughly research and test your hardware components. The compass module used in the input device had been well-tested individually, as well as with other components. However, when the entire device was assembled an issue was discovered. Lacking an accelerometer to account for tilt, the compass module must be kept level in order to receive accurate readings. The sensor had been soldered to the circuit board in a way that positioned it at a 90 degree angle when worn by the user, resulting in inaccurate and unusable data. As a result, the soldering had to be undone and the sensor reattached in a way that positions it flat when worn by a user.

One strong potential addition to the project would be integration with a handheld device. As a solution to the previously stated problem of displaying information, a tablet or smartphone could be integrated with the system to serve as the main point of information for the player. Real-time scoring and notification data could be continuously streamed and displayed on the device. A most valuable feature would be a mini-map or radar system, providing the player with real-time information on the whereabouts of enemies and valuable targets.

Improvements could also be made to the input device. Using a 9 volt battery to power the device works for short-term demonstration, but is inefficient and expensive for long-term use. Additionally, while the detection of footsteps is pretty accurate, improvements could be made to eliminate false readings, missed readings, and even to incorporate jumping into the available interactions. An upgraded compass sensor with tilt compensation could also prove very useful in improving the player's experience and making for more reliable and accurate movements.

Another area to explore would be the use of this setup in applications other than gaming. The system could be used to participate in virtual tours of real places—like Google Earth but more immersive. One could travel the world without ever having to set foot on a plane. Aside from real locations, artificial environments could be created as well to explore and serve as a motivation to exercise or even a space of tranquility and relaxation.

A final example of future work is to incorporate the software application with smartphones or other mobile devices. Most smartphones today include all of the sensors put into the input device for this project. Altering the application to work with these devices would allow a player to simply use their own equipment when entering the cube to play the game. This could simplify and expedite entering a game, and also allow for the creation of applications which connect and accept input from multiple players at the same time.

While strides are continuously being made in the fields of immersive gaming and tangible interfaces, there's still an incredible amount of potential within these areas. The video game industry is a huge and ever-expanding one with much untapped potential, but also one with some negative impacts on society. Increased use of video games is a continuous contributor to the growing epidemic of obesity in the United States and other countries. Tangible interfaces offer a way to not only help combat the epidemic, but also contribute to the growth and profitability of the gaming industry. Incorporating natural, physical interactions with immersive environments helps draw the player into the virtual world, blurring the boundaries between realms and creating a richer experience.