A regular two-dimensional computer screen has been the de facto interface for humans to interact with computers for years. With the recent monumental advances of consumer virtual reality (VR) technology, the consideration for alternative interfaces to interact with computers has become more prominent in computer science. This VR interface was a case study exploring designs meant to replace the traditional desktop computer graphical user interface (GUI). While many popular "VR desktop" applications simply add a flat plane onto which a regular desktop GUI is projected, we wanted to look at the interface as if everyday computing was native to VR and used all the concepts that make VR great. This means minimal use of flat-screen interfaces and applications that have an innate sense of depth, allowing the user to interact with the software in ways they would expect.

This project was developed as part of a two-semester-long senior design project along with three other team members. We developed the environment using the Unity game engine and utilized the original Vive VR headset. To program against this hardware we utilized the SteamVR SDK which provides easy access to the headset and controllers during the development process. Since this is an everyday computing environment we decided to forego the controllers in favor of hand tracking which was provided using the UltraLeap controller and corresponding SDK. Capturing keyboard input was still a problem as we assumed the user would be mobile when using this environment and it cannot be assumed they will have a regular physical keyboard within reach. To solve this we developed a custom keyboard solution using wireless, touch capacitive gloves which are described later. Using this array of technology we were able to create our creative environment for this case study.

Imagine being able to build computer programs the same way that Lego creations are made. By placing brick after brick you can build up interesting, custom creations that express creativity and individuality. Recipes and instructions can be followed to create pre-defined objects or you can use the blocks to create entirely new objects that have never existed before.

This is the idea behind the block programming application we created in this case study. This application uses a block programming interface and was designed to be used by both children and adults alike. The visual programming interface allows users to select and place blocks together in order to create computer programs called scripts which tell a model of our favorite office assistant what to do.

Users start off with a blank, infinite canvas that represents the computer's memory. Scripts are built by dragging and dropping blocks onto the canvas and connecting them together to form a chain of commands for the computer to execute. Once these scripts have been created, Clippy performs the actions defined by the user-created script on the giant stage in front of the user.

In order to manage the applications in our VR environment, we wanted to design an application launcher that would provide the user with an efficient way to store, organize, and select applications within the environment. Whereas the traditional taskbar on Windows operates to fulfill more basic functions, i.e., starting applications or managing open documents, our virtual reality version aims at being a persistent platform for users to interact with the applications they have installed.

The design of our taskbar aims to let the user quickly see which application is currently running, and also provides a quick way to switch between running applications without requiring the user to leave their current virtual environment or needing multiple screens. Since the theme we were designing around was that of a Tron-like programming bench, our app drawer was a cylinder that could be pulled up out of the desk to display the favorite applications. The icons for each of the applications was a tile-like cube with the 2D program icon displayed on top. Tapping or clicking on the icon activated the application. The user could slide these tiles out of the cylinder, allowing them to place it anywhere within their virtual environment. After the desired program was selected the drawer would collapse back into the desk.

Despite having the goal of developing for an intuitive, 3D environment, we as developers had a hard time getting rid of the usual terminal interface that we commonly use. Therefore we decided to create another example application for this environment that allowed us access to the command line. This application consisted of a small window that, when selected, would direct keyboard input into a command line interface. This interface communicated with a local server (external to the Unity project) which would receive the keystrokes sent from the VR environment and treat them as CLI commands and run them through a Javascript bash emulator.

The final aspect of these project we're going to cover is the custom keyboard gloves that act as a physical input mechanism that is more accurate and easier to use than virtual buttons. These gloves used capacitive cloth and thread to register touches from the skin on each of the exposed thumbs. The capacitive cloth and thread signals were transmitted to an Arduino board with a Bluetooth antenna module. The Arduino board was then paired with the computer using a wireless keyboard dongle and programmed to send alphanumeric characters to the computer.