(DRAFT) Project Wonderland HUD

1. Overview

This document describes two components which collectively provide HUD ("Head Up Display") functionality in Wonderland. We're making a distinction between the HUD display surface and the UI Toolkit used to construct interactive user interfaces which can be displayed in the HUD and elsewhere.

This document covers generic HUD requirements, and then drills down into a particular implementation for Project Wonderland. Except where noted, it is not a commitment to the functionality of a particular Wonderland release. In 0.5 the HUD will be functionally similar to the HUD in Wonderland 0.4, though the look and feel will differ and many of the limitations of the 0.4 HUD will be removed.

Note to reviewers: HUD visual design is not part of this design document. General design concepts are covered, but the look and feel of the HUD images in this document are only to help illustrate concepts. Don't review the look and feel.

1.1 HUD

A HUD is a 2D, rectangular, transparent overlay that floats above the 3D scene graph. It's a surface upon which the Wonderland client can display information, controls and images. Note that multiple HUDs may share a single application display.

A HUD API provides a set of interfaces that developers can use to create, modify and display HUDs.

1.2 UI Toolkit

In conventional applications, the HUD provides a toolkit for creating controls and for displaying them on the HUD. In Wonderland, it's sometimes useful to display these controls in-world as well, so we're providing a UI Toolkit which can be used to create controls that are displayed in-world and on the HUD. Controls can be moved from the world and "pinned" onto the HUD, and vice versa.

2. HUD Use Cases

The Wonderland HUD will support the following use cases:

2.1 Informational Messages

These are messages that inform the user about changes in the state of Wonderland, Wonderland applications or users.

For example:

Figure 1: a status message when loading a document

Figure 2: a page change notification

These are short-lived messages that appear and disappear on their own. The user doesn't interact with these messages. The developer can choose whether informational messages animate on and off screen, fade in and out, or just appear and disappear instantly. This behavior can be customizable by the user by setting HUD preferences.

2.2 Error Messages

When something goes wrong, an application needs to display an error message and explain what to do to resolve the issue.

Figure 3: application error message

Error messages can be modal or non-modal. Error messages can contain multiple controls (e.g., a text message and a button.)

2.3 Application Controls

These are HUD elements with buttons for controlling an instance of an application.

Figure 4: video player controls

These controls appear when the user has control of the application, and disappear when they release control. Users should be able to hide these controls if they'd rather use direct manipulation to interact with the application or use keyboard/mouse controls.

2.4 Context Sensitive Help

This is mouse over help, like tooltips in Windows, where you mouse over an object and a brief message appears. These are a type of Informational Message (see 2.1), but might have a different appearance or behavior.

Figure 5: application control panel tooltip

These appear on mouse over and fade away quickly when the mouse focus changes.

2.5 Application Help

Applications might need to display more extensive help, for example detailed usage information. Detailed help might be displayed in a scrolling text overlay with navigation controls.

Figure 6: help browser

2.6 Application Best View

Best View mode allows a user to see an in-world application face on, in the same way that they view applications running on a conventional 2D desktop. In best view mode, a duplicate view of an application window is displayed on the HUD. Users can interact with the application just as they would in-world and changes made in best view will be reflected in the in-world application. In Wonderland 0.4, best view simply moved the user's camera to an optimal in-world viewing position.

Figure 7: application in best view mode

2.7 Additional Use Cases

The following documents contain additional HUD use cases:

• Proposal for Voice Calls in Wonderland
• Idea for Improving Presentations in Wonderland (to be published)

3. HUD

A HUD is a rectangular 2D display surface that overlaps part or all of the 3D scene graph. A HUD is visually transparent by default and passes keyboard and mouse events through to the scene graph. In the simplest case, a single HUD covers the entire 3D scene graph, but it's also possible to divide the screen into HUD regions. HUDs can also overlap. For example, a full screen HUD might be displayed above all other HUDs to display a blocking dialog, such as the Wonderland login dialog. HUDs can request a particular offset from the default stacking order. The visual and event transparency of the HUD can be changed.

A HUD is a display surface for UI Toolkit Components. User interfaces created with the UI Toolkit can be displayed on a HUD:

Figure 8: UI Components displayed on a HUD

HUDs can be fixed or variable sized. For example, a task bar HUD might occupy 100% of the width of the application window and 10% of the height. So, when the application window is resized, the HUD scales proportionally. HUD position can be specified in absolute or relative coordinates with affinity to the center, top, bottom, left and right of the application window.

In Figure 9, the login dialog has fixed size and is centered in the upper HUD. The upper and lower HUD areas (outlined in red) are variable size. For example, the lower HUD occupies 100% of the width of the application window and 15% of the height. When the application window is resized the login dialog remains the same size and centered, while the upper and lower HUDs scale proportionally.

Figure 9: HUD scaling

A pluggable layout manager controls the position of multiple HUDs.

The number and configuration of HUDs is under the control of the developer. Wonderland will have a particular default configuration, and this is described in section 6.

4. User Interface Toolkit

The UI Toolkit provides a set of objects from which developers can construct interactive user interfaces. The Toolkit is loosely modeled on Java's Swing Toolkit.

4.1 Components

A Component represents an interactive visual element such as a button or a text field. All UI Toolkit objects are Components.

4.2 Container

A Container is a special type of Component which can contain other Components. Just like the Swing toolkit, Containers are Components, so Containers can contain other Containers as well as Components. A login dialog is an example of a dialog that might be constructed from a Container and multiple Components.

Components are laid out within a Container with a pluggable layout manager. The UI Toolkit will provide some basic layout managers.

The UI Toolkit will provide a single Container type:

• Panel

The UI Toolkit will provide a set of Panel subclasses for building common types of Panels, such as horizontal and vertical control panels.

4.3 Component Types

A Component is an interactive visual element that can be added to a Container. For example, a text message, a button, or an icon. Ultimately, the UI Toolkit will support a range of Components, from which developers can create a wide range of user interfaces.

The initial list of Components is:

• Button (with text or image label)
• Radio Button
• Label
• Text Field (editable)
• Image
• Scrolling List (containing text or icons)
• Menu (text list)

4. User Interface

4.1 General concepts

The HUD user interface design will use design cues from desktop and web applications that are familiar to users. The goal is to make it easy for users to recognize the types of controls visually. For example, it should be clear whether a component is a button, whether it is enabled and can be clicked. A text field should be easily distinguished from a text label so the user knows that they can type in it. Interactive controls should respond to user input. For example, a button should react in some way to indicate that it has been pressed. HUD Components are 2D. Although HUD Components might have a pseudo-3D appearance, they are not 3D objects. This HUD specification does not specify the design of in-world controls for 3D applications.

Since the HUD overlays the 3D scene, transparency can be used to give the user a clear indication of the extent of the HUD control area, while allowing them to see activity under the HUD.

4.2 Visual identification

The HUD is used for a variety of purposes, for example, to display transient informational messages, to show application control panels, to pop-up modal dialogs etc. The visual design of these different HUD interfaces should clearly indicate the type of the interface. For example, an informational dialog might use a particular color scheme or utilize a "notification area" of the HUD. An application control panel might have a horizontal layout and a close button. A modal dialog might reduce the transparency of the entire HUD to "gray out" the 3D scene to indicate that the dialog will block interaction with the 3D environment.

4.3 Animation

Most desktop operating systems use animation to provide visual cues such as to indicate the source of an event or to show that a user has focus over an application in the desktop task bar. These techniques will be used in the HUD for similar purposes.

4.3.1 HUD Panel Animation

The HUD will support following types of animation of HUD Panels:

• Slide in/slide out
  In this animation, a HUD Panel will slide into view from off screen. This might be used for informational messages, which appear on screen for a limited period of time and then auto-dismiss. By using animation, the user is not forced to click and dismiss the dialog. The origin, direction of slide, destination and the timing of the animation will be under the control of the developer and configurable as a user preference.
  
• Fade in/fade out
  This mode fades the HUD Panel into view at its final location. The panel would appear and disappear gradually by reducing transparency of the entire panel and its controls.
  
• Zoom in/zoom out
  In this mode, the HUD Panel zooms in as if flying towards the screen from an infinite origin.
• Instant appear/disappear
  HUD Panels can appear and disappear instantaneously. This mode might be used to display long term status information such as the avatar's mute status where the user needs instantaneous feedback of a change of state.

4.3.2 HUD Component Animation

HUD Components will support a simple built-in animation:

• Focus animation
  When any HUD component receives mouse or keyboard focus, it will animate to indicate that it has focus. Animation might be used to differentiate inactive controls from active controls.

Developers will be able to implement their own custom animations of HUD Components by using a standard Java Timing Framework. So, if a developer wanted to create an icon that flashes, they could use the framework to trigger changes in the visual appearance of a HUD Component.

5. Layout Management

HUDs and HUD Panels both support pluggable layout managers which control how HUD Components are laid out. Two types of layout will be supported initially:

• Flow layout
• Absolute layout

5.1 Flow Layout

With Flow Layout, HUD Components are placed within the HUD or HUD Panel in a simple horizontal or vertical flow. Components are placed left to right or top to bottom as they are added to a HUD container. The width or height can be constrained, so when there is insufficient room to place a component within the fixed width or height, the next component "flows" to the next row or column just as text does in a text editor. The behavior of the Flow Layout is similar to the FlowLayout in AWT. Flow Layout is most suitable for control panels where multiple buttons are placed side-by-side. The goal of Flow Layout is to make it easy for developers to place controls without having to calculate correct pixel dimensions.

5.2 Absolute Layout

Absolute layout provides developers with ultimate control over the placement of components within a HUD or HUD Panel. Components are placed at pixel coordinates within the container.

6. Wonderland HUD

In Wonderland 0.5, the default HUD configuration will comprise a number of HUD regions:

Figure 10: HUD regions

6.1 System HUD

The System HUD is used for system dialogs, alerts and other system information, including:

• login dialog
• server disconnected dialog

It occupies the entire Wonderland window, and is at the top of the HUD stacking order, overlapping all other HUDs.

The System HUD supports blocking and non-blocking modes. In blocking mode, events are only delivered to HUD Components in the System HUD. In non-blocking mode, events are passed through to underlying HUDs if not consumed by HUD Components in the System HUD. The login dialog is displayed in blocking mode, for example.

6.2 App HUD

The App HUD is available to Wonderland applications to use as they wish. For example a Wonderland app might use this HUD to display a configuration dialog. Application control panels (e.g., for the PDF Viewer) can be displayed here.

This HUD occupies the full width of the window, overlapping the Status and Control HUD. It abuts the Task HUD at the bottom of the window.

6.3 Task HUD

This HUD is similar to a task bar or dock in a desktop operating system. It might display a horizontal list of icons that can be launched, as an alternative to the Shared Apps menu in Wonderland 0.4, or it might show thumbnails of applications the user has launched or currently has control of.

The Task HUD occupies the lower portion of the window.

6.4 Status and Control HUD

The Status and Control HUD occupies the right edge of the window. It overlaps and is above all HUDs except the System HUD. This HUD displays status information, including:

• compass/map
• user list
• the mute state of the user's avatar
• frames per second indicator
• ping time

7. Changes from 0.4

The following secondary windows and controls will be replaced by HUD components:

• Login dialog
• Server disconnected dialog
• Collaborative 2D application file open dialogs
• Chat panel
• Compass/Map
• Users list
• Audio controls (mute, raise/lower volume)
• Phone dialog (dial out, unlock)

Items highlighted in green will be supported in 0.5.

The HUD will address a number of issues with the 0.4 HUD including:

• Improved handling of multiple, simultaneous informational messages. In 0.4 when teleporting to a region containing several PDF Viewers, each viewer would display an informational HUD message which overlapped with those from other PDF Viewers. The new HUD will scroll these application messages.
• A HUD layout manager will place HUDs automatically rather then developers having to specify the absolute screen coordinates of their HUDs.
• The HUD API will be simplified.

8. Issues

• In games such as Little Big Planet HUDs are displayed in-world, attached to the avatar. Should the HUD support the placement of HUD Panels at arbitrary places in-world? This is certainly far more flexible than limiting the HUD to a 2D overlay that's always next to the screen.
• Should the HUD toolkit allow for the creation of custom components? How are new HUD Components made available?

9. Further Information

• HUD Research Notes
• HUD Feature List
• HUD API Specification^?
• HUD Graphic Design Requirements^?
• HUD Graphic Design^?