Video Production Narrative Techniques in VR

This report offers the perspective of a team that worked alongside a video documentary production. That necessarily influences the frame of reference. The notes about shot composition and editing, for example, invoke comparisons with traditional video, as opposed to audio or the written word.

However, in VR, the director needs to manage all of this in new ways. This section outlines the tools at filmmakers’ disposal, and some notes on their evolution to this new medium.

Audio and Visual Tools for Narrative

Framing/Composition/Perspective

Narrative Change: Very high
Technical Change: Very high

These are the directorial elements that are different from most recorded, visual media. Traditional screen-based media works within a rectangular frame; filmmakers generally expect a landscape orientation, although the adoption of mobile platforms has recently pushed square and portrait compositions into consideration.

With head-mounted displays, viewers have the ability to look in any direction through a 360-degree sphere. The horizontal field of view at any one time is generally greater than 90 degrees, with high-end devices allowing for 110 degrees.

Therefore, in virtual reality, directors must expect to work with framing that is not fixed. This has implications for visual conventions, including the rule of thirds, left or right of frame, and even the concept of out-of-frame.

Likewise, traditional directors often work with perspective. They may be accustomed to positioning a camera so that the angle of a shot evokes a particular relationship between the viewer and subject; shots from below may indicate that the subject of a shot has more power, for example.

Editing

Narrative Change: Very high
Technical Change: Very high

In traditional filmmaking, cuts are a primary tool. But that changes for VR. In traditional filmmaking a shot can last a few frames or many minutes. Clips from different moments and locations are strung together, often in rapid succession, to convey meaning. (In VR the concepts of a shot and a scene converge. It seems the most natural word to describe a continuous piece of immersive, 360-degree video is “scene,” even though it’s made up of a single “shot.”) The authors are unaware of any VR works that have successfully used immersive scenes that are just a few seconds long. In VR, thus far, it appears that the only way to use a sequence of short shots is if the director layers 2D video inside a frame as a part of the immersive space.

This has significant implications for how documentarians do their work. If directors want to avoid relying on 2D video, or other overlays, it appears they will need to capture the action, or the interviews, as they will be experienced by the viewer; that is to say, without cuts other than the starts and ends of the scene.

Movement Within the Frame

Narrative Change: Moderate
Technical Change: High

A VR scene can include movement. Indeed a director can use moving objects or people to guide audience attention toward the important action in a scene.

However, some movement types tend to cause viewers more problems than others. Movements toward or away from the camera are effective and easy to watch, as are lateral movements through only part of the sphere; whereas an object or person traveling 360 degrees around viewers can frustrate them, as they feel compelled to swivel around to follow the action.

Camera Movement

Narrative Change: Moderate
Technical Change: Moderate

VR cameras move less than traditional ones. When they do move, they tend toward simple trajectories and steady speeds. Moving a VR camera initiates problems; it appears to cause greater nausea for users, and their feeling of immersion reduces. These are both likely to be the result of a mismatch between sensory inputs; the eyes are passing a message of movement, but the users’ other senses disagree.

Color and Lighting

Narrative Change: Low
Technical Change: High

The narrative functions of color and lighting don’t appear to differ very much between mediums. However, in current VR production, color and lighting are more difficult to manage than in traditional documentary filmmaking. This, however, appears to be an artifact of available technology, as opposed to a fundamental difference in the VR medium.

Technical difficulties stem from the fact that it’s a lot more challenging to find attractive and consistent lighting for 360 degrees around the camera. Also, 360-degree cameras necessarily capture light from multiple lenses to separate sensors. The Frontline production used a rig of 12 GoPros. As a result, each video file ran the risk of under- or over-exposure, and had a different color temperature from the next.

Focus

Narrative Change: Low
Technical Change: High

Filmmakers often use focus to direct viewers’ attention toward parts of a frame. While there’s nothing, theoretically, which makes that device unavailable in virtual reality, there are currently considerable technical and practical barriers to producing the effect in-camera. The GoPros used in the Frontline production had fixed, deep depths of field, so it was not possible to blur part of the view.

Stereoscopy is a new, related factor. The strongest stereoscopic effect is found when an object is around five to 15 feet away from the camera.

Graphics and Data Visualization

Narrative Change: High
Technical Change: High

VR provides great potential for the evolution of on-screen data visualization. This comes from the medium’s interactivity, the added three-dimensionality, and a new set of human computer interfaces.

Flat videos regularly show non-interactive charts, while interactive graphics have become common on desktop computers, tablets, and mobile devices. The human interfaces for those platforms are commonly limited to keyboards, pointing devices (a mouse for example), and touch via gestures. The VR industry has developed links to companies building more sophisticated human-computer interfaces including hand- and finger-tracking. This hints at new possibilities to manipulate data visualizations in VR, whether with natural hand gestures and actions (grabbing, turning, pushing, pulling, or with multiple fingers or hands), or by way of more intuitively understanding extra dimensionality.

Audio

Narrative Change: Moderate
Technical Change: High

In virtual reality, audio contributes a great deal to the feeling of immersion and, therefore, presence. In particular, audio that seems to come from the specific direction of on-screen objects makes the sensory feedback consistent and more believable. This appears to be true for dialogue, actuality, and foley sound. Directional audio is also a powerful tool for directing a viewer’s attention toward the part of the 360-degree scene where the filmmaker intends to show action.

Interactivity Tools

Virtual reality headsets, having computational functions, potentially give their users some agency and freedom. This depends on a range of factors, from the producer’s intent through to the computer power and memory, and the available human-computer interfaces (joysticks, pointers, et cetera).

Virtual reality researchers identify three high-level groups of interactions: object manipulation, viewpoint manipulation, and application control. These aren’t specific to journalistic virtual reality, but the following points draw on those.

Viewing Direction

This refers to the orientation and position of the user’s view within the virtual environment. The points above that discuss shot composition and camera movement show how viewing direction has implications for narrative.

Manipulating Objects

Although the virtual reality medium can allow users to manipulate objects, this may be the type of interaction that presents the greatest challenges for producers from traditional documentary backgrounds. For all practical purposes, this type of interaction is only possible in computer-generated environments (as opposed to camera-recorded video). It is therefore most commonly associated with games and open-ended experience

Navigating Through Content

Virtual reality producers can allow users to play content in individual chapters or segments. These can be made available one by one, forcing users to proceed in a linear path; or producers can make many segments available for users to access as they choose.