360Stories: The making of

2015-04-09 12.12.25

POSTED 2015.04.13 by

At Äventyret, we’ve been playing with virtual reality since the first Oculus Rift prototype became available back in 2013. We recently had a chance to build an immersive panorama viewer for Samsung’s virtual reality headset, the Samsung Gear VR, which incorporates the Samsung Galaxy Note 4 to run Android apps built with Oculus’s software development kit.

Our client: 360Cities, the world’s largest repository of 360-degree panoramic images. Äventyret developed the concept, while our dev partner Prototyp wrote the code. The result was 360Stories, an app that lets photographers add voice-over narration to their immersive panoramas. We learned a lot from the process; read on for some of our take-aways.

The concept:
At its core, 360Stories is a panorama-based storytelling app. You use it to download and play stories showing immersive panoramas accompanied by voice narration and ambient sounds. But the app’s real distinguishing feature is that anyone can build these stories, because 360Stories supports an open XML markup language called RPL. Just as a web browser renders pages built with HTML markup, 360Stories renders stories built with RPL markup. You can download ready-made stories from 360Cities, or you can make your own and upload them to the app.

One thing we know from previous projects developing content for immersive environments (starting with a project for Second Life back in 2007) is that you don’t want to convey information via walls of text. Reading text in a virtual environment is tedious and inefficient. Instead, it’s much better to let the user focus on exploring the environment while listening to a soundtrack of relevant information.

Technical challenges:
One challenge when developing an app for a prototype of an “Innovator Edition” of a consumer product is that you’re building something for a moving target, both software-wise and hardware-wise. We experienced this with the beta of Oculus’s software development kit (SDK). For making games, many developers are rightly choosing to use the Unity 3D rendering engine on top of the Oculus SDK, because it provides a mature, additional layer of abstraction, shielding developers from an SDK still in flux.

2015-04-09 12.17.02

But Oculus’s own SDK comes with some highly optimised built-in panorama rendering tools, and it was these that we decided to build upon, only to find half-way through the process that their core implementation changed in an update to the SDK. Still, despite the extra work, the end result is a very smooth render.

Outside connectivity:
Originally, our intention was to let users find playable files anywhere on the web using a web browser. When downloaded, these files would then open in 360Stories. This wasn’t possible: Apps that work with the Gear VR are sandboxed, so that they are only usable within the context of the Gear VR and the Oculus “Home” launch app/store. This isn’t a problem with apps such as games, where the content to be explored is fixed. But we wanted 360Stories to be a universal player for a special subset of web-based content—how to connect this to a sandboxed app?

2015-04-09 12.17.51

Our solution was to hard-code an RSS feed containing playable content files into the 360Stories app. This way the app can get regularly updated content without us needing to update the app itself. That main RSS feed is controlled by 360Cities, but anyone can create and upload a playable file to the 360Stories app, and then play it by launching the app via Gear VR.

2015-04-09 12.18.00

Future development:
Right now, befitting the minimum-viable-product (MVP) nature of the 360Stories app, the RPL markup supports a limited number of features and content types, such as popup-text captions and stereographic (3D) panoramas, but not immersive video, directional sound, or captions fixed at specific spots to the panorama. But the RPL specification was written to anticipate such developments. There is also much that can be done to improve the rather basic navigation screen. And finally, current panoramas are rendered from equirectangular files, which generates discontinuities at the nadir and zenith. Future versions of 360Stories will hopefully also support cubic projections.