Songverse - VR Music Creation Tool

About the Project

Songverse was an independent project with the goal of exploring VR technology to create a musical authoring tool for non-musicians, based on a cosmic metaphor: a Musical Universe.

A Digital Musical Instrument (DMI) can be defined as hardware-software solutions that are interactive and crafted to output sound according to users' input. Songverse allows users to create music by interacting with an environment designed to resemble outer space. By adding systems, planets, and satellites to the virtual environment, the user can shape the produced sound through interactions that were extensively tested during the development phase.

Concept: A playful and immersive music creation tool for non-musicians

Platform: High-end Virtual Reality Head-mounted Displays

Tools: Unity3D for development and Tilt Brush for modeling

Process

The goal was to build a VR musical application that was easy to use, providing an enjoyable experience that didn't demand any technical music knowledge from the user.
‍

Research

We interviewed people with a music background to understand how to abstract the music knowledge and make it work for our concept.

We also reviewed related works published and other existing music creation applications such as Reactable, Illusio, VRMin and IncredBox

‍

Ideation

Techniques such as Bodystorming and Sketching were used in this phase to explore ideas of concepts/interactions for a music creation metaphor

The chosen concept was a musical universe with celestial bodies representing instruments, sounds, and so on

‍

Prototyping

Several VR instruments and composition mechanics were created to explore how to provide an interesting authoring experience.

the more custom the sounds could be, the harder to start getting decent music results, so we decided to provide samples instead of letting the user compose the sound from scratch

‍

Validation

We then evaluated the instrument with musicians and non-musicians to validate the concept and interactions by interviewing and applying the System Usability Scale (SUS) to assess the easiness of people to create music using Songverse.

‍

‍

Result

In the final version of the project, to compose the universe, the user can insert stars, planets, and satellites orbiting around each other and every celestial body is a musical metaphor.

Users can create the base track composition through the addition of systems, planets, and satellites to the universe. Systems serve as effects that the user can apply to the samples coming out of it, transforming the sound with effects such as reverb and chorus. Planets are placed inside each system, and each planet represents a specific type of sound, such as drum beats for rhythmic planets, guitar, and piano chords for harmony planets and guitar riffs for melodic planets. Each of these planets serves as visual cues to the user, since they do not produce sound by themselves, but serving as anchors for satellites.
‍

‍

To create the musical rhythm, the user can select a drums sound planet, and different kinds of samples (satellites) can be attached to it. By design, all the presets available can be freely combined without friction or losing the musical coherence of the planet or the universe as a whole.

The tools menu. The first row (bottom-up is responsible for creating systems, the second row contains the planets and their types, and the top row are the satellites.

‍

Interaction

Songverse was designed for high-end HMDs with 6 Degrees of Freedom controllers. On one hand, users can see a menu palette with all the stars, planets, satellites available and all the universe can be controlled with raycasting and gestures.

Placing a planet in the scene creates an orbit around it, which the user uses to place satellites, the elements that generate sound. By dragging a satellite with the controller’s raycast to a planet, it starts to orbit around the planet, and, from the next beat’s turn onward, it begins producing one of the samples associated with the planet. The system plays these samples in a loop, and the user manipulates them from a distance using raycasting

‍

‍

Visuals

At the beginning of the development process, we decided that we did not want to replicate a real-life universe, abstracting from its physics laws and appearance. Our idea was to create a playful and entertaining experience, taking advantage of the inherent proprieties of VR, such as immersiveness and spatiality.

‍

All the modeling was done using Google's Tilt Brush. That allowed us to quickly create different styles for each celestial body, with paint aesthetics and particles/lighting that reacts to the sound.

Songverse project was published at 2019 21st Symposium on Virtual and Augmented Reality (SVR) and ran for best paper: Songverse: A Music-Loop Authoring Tool Based on Virtual Reality

‍