Exploring film soundtracks with Radio 2 and BBC R&D

With developments such as Bamzooki, the Piero Sports Graphics system and the BBC Election Swingometer, BBC R&D has a long history of developing new ways of mixing video with graphics in order to offer new ways of presenting interesting content to the viewer. An earlier version of Venue Explorer, our experimental web application, was trialled at the Glasgow Commonwealth Games and we are now trying it out with a series of concert sessions and a special episode of Radio 2's Friday Night is Music Night.

We first trialled the concept at the Glasgow Commonwealth Games in 2014, and we have since been working with the BBC Philharmonic and BBC Concert orchestras to develop the technology further. As well as offering extra information to the viewer in the form of graphical overlays describing the performance, both of these recordings also include audio ‘zooming’.

As you zoom your view into the video you will notice the audio mix changing, highlighting the playing of the section of the orchestra which you are viewing.

The server uses NodeJS and Express to deliver the web application, and SocketIO to act as the real-time messaging interface between the editing tool, control interfaces, server and client browsers. We currently deploy instances of the service using Amazon EC2.

This technology stack provides us with a simple deployment route, as well as the ability to handle both recorded and on-demand video services. The real-time comms channel via SocketIO/WebSocket allows live control over the service - triggering, adding or editing the graphical overlays during a streamed performance for example, or displaying live data from an external source, such as a musical score follower.

Audio Zooming

To produce the audio remixing effect, we capture multiple audio channels from the BBC outside broadcast truck.  Rather than streaming the stereo broadcast mix to the browser we split the audio into its component parts which when combined (or summed) produce the stereo mix.

This means that at any time we can be streaming up to 24 channels of audio to the browser. The streams are generally separated by instrument group, for example, Piano, First Violins, Harp, Woodwind...When you zoom in or pan the video we track the zoom-level and pan-position in the video scene, and modify the loudness and spatial position of the audio sources or objects in response. Static sources such as the overall sound of the venue are louder when zoomed out and quieter when zoomed in.

Conversely, dynamic sources such as individual instrument groups are louder when zoomed in and quieter when zoomed out; and louder when closer to the pan-position of the user. The level of the dynamic sources in the mix is adjusted to reflect the distance of the sound source from your pan position in the scene.Audio processing is all carried out in the browser using the Web Audio API.

Try for Yourself

 @KermodeMovie explains how you can play around with some of the most iconic film soundtracks of all time.  https://t.co/EqE6QLVHkC pic.twitter.com/0PSjZBjUJz

— BBC Radio 2 (@BBCRadio2) 10 February 2017

Friday Night is Music Night Remixed

Recently, Venue Explorer has been used to create a demo with the BBC Concert Orchestra, called Friday Night is Music Night Remixed. Three pieces of music were recorded for it (Star Wars, Jaws and Jurassic Park) and all are available to view.

BBC Philharmonic - The Red Brick Sessions

We have been working with the BBC Philharmonic to offer enhancements to the Red Brick Sessions concert series. Recordings are available for 30 days on our concert server.

• Tweet This - Share on Facebook
• BBC Radio 2 - Friday Night Is Music Night (Full Episode)
• BBC Music
• BBC R&D - How the Commonwealth Games is defining the future of broadcasting
• BBC R&D - Augmented Video Player