History of the 'BBC Redux' project
At Mashed in August 2008, we showed developers an internal research project called "BBC Redux". At the time, it was mentioned in the Guardian and on Click, and you may have wondered what it is.
In the summer of 2007, freetards (me too), the OSC and others were calling for the BBC to make iPlayer cross-platform, and the Trust had committed the BBC to doing this.
Slated for a Christmas launch, the system only supported Windows Media with DRM and P2P delivery. There were some concerns from ISPs about how this might affect them.
Cross-platform support has always been a source of grief [1] leading us to duplicate systems and to reduce functionality in order to support multiple formats equally.
The problem wasn't going to go away and with the expansion of video and audio into devices such as games consoles, it was getting worse.
We needed a different approach, to divorce our content production from delivery format and method - something quickly adaptable to new devices and the unmentionable "C word": convergence. [2]
I'd had an idea for a system to do this festering for a few years - the mother of all VoD systems. I'd proposed this at the start of the iPlayer project and now it seemed time to just get on with it.
Armed with a couple of trusty developers, Tom and Dickon, a demonstrator was quickly built. We called this project "BBC Redux".
It's a video-on-demand test bed where we can try out the systems that acquire, store, search and deliver content. It's like your PVR but much larger and can convert and stream content to suit playback on more than just TVs.
So how does it work?
To start with we ingest DTT, though sourced from multicast, as direct programme feeds take time to organise. There's a server that records each stream and segments it into programmes just as your PVR does.
Each content recorder (there may be more than one of each subsystem for redundancy) offers its files to the content manager(s) that look after storage distribution and management. The content manager(s) determine which content store to save them on. Then there are front-end servers to test applications for delivering to users and a stack of transcoders that are the glue between the neutral high quality storage format and what the users want.
In the summer of 2007, we showed our prototype to Ashley and the FM&T board.
The point was made (streaming video could work) and iPlayer was set to have Flash streaming for the Christmas launch (after some quick work by the iPlayer team). A bonus for the ISPs is that Flash streams are approximately a third the size of the downloads and people only transfer them once. Currently, around 90% of BBC iPlayer use is Flash streaming.
We also demonstrated to the board a set-top box that plays our multicast live streams and iPlayer VoD - part of the longer term aim to make internet TV more usable in the home.
We continued to experiment with BBC Redux, making it work for various games consoles - PS3; PSP; Wii - and when the iPhone came out in November 2007, we added iTouch/iPhone and later 3G phones.
Some of this work was later used for iPlayer beta sites (iPhone is 3% of use).
At Mashed08, we let the public see and use one of our development front ends for the first time with the freshly-hacked that week subtitle support - so new it was still being worked on through the Saturday.
Redux was built to support the rapid development of new services and was designed to scale in many directions.
The transparent on-demand transcoding is how we can quickly adapt to new devices. When iPlayer chose to go Wii, they wanted a slightly different encoder profile than the one we'd used for demos. No problem: we defined the coding profile, allocated a subset of the transcoder pool to them and let them trawl through the store collecting all the programmes they needed for the seven day window. A few iterations were needed as the launch deadline approached, so more transcoders were added to their pool as time ran out. The risk was too high to try all this in the iPlayer live production systems; they could be adapted later.
Redux has been quite versatile - over the past year it has grown in use as a demonstrator of a tapeless world - a way of trying out ideas like:
- instant access programme compliance store
- immediately accessible archive - no delays waiting for a DVD/VHS in the post, so programme teams are using it for research.
- a usable infinite archive - why would we ever throw programmes away again?
- adaptable to produce new content formats on demand - when bbc.kongjiang.org/www.bbc.co.uk added WM alongside Real, nobody had the content readily available to make WM versions of all the previous content; now it's not a problem
- IPTV
- closed user group access to content, either in production or, as is now being developed, for TV listing magazine reviewers who used to watch on postal DVD
Redux consists of three racks of equipment - two are storage nodes - 342TB according to the disk manufacturers, 297TB of usable space with 152K programme files so far.
Full size images in our Flickr account [1 | 2]
There is also a baby one feeding iPlayer as, being a development system, we like to break the main one regularly.
Redux had its first birthday in July. We hope it will have many more before the experiment ends.
§
[1] Used since 1995 and selected as the most cross-platform system we could find, Real still met resistance from some Windows users: often corporate IT wouldn't let them install it or they and Linux users didn't like the free player trying to entice them to upgrade to a paid-for version or use advertising.
BBC TV and Radio services work on any manufacturers device, they compete on quality and features, not on exclusive access to content. It seemed silly to have to make special content for any particular internet device manufacturer but that's how the market, governed by commercial rather than technical interests, has developed. [Return to post]
For years, we've encouraged the adoption of standards-based systems, developing DIRAC, a royalty-free video codec to reduce commercial barriers to adoption of a common standard. It's starting to make ground in high end broadcasting, but is slow progressing to the internet. In the meantime, we've been supporting H264 and AAC, which are taking off. The aim remains - people can choose their player; we don't need to care.
[2] Internet media are still much like that start of TV broadcasting where multiple systems were run in parallel before choosing one. [Return to post]
Brandon Butterworth is Principal Technologist, Kingswood Warren, BBC FM&T. Image of Brandon by Chris Capstick. Image of protest by Matt Cashmore. Images of Redux kit by Brandon. UPDATE 24/9/09: Brandon Butterworth is now Chief Scientist, BBC.
Comment number 1.
At 15:00 27th Oct 2008, Briantist wrote:Thanks for the article, I always enjoy pictures of racks!
The best bit, of course, would be that if a decision is taken to allow the iPlayer to have a "long tail", it could be there "at the flick of a switch"...
Complain about this comment (Comment number 1)
Comment number 2.
At 15:12 27th Oct 2008, Ed Lyons wrote:Why are some programmes not available on the iPhone, even quite some time after broadcast? E.g. Episode 8 of Lark Rise over Candleford (broadcast Saturday evening) only became available this morning, while Episode 7 (broadcast earlier the same evening) still isn't available...
Complain about this comment (Comment number 2)
Comment number 3.
At 15:55 27th Oct 2008, Jon Jacob wrote:Thank you Brandon. It's a pleasure to read about a system I use regularly and admire from afar. Please don't change anything about it. ;)
Complain about this comment (Comment number 3)
Comment number 4.
At 04:30 9th Sep 2009, felicioo wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 4)