Developing Search At The BBC - Pt 2
Part 1 is here. This post is part of the tenth birthday celebrations of bbc.co.uk
When he was at Ask, Tuoc Luong once put the demands placed on search engines very succinctly:
"We need to read users' minds."
He was referring to the tricky job of interpreting the very small amount of information that the user gives when the type in a search query. Does someone looking for "jam" on the BBC website want some recipes, to listen to a documentary featuring Paul Weller, or to find out what the traffic will be like on their way home?
The "lilac" search results I described in the first part of this post only lasted for a few months, as in 2002 there was a major push to integrate web search within the BBC's search engine. Previously, as part of the BBC's remit to be a "trusted guide to the web for the UK", there was a directory of links called WebGuide - a small-scale Yahoo!-style directory which linked to sites of UK interest.
By now though, Google was indexing billions of web pages, and it was obvious that maintaining a small directory of sites wasn't going to scale up to the growing size of the internet. The BBC turned to search as a tool, trying to provide a web search that was safe for children, promoted UK content, and which, unlike all of the major commercial search engines, didn't take advertising in return for prominent placement.
In the summer of 2002 this launched as the main search box on a newly designed BBCi homepage. Getting involved in web search wasn't a popular move in some quarters. Commercial sites saw it as a threatening land-grab, and the BBC was accused of artificially inflating the ranking of BBC content within the results.
Not everyone was that keen inside the BBC either and for the next few months, virtually every presentation or meeting I went to started with someone ranting at me that when they searched for 'x' using the BBC homepage, all they got were external sites, and not the content about 'x' that they had just published.
In his 2004 DCMS review of BBC Online in 2004, Philip Graf wrote that:
"Respondents, in our audience research, could not clearly identify the value of the BBC search engine. Many were content with commercial tools, such as Google. On balance, however, I feel the BBC should retain its search engine. Given that search is becoming such a fundamental part of how the internet is used, it is worth keeping a publicly funded, UK competitor in the market place."
The move into web search could not ultimately be described as a success. It is now tucked away behind the fourth tab on a BBC search results page, and has no direct search box entry point on the site. Other search developments have been more welcome, however.
CBBC Search provides a safe environment for kids to start learning about searching the internet, with parents able to feel safe that nothing bad will be lurking in the results. And in 2004, in one of the first examples of a public BBC "beta test", News launched an audio/video component in the search results. With the arrival of the iPlayer, the ability to search through video content is a crucial future development for the BBC.
The last significant redesign of the BBC's search systems took place in 2006. This unified all of the BBC's content in one massive 1.5 million document index, and removed the concept of 'scoped' searching of just one area of the site. Finally, the technical and organisational hurdles had been overcome, and wherever you were on the BBC site, a search should produce the best results, from the whole range of BBC content, including audio and video clips.
It isn't quite a mind-reader yet - but the BBC's search engine has come a long, long, way from the days when I had to edit the results by hand!
Martin Belam is a former Senior Development Producer, New Media
Comments
"This unified all of the BBC's content in one massive 1.5 million document index, and removed the concept of 'scoped' searching of just one area of the site."
Except that there's still a "News & Sport" scoped search. Which still confuses me by not being the default when you search from within the BBC News site...