Google is keeping a close eye on a small, suburban Melbourne start-up that claims to be developing a search engine that improves on the world leader. MyLiveSearch, which plans to go live as a public beta in a few weeks, is the brainchild of self-taught software developer Rob Gabriel.
Mr Gabriel, 35, says his search engine gives better, more relevant results than the search king because it is a truly “live” search. Google’s search engine works by building a vast index of web pages, via automated “spiders” that crawl through billions of web pages a year. However this represents only a fraction of the enormous, sprawling internet, and the index can never be entirely up to date.
Yahoo! search and (despite its name) Microsoft’s Live search work the same way. MyLiveSearch is fundamentally different. It works through a small browser plug-in. The search terms are put through Google, or other indexed search databases, but those results are treated as “starting points” alongside the user’s bookmarks and other popular web hubs.
From there, the live search takes over, crawling through hundreds of web pages connected to those starting points in search
of more information relevant to the search. Mr Gabriel says the results come back in seconds, and are almost always richer, more detailed and more useful than a standard, index-based search. His product can also search the so-called “invisible web” of dynamically-generated web pages that search engines have trouble indexing.
“This has the potential to change the way people search the internet,” he said. “Google can’t search every page every day (to
build its index). The web is so dynamic and changes so often – MyLiveSearch turns your own computer into a ‘super-spider’ to search it in real time.
“This technology could be snapped onto any of the major search engines and improve them.” A Google business development representative has met with the MyLiveSearch team at least twice – once when the technology was at a very early stage, and again last week after Next made inquiries about Google’s interest in the project.
Google did not want to comment on MyLiveSearch, except in general terms. “We’re really keen to support Australian developers and IT entrepreneurs,” spokesman Rob Shilkin said. “Google Maps was a small Aussie start-up. And Google itself started as two guys in a garage.”
The company has a history of acquiring technology that might help a competitor. In April last year Google bought the rights to
an advanced text-search algorithm invented by University of NSW student Ori Allon. Mr Allon, 26, was given a job at Google’s
headquarters at Mountain View, California.
Mr Gabriel said the meeting last week inspired him to go to Silicon Valley next month to spruik his technology. “It started as
a dream and the dream is slowly developing into a reality far greater than all my expectations,” he said.
Mr Gabriel has worked on MyLiveSearch for two years, along with his younger brother, Mark, and systems engineer Mende Jurukovski – though the original idea came to him 10 years ago. Live search is one solution. Despite Google’s immense computing
resources, the ranging “googlebot” cannot track updates to the web in real time.
The more popular websites are “recrawled” more frequently. Outside the net mainstream, however, visits from the Googlebot are more rare. Even sites that are actively trying to promote themselves can take weeks or even months to turn up on Google’s
index.
Some estimate that even Google has only indexed less than a fifth of the internet.
One solution to this problem is “live indexing”, where pages are submitted to the search engine and instantly indexed. Another solution is ad-hoc distributed web crawling. The open source Nutch project hopes that tens, hundreds or even thousands of
Nutch-powered, specialised web search engines could combine into an index more comprehensive than any single proprietary site.
