In the beginning there was the web and to find things on the web you need a search tool Webmasters built their sites based on what they thought people wanted to see with a minimal of SEO. Search was pretty easy at the start but as the internet evolves and grows into a viable commercial avenue for business as well as a source of the worlds information search gets harder to manage and the stakes get much much higher. As the commercial potential of the web is now being harnessed,  money is a primary motivation for an online presence for so many companies this has fuelled and driven search technology into uncharted territories in terms of indexing and storing the worlds information. Be it commercial motivation or the information freely available over the internet itself  the battle over the top places in the "search engines" had begun in a very big way. SEO became a marketable skill and somewhere in the battle for the top spots the people were quickly forgotten and sites were and still are being built to suit the search engines - machines. As a result of this a whole new set of problems has evolved with the ongoing battle against "search engine spam" spiraling out of control as millions of webmasters try and second guess every measure google takes to curb the over optimization of websites. The message is quite clear that trying to cheat the new search technology will get your site banned or panelized - maybe. With the advent of super sites, massive database driven auto generated sites designed to capture as much free search traffic as possible things have only heated up further, Google creates a revenue model - Adwords - and allows webmasters to sell the adwords product in the form of adsense via their websites. This in turn  motivates webmasters to get as much free traffic as possible and as Google is the single best source of free web traffic these same webmasters are building more SE spam! So in effect Google is now funding part of it's own problem. We now see quite a volatile search index where google is concerned, they are constantly updating their technology to counter the spam problem but as they change things with their algorithms, storage and crawl technology it would seem to many legitimate webmasters  things just become more unstable and less predictable. Google makes a change and whole industries spring up - Link building as an example is an industry based on getting inbound links to websites to help your SE rankings in particular. This applies more to Google because it's key in their search algorithm. Moving forward search relevance is still and will always be the most important factor in the survival of any search engine including Google. Their ability to overcome storage and crawling issues on such a large scale may be a telling factor, recent volatility in the Google serps indicates things are not perfect,  they may be a giant but they also face a giant challenge.