How Google crawls, indexes, and serves the web
When you sit down at your computer and do a Google search, you’re almost instantly presented with a list of results from all over the web. How does Google find web pages matching your query, and determine the order of search results?
In the simplest terms, you could think of searching the web as looking in a very large book with an impressive index telling you exactly where everything is located. When you perform a Google search, our programs check our index to determine the most relevant search results to be returned (“served”) to you.
The three key processes in delivering search results to you are:
- Crawling: Does Google know about your site? Can we find it?
- Indexing: Can Google index your site?
- Serving: Does the site have good and useful content that is relevant to the user’s search?
Our suite of webmaster tools provides you with a free and easy way to make your site more Google-friendly. They can show you Google’s view of your site, help you diagnose problems, and let you share info with us to help improve your site’s visibility.
Getting Google’s view of your site, and diagnosing potential problems
The first step to increasing your site’s visibility on Google is learning how our robots crawl and index your site.
· Crawl info: You can make sure we have access to your site, and see when Googlebot last visited. You can also view URLs that we’ve had trouble crawling and why we couldn’t crawl them. This way, you can fix any problems preventing us from indexing all of your pages.
· Robots.txt file validation: See if we’re having trouble with your file, and test out changes to that file before you change it on your server.
· Website content: View top content from your site and see the words that other sites use to link to it.
Seeing how your site performs
A second step is learning what drives traffic to your site.
· Top queries: Find the top queries that drive traffic to your site and where your site is included in the top search results. This will let you learn how users are finding your site.
· Indexing information: See how your site is indexed and which of your pages are included in the index. If we find violations in your site, we’ll give you the opportunity to fix the problems and request reinclusion of your site.
source :http://www.google.com/support/webmasters/bin/answer.py?answer=70897&topic=8843