Google's Dancing Robots

by Jeremy Wilson
 

Getting listed in Google can be extremely
frustrating.

It takes patience.

At times, it seems like your website will never
be listed. You wait and wait and, well, wait
some more.

You have taken all of the right steps.

You've submitted your site, careful not to
resubmit to often. You're listed in DMOZ, you
even have a few people linking to you from
other sites, but still nothing, no mention of
your site anywhere in Google's index.

At the very best, a few random pages show up
when you search using your domain name.

If this is you don't lose hope, this happens to
a lot of people.

Google knows you are there. In fact, Google has
probably found your site already. You just need
to follow the trail of clues.


---Attack of the Robots---

Google sends out "robots" to scour the Internet
for new sites and web pages.

They visit your site, poke around a bit, and if
they see something new or interesting they make
note of it and move on.

Pretty much the fundamental way most search
engines expand their database of websites.

This all occurs in the background unknown to you
and your visitors. But if you are willing to
invest a little time. You can track the progress
of Google's robots as they move through your
website.


---Google's Two Robots---

Google's main spider is called Googlebot. He
normally comes around at the end of the month as
part of Google's re-indexing or update to their
database of websites.

Once the monthly update has completed,
Googlebot's job is done until next month.

For a while, this was all that was needed.
Googlebot could handle all of the updates
himself.

But the web is constantly changing and new sites
are published daily. The sheer volume of new
pages was becoming too much for Googlebot to
handle on his own.

A little over a year ago, Google introduced
another robot named Freshbot to relieve some of
the burden, and to ensure their index is fresh
and relevant.

Unlike Googlebot, Freshbot is constantly on the
lookout. He can come around anytime during the
month to take a peak at your site.

It is not uncommon to have a page crawled by
Freshbot several times, see the page added to
Google's index, and then have the page completely
disappear from the index all in a matter of days.

If you find this happening to you, don't worry.
This is not uncommon.

Freshbot is constantly finding new sites and
and will often push older listings out of his
database to make room for the new.

This does not mean your site is forgotten.
Everything that Freshbot finds is added to the
list of updates to be made during the monthly
reindexing.

Your site might not get indexed the first month
or two but if Freshbot found it, rest assured
Googlebot will find and add it to the main
databse of sites.


---How Do You Know They Have Visited?---

Keeping tabs on the bots is really not that
difficult.

First, you need to have access to your website's
log files. They are usually found in your
website's file manager under a folder named
'logs'.


----------------Note---------------------------------------
If you can't locate your log files, ask your
webhost for assistance.
-------------------------------------------------------------

Log files are packed with cryptic text and
information that can be a little overwhelming the
first time you take a look. Luckily, there is no
need to read the entire file. You can simply
search the file for the word 'googlebot'.

After opening the file, press ctrl-f . A search
box will appear. Search for the word
'googlebot' and if you are lucky, you will see
something like this:

64.68.82.74 - - [26/Jan/2003:17:21:55 -0500]
"GET /yourpage.htm HTTP/1.0" 200 42551 "-"
Goglebot/2.1 (http://www.googlebot.com/bot.html)"

See how easy that was? Google knows you exist! :o)

Now, let's take a look at what this stuff means.


---What Does All That Log Stuff Mean---

Going back to our sample log file entry, you
will see the first thing listed is this IP
address:

64.68.82.74

An IP address beginning with a 64 normally
indicates a visit by Freshbot. If the address
began with 216 we would know that Googlebot had
visited.

Once you confirm a visit from Google, you need
to determine which pages and parts of your site
Google has visited.

Going back once again to our logfile entry, you
will notice the next part after the IP address is
a date and a file name.

[26/Jan/2003:17:21:55 -0500] "GET /yourpage.htm
HTTP/1.0"

From this, you can determine the date and time
Freshbot visited and the pages he indexed.

To find all of the pages Googlebot and Freshbot
have visited, search the entire log file for
references to GoogleBot.

Using this technique, you can get an idea of
what pages will eventually show up in Google's
index and begin the process of determining why
certain pages are not getting indexed.


---Final Thoughts---

Getting listed in Google is not an exact
science. There are as many theories as there are
webmasters.

But keeping tabs on what Google is doing at your
site is easy if you follow the above steps. It
takes less than 5 minutes to check you daily log
file.

Make this part of your routine.

You will find it surprisingly rewarding to know
Google has found your site! :o)