Search engine listing delays have come to be called the Google Sandbox effect are actually true in practice at each of four top tier search engines in one form or another. MSN, it seems has the shortest indexing delay at 30 days. This article is the second in a series following the spiders through a [...]

The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them “Bad Bots” or trespassers. We block [...]

There has been endless webmaster speculation and worry about the so-called “Google Sandbox” – the indexing time delay for new domain names – rumored to last for at least 45 days from the date of first “discovery” by Googlebot. This recognized listing delay came to be called the “Google Sandbox effect.” Ruminations on the algorithmic [...]

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren’t humans, you can’t tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time? Not quite sure what I am talking about? Here [...]