Watching a recent football game, I imagined two very different teams: one called ??The Horders?? and the other, ??The Hunters??. In the game, it takes planning and skill to carry a football a few yards. There??s interference and distractions. Scantily clad dancing girls are screaming cheers nearby. How different, I wonder, is this from finding [...]

The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them “Bad Bots” or trespassers. We block [...]