Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it’s a problem that is easily rectified. Your primary weapon [...]

The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them “Bad Bots” or trespassers. We block [...]

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats! [...]

Use text links, avoid image links. Anyhow, if you have used image links, then always make sure to put your keywords in the alt tags. Put your prime keywords in text links and always insure to put a short descriptions of your website/page in minimum of 20-30 words or more if allowed. In text links [...]

So you heard about someone stressing the importance of the robots.txt file, or noticed in your website’s logs that the robots.txt file is causing an error, or somehow it is on the very top of the top visited pages, or, you read some article about the death of the robots.txt file and about how you [...]

THE ROBOTS.TXT FILE You know that search engines have been created to help people find information quickly on the Internet, and the search engines acquire much of their information through robots (also known as spiders or crawlers), that look for web pages for them. The spiders or crawlers robots explore the web looking for and [...]

Once we have a website up and running, we need to make sure that all visiting search engines can access all the pages we want them to look at. Sometimes, we may want search engines to not index certain parts of the site, or even ban other Search Engines from the site all together. This [...]

1. Upload robots.txt file in to your root directory and include the folder name where you set your downloads. More information on how to set robots.txt: http://www.webmasters-central.com/wp/se/robotstxt.shtml 2. Set the permission of the download folder to 711 OR upload an index file to that folder. This makes that folder web inaccessible. For example create a [...]

Since the beginning of Internet there is a need to index the Web and many robots are built for this purpose. You already know that famous Google bot which is indexing the Web to keep track of urls and build a scheme out of it (link popularity algorithm…). There are not so many way to [...]

Since the beginning of Internet there is a need to index the Web and many robots are built for this purpose. You already know that famous Google bot which is indexing the Web to keep track of urls and build a scheme out of it (link popularity algorithm…). There are not so many way to [...]