On-Page Optimization ’ Category

How to Remove Robots.txt in Google Search Engine Index

Direct answer to this question is.. YOU CAN’T..

Protocol / Exclusion Standard Disallow/Prevent Indexation
Web pages (.html, .php, .asp) Images (.png, gif, .jpg, .bmp) Audio / Video(.mp3, .wav, .avi) Documents (*.doc, *xls, .pdf)
Meta robots tag Yes No No No
Robots.txt Yes Yes Yes Yes
X -robots-tag Yes Yes Yes Yes

Table. Exclusions Standard Table 


But why do you want to de-index your robots text file?

  • Some advanced affiliate marketers are using redirects to show a more user-friendly URL, and disallow these pages, they sometimes use robots text file to keep their products private and safe for more competition.

URL Rewrites

I’ve been in contact with a client’s developer this past few days regarding URL rewrites of plain old *.html files to *.aspx. The SEO principle involve in this is pretty simple, “301 redirect all old *.html pages to each equivalent new *.aspx pages”. This is pretty tricky and requires more advance coding skills and familiarity with server side scripting, but once implemented properly, you don’t have to worry about URL rewrites affecting your search engine rankings.

Why use 301 redirect?
There are several types of redirects to choose from, but 301 redirect which stands for “permanent redirection” transfers link power from the old pages to new pages.