I do not enter anything in the Robots.txt tab of the Settings window
Here is the URL for my Robots.txt http://photografix.pro/robots.txt
It produces a blank robots.txt file or one with just my sitemap address (I use Rage Sitemap Automator 2.5.2)
Of course if I don't specify a sitemap for SE, it's a completely blank file.
I've examined the file and I get errors when I use sites to diagnose the problem.
This site says http://tool.motorice...-checker.phtml:
You forgot to add a Disallow line in this block of code. You MUST insert at least one Disallow command. Please, remove all the reported errors and check again this robots.txt file. Line 2 Sitemap: http://www.photografix.pro/sitemap.xml
And google webmaster tools says:
Crawl postponed because robots.txt was inaccessible
Finally, I did a test of choosing a "Do not index these pages" on a specific page and got "pagename.html" and not the specific page name. Not sure if that makes a difference. I didn't save or publish this test.
Please help me.
Edited by Eric B, 15 June 2012 - 08:36 PM.
