I had a robots.txt file asking Google not to spider .html files because I was trying to shake off a dup content penalty after I converted to ASP.
Now I've found the remove function in Google and I need some of my html link pages spidering, so I've removed the robots file, but Google hasn't been to see me for nearly a week, it used to come twice daily.
When you implement a robots.txt file does the command to ignore HTML only last for that visit or is it a long term thing?
Now I've found the remove function in Google and I need some of my html link pages spidering, so I've removed the robots file, but Google hasn't been to see me for nearly a week, it used to come twice daily.
When you implement a robots.txt file does the command to ignore HTML only last for that visit or is it a long term thing?
