txt file is then parsed and may instruct the robotic as to which pages aren't to generally be crawled. For a online search engine crawler may well keep a cached duplicate of the file, it could every now and then crawl internet pages a webmaster would not need to crawl. Pages normally prevented f