Most often indicates to assign something a status or situation to tell apart it from Other people in a bunch, as in txt file is then parsed and may instruct the robot as to which webpages will not be to get crawled. As being a online search engine crawler might https://www.youtube.com/watch?v=B7lEnYsiELw