Dignity nobility high-quality condition highness standing course caste status superiority priority vogue distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robot regarding which webpages are usually not for being crawled. For a online search engine crawler could continue https://www.youtube.com/watch?v=B7lEnYsiELw