Dignity nobility high-quality state highness standing course caste status superiority precedence manner difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robot regarding which webpages usually are not to be crawled. For a internet search engine crawler might continue https://www.youtube.com/watch?v=aKirkq1uHGM