Social Media Affect: Common tendencies on social websites can spill above into search engines like google and yahoo, impacting the lookup quantity of related keyword phrases. txt file is then parsed and will instruct the robot concerning which web pages will not be for being crawled. Like a internet search https://www.youtube.com/watch?v=imiGL2YS2uo