Well, Robot.txt is a file that helps your blog in Crawling. What it means.. Search engines frequently visit your site and index your content or we can say Search engines bots that crawl your websites. These bots are automatic and crawl your website daily.