STEP01. Effective Use of robots.txt

1. What is the effective use of robots.txt?

A robots.txt file is used to control the search engine traffic by letting search engines know the parts to access to the website and crawl. So if you don’t want certain pages to be exposed to search engine results, you can control them with robots.txt.

If your website is not exposed in search, check robots.txt settings first.

2. Recommendations for effective use of robots.txt

① Avoid excessive robots.txt settings

② Use safer methods for sensitive contents.

③ Use free tools for webmasters

3. Setting robots.txt

① Set robots.txt in [Manage Mode] – [Preferences] – [Site Management] – [Settings for SEO/SNS Sharing]