ãŠã§ããµã€ãã®æ€çŽ¢ãšã³ãžã³æé©åã«ãããŠããSEO robots.txtããéèŠãªåœ¹å²ãæãããŸããããã®å ·äœçãªçç±ãå¹æçãªå©çšæ¹æ³ãææ¡ããŠããããžãã¹ããŒãœã³ã¯å€ããããŸãããrobots.txtãã¡ã€ã«ã¯ãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒããµã€ãã®ã©ã®éšåãã€ã³ããã¯ã¹ããããæ瀺ããããã«äœ¿çšãããæ€çŽ¢ãšã³ãžã³æé©åïŒSEOïŒã®å¹æãé«ããããã«æé©ã«èšå®ããããšãäžå¯æ¬ ã§ããæ¬çš¿ã§ã¯ãrobots.txtã®åºæ¬ããé«åºŠãªäœ¿ãæ¹ãŸã§ããµã€ãã®å¯èŠæ§ãšæ€çŽ¢é äœãåäžãããããã®å ·äœçãã€å¹æçãªæåãæäŸããããšãç®æããŠããŸãã
robots.txtãšã¯
robots.txtãã¡ã€ã«ãšã¯ããŠã§ããµã€ãã®ã«ãŒããã£ã¬ã¯ããªå ã«é 眮ãããããã¹ããã¡ã€ã«ã§ãããæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã«å¯ŸããŠã©ã®ããŒãžãååŸããã¹ããããŸãã¯ç¡èŠãã¹ãããæ瀺ããŸãããã®ãã¡ã€ã«ã¯ã”User-agent”ã”Disallow”ã”Allow” ãªã©ã®æ瀺ãçšããŠããŠã§ããµã€ãã®ã¯ããŒã«æ¹æ³ãæ§æããããã®æšæºã§ãã
æ€çŽ¢ãšã³ãžã³ãæ°ãããŠã§ããµã€ããçºèŠããå ŽåããŸãrobots.txtãã¡ã€ã«ãååšãããããã§ãã¯ããŸããååšããå Žåããã®æ瀺ã«åŸã£ãŠãŠã§ãããŒãžã®ã€ã³ããã¯ã¹äœæãé²ããŸãããããã£ãŠãrobots.txtã¯ãŠã§ããµã€ãã®èšªåã«ãŒã«ãå®çŸ©ããã¯ããŒã©ãŒã«ããã¢ã¯ã»ã¹ã®ç®¡çãå©ãã圹å²ãæãããŸãã
é©åã«èšå®ãããªããšãéèŠãªã³ã³ãã³ããæ€çŽ¢çµæããé€å€ãããå¯èœæ§ãããããããŠã§ããã¹ã¿ãŒã¯robots.txtãã¡ã€ã«ã®èšå®ã«æ³šæãæãå¿ èŠããããŸãããŸãã誀ã£ãæ瀺ããããšãã©ã€ãã·ãŒã«é¢ããããŒãžãå ¬éããããªã¹ã¯ããããããæ éãªç®¡çãæ±ããããŸãã
ãªãSEOã«robots.txtãéèŠãªã®ã
SEOã®èŠ³ç¹ããrobots.txtã®éèŠæ§ã¯ãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒããµã€ããå¹ççã«ã€ã³ããã¯ã¹ã§ããããã«ããããã«äžå¯æ¬ ã§ããrobots.txtãã¡ã€ã«ã䜿çšããŠãã¯ããŒã©ãŒã«éèŠãªããŒãžã«éäžããããæ瀺ããç¡é¢ä¿ãŸãã¯éè€ããã³ã³ãã³ããã€ã³ããã¯ã¹ãããªãããã«ããããšãã§ããŸãã
ããšãã°ãã»ãã·ã§ã³IDãå«ãURLãªã©ãã€ã³ããã¯ã¹äœæãé¿ãããURLãããå Žåãããããrobots.txtã«èšèŒããŠã¯ããŒã©ãŒãã¢ã¯ã»ã¹ããªãããã«ããŸããããã«ãããæ€çŽ¢ãšã³ãžã³ã®ãªãœãŒã¹ãæé©ã«å©çšãããŠã§ããµã€ãã®éèŠãªã³ã³ãã³ããé©åã«è©äŸ¡ããããããªããŸãã
ãŸããrobots.txtã¯ç¹å®ã®ãŠãŒã¶ãŒãšãŒãžã§ã³ãã®ã¯ããŒã©ãŒãžã®ã¢ã¯ã»ã¹å¶éããã€ã³ããã¯ã¹åãããªãããšãæããªãœãŒã¹ã®æå®ã«ã䜿ãããŸããããã«ãããSEOããã©ãŒãã³ã¹ãæ倧åããäžã§éèŠãªã³ã³ãã³ãã®å¯èŠæ§ã確ä¿ãã競åä»ç€Ÿãšã®å·®å¥åãå³ãããšãå¯èœã«ãªããŸãã
robots.txtã®åºæ¬çãªæ§æ
robots.txtãã¡ã€ã«ã®æ§æã¯ã·ã³ãã«ã§ãäž»ã«User-agentãDisallowãAllowã®3ã€ã®ãã£ã¬ã¯ãã£ãã§æ§æãããŸãããUser-agentãã¯ç¹å®ã®ã¯ããŒã©ãŒãæå®ãããã£ãŒã«ãã§ãå šãŠã®ã¯ããŒã©ãŒã«å¯Ÿããæ瀺㯠‘*’ ã§è¡šãããŸãããDisallowãã¯ã¯ããŒã©ãŒãã¢ã¯ã»ã¹ãçŠæ¢ãããŠãããªãœãŒã¹ãæå®ãããAllowãã¯ã¢ã¯ã»ã¹ãèš±å¯ãããªãœãŒã¹ã瀺ããŸãã
äŸãã°ããã¹ãŠã®ã¯ããŒã©ãŒã«å¯ŸããŠç¹å®ã®ãã£ã¬ã¯ããªãžã®ã¢ã¯ã»ã¹ãçŠæ¢ããå Žåã以äžã®ããã«èšè¿°ããŸãã
User-agent: * Disallow: /example-directory/
éã«ãç¹å®ã®ã¯ããŒã©ãŒã«ã®ã¿èš±å¯ãŸãã¯çŠæ¢ãæå®ãããå Žåã¯ããã®ã¯ããŒã©ãŒã®ååã§ãUser-agentããèšå®ããŸããä»ã«ããCrawl-delayããšãããã£ã¬ã¯ãã£ãã䜿çšããŠãã¯ããŒã©ãŒã®èšªåé »åºŠãå¶åŸ¡ããããšãå¯èœã§ãã
robots.txtãã¡ã€ã«ã®äœææ¹æ³
å¹æçãªrobots.txtãã¡ã€ã«ãäœæããããã«ã¯ããŸããŠã§ããµã€ãã®ã¯ããŒã«ãæãŸãªãã³ã³ãã³ããæ確ã«ããããšã倧åã§ããç¡é¢ä¿ãªãã©ã¡ãŒã¿ãŒãå«ãURLãããã©ã€ããŒããªæ å ±ãå«ãŸããããŒãžãªã©ãæ€çŽ¢ãšã³ãžã³ã«è¡šç€ºãããããªããªãœãŒã¹ãç¹å®ããŸãã
次ã«ãããã¹ããšãã£ã¿ã䜿çšããŠrobots.txtãã¡ã€ã«ãäœæããŸãããã¡ã€ã«ã®åœåã¯å°æåã®ãrobots.txtããšãããµã€ãã®ã«ãŒããã£ã¬ã¯ããªã«ã¢ããããŒãããå¿ èŠããããŸãããã¡ã€ã«ã®æ§æã¯ç°¡æœãã€æ£ç¢ºã«è¡ããééãããªãããšã確èªããäžã§ãµã€ãã«é©çšããŸãã
æçµçã«ã¯ãäœæããrobots.txtãã¡ã€ã«ãæ£ãããã©ãããæ€èšŒããããã«Googleã®ç¡æããŒã«ãGoogle Search Consoleããªã©ã掻çšã§ããŸããããã«ãã£ãŠãæœåšçãªãšã©ãŒãåé¡ç¹ãäºåã«ç¹å®ããä¿®æ£ããããšãå¯èœã«ãªããŸãã
robots.txtã®æé©ãªå®è£
robots.txtã®æé©ãªå®è£ ã¯ãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒããµã€ããé©åã«èªèããŠãå¿ èŠãªããŒãžã®ã¿ã€ã³ããã¯ã¹ããã®ã«éèŠã§ãããã®å°ããªãã¡ã€ã«ããµã€ãå ã®ç¹å®ã®ã»ã¯ã·ã§ã³ãžã®ã¢ã¯ã»ã¹ãå¶åŸ¡ããSEOã®ææãå·Šå³ããå¯èœæ§ããããŸãã
robots.txtãã¡ã€ã«ã¯ããŠã§ããµãŒããŒã®ã«ãŒããã£ã¬ã¯ããªã«çœ®ãããã¯ããŒã©ãŒããµã€ãã蚪ããéã«ãã©ã®éšåãžã¢ã¯ã»ã¹ã§ããããæ瀺ããŸããé©åãªåœ¢ã§èšå®ãããŠããªããšãã¯ããŒã©ãŒãç¡é¢ä¿ãªããŒãžãã€ã³ããã¯ã¹ããŠããŸã£ãããææªã®å ŽåãéèŠãªããŒãžãã€ã³ããã¯ã¹ããé€å€ãããããšããããŸãã
ãã®ãããrobots.txtã®èšå®ã¯ããµã€ãã®SEOæŠç¥ã«ãããŠæ¬ ããããšã®ã§ããªãéèŠãªå·¥çšãšãããŸãããµã€ããªãŒããŒã¯ã¯ããŒã©ãŒã®åããç解ããæå³ããéãã«ãŠã§ããµã€ããæ¢çŽ¢ãããããã«ãæ£ç¢ºãªèšå®ãå¿ èŠãšãªããŸãã
Webãµã€ãã®æ§é ãšãç解ãã
Webãµã€ãã®æ§é ãç解ããŠããªããšãrobots.txtã®å¹æçãªæžãæ¹ã¯é£ããã§ããçæ³çã«ã¯ãrobots.txtã¯ãµã€ãã®éå±€çãªæ§é ãåæ ããã©ã®ã¬ãã«ã®ããŒãžãã€ã³ããã¯ã¹ãããã¹ãããæ確ã«ããã¹ãã§ãã
ããšãã°ããŠãŒã¶ãŒãšã¯ã¹ããªãšã³ã¹ãæãªãå¯èœæ§ã®ããéè€ã³ã³ãã³ããããã©ã€ããŒããªæ å ±ãå«ãããŒãžã¯ã¯ããŒã©ãŒã«èŠããããªãã§ããããããããé©åã«æé€ããã«ã¯ããµã€ãã®å šäœçãªæ§é ãææ¡ããéèŠãªããŒãžãšããã§ãªãããŒãžãåºå¥ããå¿ èŠããããŸãã
ããã«ãWebãµã€ãããªãã¥ãŒã¢ã«ããããªã©ãéææ§é ãå€ããå Žåã¯ãrobots.txtãããã«åãããŠæŽæ°ãç¶ããããšãäžå¯æ¬ ã§ãããµã€ãèšèšæã«robots.txtãèšèšãã人ãšååããããšã§ãæé©ãªãŠã§ããµã€ãæ§é ãšã®åæãä¿ã€ããšãã§ããŸãã
Webãµã€ãã®æ§é ã«ã€ããŠã¯ãäžèšã®èšäºã§è©³ãã解説ããŠããŸãã
ã¯ããŒã©ãŒã«ãšã£ãŠçæ³çãªrobots.txt
ã¯ããŒã©ãŒã«ãšã£ãŠçæ³çãªrobots.txtã¯ããŸãã¯ã·ã³ãã«ã§åãããããããšãæããããŸããäœãèš±å¯ããäœããããã¯ããããæ確ã§ããã°ãã¯ããŒã©ãŒã¯å¹ççã«ãµã€ããã€ã³ããã¯ã¹åã§ããŸãã
User-agentãšDisallowã®ãã£ã¬ã¯ãã£ããé©åã«äœ¿çšããããšã§ãç¹å®ã®ã¯ããŒã©ãŒãã¢ã¯ã»ã¹ãçŠæ¢ãããã³ã³ãã³ããæå®ããããšãå¯èœã§ãããã ããããŸãè€éã«ãéãããšã誀ã£ãŠéèŠãªã³ã³ãã³ããã¯ããŒã©ãŒããé ããŠããŸããªã¹ã¯ãé«ãŸããŸãã
ãŸããAllowã®ãã£ã¬ã¯ãã£ãã䜿çšããŠããããã¯ããããã£ã¬ã¯ããªå ã®ç¹å®ã®ãã¡ã€ã«ãã¯ããŒã©ãŒãã¢ã¯ã»ã¹å¯èœã«ãããšãã现ãã調æŽãå¯èœã§ãããããã®çŽ°ããèšå®ã¯ãã¯ããŒã©ãŒã«å¯ŸããŠãµã€ãã®æãéèŠãªèŠçŽ ã«çŠç¹ãåœãŠãããæå©ããããŸãã
ã¯ããŒã©ãŒã«ã€ããŠã¯ãäžèšã®èšäºã§è©³ãã解説ããŠããŸãã
XMLãµã€ãããããšrobots.txtã®é£æº
XMLãµã€ããããã¯ã¯ããŒã©ãŒã«å¯ŸããŠããŠã§ããµã€ãäžã®å šããŒãžã®æ§é ã瀺ã圹å²ãæ ããŸããrobots.txtãã¡ã€ã«ã«ãµã€ããããã®URLãè¿œèšããããšã«ãããã¯ããŒã©ãŒã¯ãµã€ãæ§é ãããå¹ççã«ç解ããã€ã³ããã¯ã¹ãè¡ãéã®åèãšããããšãã§ããŸãã
Sitemapãã£ã¬ã¯ãã£ããrobots.txtã«å«ããããšã§ããµã€ãã®ãã¹ãŠã®ããŒãžãã¯ããŒã©ãŒã«ãã£ãŠçºèŠãããå¯èœæ§ãé«ããŸããç¹ã«å€§èŠæš¡ãªãŠã§ããµã€ãã§ã¯ããã®é£æºãã€ã³ããã¯ã¹äœæã®æ£ç¢ºæ§ã倧ãã«åäžãããŸãã
ãµã€ããããèªäœãå®æçã«æŽæ°ããããšã§ãæ°ããã³ã³ãã³ããå€æŽãããããŒãžãã¯ããŒã©ãŒãèŠèœãšããªãããã«ããŸããrobots.txtãšãµã€ããããã®äž¡æ¹ãé©åã«ç®¡çããããšããããSEOå¹æãæ倧åããéµãšãªããŸãã
XMLãµã€ããããã«ã€ããŠã¯ãäžèšã®èšäºã§è©³ãã解説ããŠããŸãã
robots.txtã§é¿ããã¹ãäžè¬çãªãšã©ãŒ
robots.txtã«ã¯ãé¿ããã¹ãããã€ãã®äžè¬çãªãšã©ãŒããããŸããããšãã°ããµã€ãã®å šãŠã誀ã£ãŠãããã¯ããŠããŸããããªèšè¿°ãããŠããŸãããšã¯çµ¶å¯Ÿã«é¿ããã¹ãã§ãã
User-agentã®æå®ã誀ããšãå šãŠã®ã¯ããŒã©ãŒããµã€ãã«ã¢ã¯ã»ã¹ã§ããªãããšããããŸãã’User-agent: *’ ãšèšè¿°ããã°ãå šãŠã®ã¯ããŒã©ãŒã«æ瀺ãé©çšãããã®ã§ãç¹å®ã®ã¯ããŒã©ãŒã®ã¿ã察象ã«ãããå Žåã¯æ³šæãå¿ èŠã§ãã
ãŸãããµã€ãã®å®éã®URLæ§é ãšç°ãªããã¹ãæå®ããŠããŸããšãã¯ããŒã©ãŒãæ¬æ¥ã¢ã¯ã»ã¹ãã¹ãã§ãªãé åã«å ¥ã蟌ãã§ããŸã£ãããéèŠãªããŒãžãèŠéããŠããŸãå¯èœæ§ããããŸããrobots.txtã®èšå®ãè¡ãéã«ã¯ããµã€ãã®URLæ§é ãæ£ç¢ºã«åæ ãããããšãéèŠã§ãã
robots.txtã®ãã¹ããšç£èŠ
robots.txtãã¡ã€ã«ã¯Webãµã€ãã®SEOã«éèŠãªåœ¹å²ãæãããŸãããã®ãã¡ã€ã«ã¯ãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã«ãµã€ãã®ç¹å®ã®ã»ã¯ã·ã§ã³ãã¯ããŒã«ãããã©ãããäŒããããã«äœ¿çšãããŸããé©åã«éçšãããå®æçã«ç£èŠãããããšã§ããµã€ããæ£ããã€ã³ããã¯ã¹ãããããšã確èªã§ããŸãã
robots.txtã®å€æŽããã¹ãããæ¹æ³
robots.txtãã¡ã€ã«ãå€æŽããåã«ããã®åœ±é¿ãæ£ç¢ºã«ç解ããããšãéèŠã§ããå€æŽãè¡ã£ãåŸã¯ãGoogleã®ç¡æããŒã«ã§ãããSearch Consoleãã®robots.txtãã¹ã¿ãŒã䜿çšããŠããã¡ã€ã«ãæå³ããéãã«æ©èœããŠãããããã¹ãããŠãã ãããããã«ãããæ€çŽ¢ãšã³ãžã³ãããããŠã§ããµã€ãã®ã©ã®ã»ã¯ã·ã§ã³ã«ã¢ã¯ã»ã¹ã§ããããã§ããªãããæ£ç¢ºã«åæã§ããŸãã
ããã«ããŠã§ãããŒãžã«å¯Ÿããã¯ããŒã©ãŒã®ã¢ã¯ã»ã¹ãã·ãã¥ã¬ãŒããããµãŒãããŒãã£è£œã®ããŒã«ãååšããŠããŸãããããã®ããŒã«ãå©çšããŠäºæãã¬æåããšã©ãŒãäºåã«çºèŠããå®éã«éçšç°å¢ã§ã®åé¡ãæªç¶ã«é²ãããšãå¯èœã§ãã
ãŸããããŒã«ã«ç°å¢ã§ã®ãã¹ããéèŠã§ããéçºãå®äºããããã¹ããŒãžã³ã°ç°å¢ãªã©ã§å€æŽãæœããäºæããªãã¯ããŒã«ã®åé¡ããªãããšã確èªããŸãããããã¹ãã¯æ éã«ããã€å®æçã«è¡ãå¿ èŠããããŸãã
æ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã«ç¶æ³ãç£èŠãã
robots.txtã®å€æŽã¯æ€çŽ¢ãšã³ãžã³ã«ãããµã€ãã®ã¯ããŒã«ã«åœ±é¿ãäžããŸããããããã«ãå€æŽåŸã®ã¯ããŒã«ç¶æ³ãæ éã«ç£èŠããããšãå¿ èŠã§ããSearch Consoleã®ãã¯ããŒã«çµ±èšã¬ããŒãããå©çšãããšããµã€ãã®ã¯ããŒã«é »åºŠãã¯ããŒã«ãããããŒãžæ°ãªã©ã確èªã§ããŸãã
ç£èŠã«ãã£ãŠãã¯ããŒã©ãŒãäºæãã¬ãããã³ã«ã«åŸã£ãŠãµã€ããã¯ããŒã«ããŠããªããããŸãã¯éèŠãªã³ã³ãã³ããã¯ããŒã«ããé€å€ãããŠããªããããã§ãã¯ã§ããŸããã¯ããŒã«ãšã©ãŒã¬ããŒããèŠãããšã§ãã¯ããŒã©ãŒãé©åã«ã¢ã¯ã»ã¹ã§ããŠãããã©ãããç解ããåé¡ãããã°ä¿®æ£ã§ããŸãã
ãã®ããã»ã¹ã«ã¯ãrobot.txtãã¡ã€ã«ã®ç£èŠã ãã§ãªãããµã€ããããã®æŽæ°ãšéä¿¡ããªãã€ã¬ã¯ãã®ãã§ãã¯ã404ãšã©ãŒã®ä¿®æ£ãªã©ããµã€ãæ§é ã®ãã®ä»ã®èŠå ã«åžžã«ç®ãé ãããšãå«ãŸããŸãã
robots.txtãã¡ã€ã«ã®å€æŽãæ€çŽ¢çµæã«äžãã圱é¿
é©åãªrobots.txtãã¡ã€ã«ã¯ããŠã§ããµã€ãã®æ€çŽ¢ã©ã³ãã³ã°ã«çŽæ¥çãªåœ±é¿ãåãŒããŸããæ€çŽ¢ãšã³ãžã³ãç¹å®ã®ããŒãžããªãœãŒã¹ãã¯ããŒã«ããŠããããããªãå Žåã¯ããã®æ瀺ãrobots.txtãã¡ã€ã«ã«èšè¿°ããããšã§ãã¯ããŒã«ãããã³ã³ãã³ããã³ã³ãããŒã«ããããšãã§ããŸãã
äžæ¹ã§ãrobots.txtãã¡ã€ã«ã®èšå®ãã¹ã«ãã£ãŠã¯ãéèŠãªããŒãžãã¯ããŒã«ããããæ€çŽ¢çµæã«è¡šç€ºãããªãå¯èœæ§ããããŸãããã®ãããrobots.txtãã¡ã€ã«ã®å€æŽã¯SEOã®èŠ³ç¹ããéèŠãªæå³ãæã¡ãæ éãªæäœãæ±ããããŸãã
å€æŽã¯æã«æ€çŽ¢çµæã«æéããããããšãããã®ã§ãçµæã確å®ã«èŠãããã«ã¯æ°é±éãšããæéãèŠç©ããå¿ èŠããããŸããå€æŽãé©åã§ããã°ããŠã§ããµã€ãã®ããã©ãŒãã³ã¹ãåäžããŸãã
å®è·µçãªrobots.txtã®ãã©ãã«ã·ã¥ãŒãã£ã³ã°
robots.txtã®åé¡çºçæã«ã¯ãå®è·µçãªãã©ãã«ã·ã¥ãŒãã£ã³ã°ãå¿ èŠã§ããæåã®ã¹ããããšããŠãæå®ããçŠæ¢ããããã¹ãæ£ãããã©ããããŸããã¢ã¯ã»ã¹ãèš±å¯ããå¿ èŠã®ããã¯ããŒã©ãŒã«åœ±é¿ãäžããŠããªãããå確èªããŠãã ããã
ãŸããæ£èŠè¡šçŸãã¯ã€ã«ãã«ãŒãã䜿çšããŠããå Žåã¯ãããããäºæãã¬è¡åãåŒãèµ·ãããŠããªãã泚ææ·±ã確èªããå¿ èŠããããŸããèšè¿°ã®ãã¹ããã£ãå Žåã¯ãçŽ æ©ãä¿®æ£ãè¡ããå床ãã¹ããå®æœããŠãã ããã
SEOã¯åžžã«é²åããŠããŸãããããã£ãŠãrobots.txtãã¡ã€ã«ã®å¹æãè©äŸ¡ãç¶ãããŠã§ããµã€ãã®èŠæ±ã«åãããŠæŽæ°ããããšãéèŠã§ãããã©ãã«ã·ã¥ãŒãã£ã³ã°ã¯ãSEOã«ãããæåãžã®ç¶ç¶çãªåãçµã¿ã®äžéšã§ãã
robots.txtã®é«åºŠãªæŽ»çš
ãŠã§ã管çè ã«ãšã£ãŠãrobots.txtãã¡ã€ã«ã¯æ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã®ååãã¬ã€ãããããã®åŒ·åãªããŒã«ã§ãããã®é©åãªèšå®ã«ããããµã€ãã®SEOãå¹æçã«åŒ·åããããšãã§ããŸããããããããšãåºæ¬ãç解ããŠãããšããŠããrobots.txtã®é«åºŠãªæŽ»çšæ¹æ³ãç¥ãããšã§ãŠã§ããµã€ãã®ããã©ãŒãã³ã¹ãããã«åäžãããããšãå¯èœã§ãã
ãã®èšäºã§ã¯ãããŸããŸãªå¿çšã·ããªãªãéããŠãrobots.txtãã¡ã€ã«ãçšããSEOæé©åã®æè¡ãæãäžããŠãããŸããå ·äœçã«ã¯è€æ°ã®ãµããã¡ã€ã³ã®ç®¡çãã¯ããŒã«é »åºŠã®å¶åŸ¡ãã»ãã¥ãªãã£ãžã®é æ ®ãåçãªãµã€ãå 容ãžã®å¯Ÿå¿ãªã©ããã®äœ¿ãæ¹ã®ãã€ã³ãã詳ãã解説ããŸãã
è€æ°ã®ãµããã¡ã€ã³ã§ã®robots.txtã®ç®¡ç
倧èŠæš¡ãµã€ãããµãŒãã¹ã§ã¯è€æ°ã®ãµããã¡ã€ã³ãæã€ããšãäžè¬çã§ããããããã®ãµããã¡ã€ã³ã«é©åãªrobots.txtãã¡ã€ã«ãèšå®ããããšã¯ããµã€ãå šäœã®SEOæŠç¥ã«äžå¯æ¬ ã§ããrobots.txtãã¡ã€ã«ã¯ãµããã¡ã€ã³ããšã«èšå®ããå¿ èŠããããããããã«é©åãªãã£ã¬ã¯ãã£ããæå®ããããšãéèŠã«ãªããŸãã
äŸãã°ãã¡ã€ã³ãµã€ãã§ã¯å šãŠã®ã¯ããŒã©ãŒã®ã¢ã¯ã»ã¹ãèš±å¯ããªããããç¹å®ã®ãµããã¡ã€ã³ã§ã¯ã¯ããŒã©ãŒãå¶éãããã±ãŒã¹ããããŸãããã®ãšãããµããã¡ã€ã³ããšã«ã«ã¹ã¿ãã€ãºãããrobots.txtãçšæããããšã§ãæå³ããã¯ããŒã«ããªã·ãŒã«æ²¿ã£ãæ€çŽ¢ãšã³ãžã³ã®åäœãå®çŸã§ããŸãã
ãŸãããµããã¡ã€ã³éã§ãªãœãŒã¹ã®éè€ãé¿ããããã«ã¯ã”Disallow”ãã£ã¬ã¯ãã£ããé©åã«æŽ»çšããããšãéèŠã§ããããã«ãããéèŠã§ãªãããŒãžãç¡é§ã«ã€ã³ããã¯ã¹ãããã®ãé²ããéèŠãªã³ã³ãã³ãã®å¯èŠæ§ãé«ããããšãã§ããŸãã
ã¯ããŒã«é »åºŠã®å¶åŸ¡ãšrobots.txt
robots.txtãã¡ã€ã«ãå©çšããŠã¯ããŒã©ãŒã®èšªåé »åºŠãéæ¥çã«ç®¡çããããšãã§ããŸãããããã”Crawl-delay”ãã£ã¬ã¯ãã£ãã«ã¯æ³šæãå¿ èŠã§ããããã¹ãŠã®æ€çŽ¢ãšã³ãžã³ããã®æ瀺ã解éããããã§ã¯ãªãã®ã§ãé©åãªèšå®ãæ±ããããŸãã
ãµãŒããŒãžã®è² è·ãæžããããšãç®çãšããŠãããã€ãã®æ€çŽ¢ãšã³ãžã³ã¯”Crawl-delay”ãã£ã¬ã¯ãã£ããèªèããŠã¯ããŒã«ã®ééã調ç¯ããŸãããããé©åã«æŽ»çšããããšã§ããµã€ãã®ã¬ã¹ãã³ã¹ã¿ã€ã ãç¶æãã€ã€æ€çŽ¢ãšã³ãžã³ã«æ å ±ãæ£ç¢ºã«æäŸããããšãå¯èœã«ãªããŸãã
ããããã¯ããŒã«é »åºŠãå¶åŸ¡ãããããšæ°ããã³ã³ãã³ãã®ã€ã³ããã¯ã¹é床ã«åœ±é¿ãåãŒãå¯èœæ§ãããããããµã€ãã®æŽæ°é »åºŠãå¿ èŠãªãµãŒããŒãªãœãŒã¹ãšç §ããåãããŠæ éã«èšå®ãè¡ãå¿ èŠããããŸãã
ã»ãã¥ãªãã£ãæèããrobots.txtã®å©çš
ç§éã¯robots.txtãçšããŠæ€çŽ¢ãšã³ãžã³ã«å¯Ÿãããµã€ãå ã®ç¹å®ã®éšåãã¯ããŒã«ãããªãããæ瀺ã§ããŸãããããã¯ã»ãã¥ãªãã£äžã®ãªã¯ãšã¹ããæºãããã®ã§ã¯ãããŸãããrobots.txtãã¡ã€ã«ã¯å ¬éæ å ±ã§ãããæªæã®ããè ãã¢ã¯ã»ã¹ãã€ã³ããæ¢ãããã®ææãããšããªãåŸãŸãã
ãã®ããã管çè ããã«ãæ©å¯æ å ±ãå«ãéšåã¯ã”Disallow”ãã£ã¬ã¯ãã£ãã§ã¢ã¯ã»ã¹ãå¶éããã ãã§ãªããIPå¶éãèªèšŒæ©èœãéããŠãã£ãããšä¿è·ããå¿ èŠããããŸããrobots.txtã¯ã¯ããŒã©ãŒã«å¯Ÿããã¬ã€ãã§ãã£ãŠãã»ãã¥ãªãã£å¯Ÿçã§ã¯ãªãããšãç解ããŸãããã
ãŸããã©ã®URLãçŠæ¢ãããŠããããæ瀺çã«äŒããããšã¯ãã»ã³ã·ãã£ããªæ å ±ãå«ãŸããŠããå¯èœæ§ãããããã泚æãå¿ èŠã§ãã代ããã«ããææ§ãªãã£ã¬ã¯ãã£ãã䜿çšããŠã該åœããå 容ãžã®ã¢ã¯ã»ã¹ãã³ã³ãããŒã«ããæ¹ãå®å šãªå ŽåããããŸãã
åçãµã€ãå 容ãšrobots.txtã®æé©å
ä»æ¥ã®ãŠã§ããµã€ãã§ã¯ãã³ã³ãã³ãã¯éçã§ã¯ãªãåçã«çæãããããšãå€ããªã£ãŠããŸããåçãªãµã€ãã§ã¯ãURLãã©ã¡ãŒã¿ãéèŠãªåœ¹å²ãæãããŸããããããã¯ããŒã©ãŒã«ãšã£ãŠæ··ä¹±ã®ããšã«ãªãããšããããŸããé©åãªrobots.txtãã¡ã€ã«ã®èšå®ã«ãããäžèŠãªãã©ã¡ãŒã¿ãæã€URLã®ã¯ããŒãªã³ã°ãé²ããã³ã³ãã³ãã®éè€ãé¿ããããšãã§ããŸãã
äŸãã°ãã»ãã·ã§ã³IDããã©ããã³ã°ãã©ã¡ãŒã¿ãURLã«è¿œå ãããããšããããŸãããããããã¯ããŒã©ãŒãç¡èŠããããã«æ瀺ããããšã§ãã¯ããŒã«å¹çãåäžãããããšãã§ããŸãã”Disallow”ãã£ã¬ã¯ãã£ãã掻çšããŠãç¹å®ã®ãã©ã¡ãŒã¿ãå«ãURLãžã®ã¢ã¯ã»ã¹ãå¶éããããšã§ãã¯ããŒã©ãŒã«ãšã£ãŠæçãªæ å ±ã«çŠç¹ãåãããããšãã§ããŸãã
ããã«ãGoogleã®Webãã¹ã¿ãŒããŒã«ãªã©ã§ã¯ãURLãã©ã¡ãŒã¿ã®æ±ããæå®ãããªãã·ã§ã³ãæäŸãããŠããŸãããããã®ããŒã«ãšrobots.txtã®äž¡æ¹ãçµã¿åãããŠå©çšããããšã§ããµã€ãã®SEOã«æé©ãªç°å¢ãäœãåºãããšãå¯èœã§ãã
æåäºäŸã«åŠã¶robots.txtã®éçš
倧æãŠã§ããµã€ãã®robots.txtæŠç¥
倧æãŠã§ããµã€ãã®éå¶è ã«ãšã£ãŠãrobots.txtãã¡ã€ã«ã®æé©åã¯SEOæŠç¥ã®éèŠãªäžç°ã§ãããããã®ãŠã§ããµã€ãã¯æ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒãå¿ èŠãªæ å ±ãå¹ççã«ååŸããäžèŠãªããŒãžã¯é€å€ããããšã§ã€ã³ããã¯ã¹ã®ç²ŸåºŠãåäžãããŠããŸãã
æåããäŸã§ã¯ããµã€ãããããžã®ãªã³ã¯ãæ瀺ããããšã§ãã¯ããŒã©ãŒã«ãµã€ãã®éèŠãªããŒãžãžãšèªå°ããŠããŸãããŸããæ§é ãè€éãªãµã€ãã§ã¯ãäžèŠãªãã©ã¡ãŒã¿ãå«ãŸããURLããéè€ããã³ã³ãã³ããçºçãåŸãã»ã¯ã·ã§ã³ãæé€ããŠéè€ã³ã³ãã³ãã®åé¡ãé¿ããŠããŸãã
ããã«ãã»ãã¥ãªãã£ããã©ã€ãã·ãŒã確ä¿ããããã«ãå éšåãã®æ å ±ãéçºäžã®ããŒãžãã¯ããŒã©ãŒããé ãããã«robots.txtãèšå®ããŠããäŸããããŸãã
SEOåäžã«è²¢ç®ããrobots.txtã®äºäŸ
SEOãåäžãããããã«ã¯ãæ€çŽ¢ãšã³ãžã³ã«ãµã€ããæ£ããç解ãããé©åã«ã€ã³ããã¯ã¹ãããããšãäžå¯æ¬ ã§ããrobots.txtãã¡ã€ã«ã®å¹æçãªæŽ»çšäºäŸãšããŠãããŒãžã®ã¯ããŒã«äºç®ã管çããã¯ããŒã©ãŒã®è² è·ãæé©åããã±ãŒã¹ãæããããŸãã
ãã®ãããªãŠã§ããµã€ãã§ã¯ãéèŠã§ãªãããŒãžãäœå質ãªã³ã³ãã³ããDisallowãã£ã¬ã¯ãã£ããçšããŠç©æ¥µçã«æé€ããæ€çŽ¢ãšã³ãžã³ãåªå çã«äŸ¡å€ã®ããã³ã³ãã³ããã¯ããŒã«ããããããŠããŸãã
ãŸããURLã®ãã©ã¡ãŒã¿ãæé€ããŠã¯ãªãŒã³ãªURLæ§é ãããŒãããããŒãžéã®éè€ãé¿ããããšã§ãã¯ããŒã©ãŒã®å¹ççãªåäœãæ¯æŽããŠããŸãã
åœéçãªãŠã§ããµã€ããšå€èšèªrobots.txt
å€èšèªãµã€ããéå¶ããŠããå Žåãèšèªãå°åã«å¿ããŠæ€çŽ¢ãšã³ãžã³ã«ç°ãªãrobots.txtã®æ瀺ãåºãããšãå¹æçã§ããrobots.txtãã¡ã€ã«ã¯URLåäœã§èšå®ãå¯èœãªãããç¹å®ã®åœãèšèªåãã®ã»ã¯ã·ã§ã³ã«å¯ŸããŠç°ãªãæ瀺ãåºããŸãã
æåäºäŸãšããŠããµããã£ã¬ã¯ããªããµããã¡ã€ã³ã䜿ã£ãŠèšèªãå°åãåºåããããããã«é©ããrobots.txtãèšçœ®ããæŠç¥ããããŸããããã«ãããããŒã«ã©ã€ãºãããã³ã³ãã³ãã®å¯èŠæ§ãåäžãããåèšèªã®æ€çŽ¢ãšã³ãžã³ã«æé©ãªæ å ±ãæäŸããŠããŸãã
ããã«ãhreflangã¿ã°ãšã®é£æºãåãããšã§ãé©åãªèšèªçãžã®ãŠãŒã¶ãŒã®èªå°ã匷åããåœéçãªæ€çŽ¢çµæã«ããããµã€ãã®ã©ã³ãã³ã°åäžã«è²¢ç®ããŠããŸãã
å°èŠæš¡ããžãã¹ã§ã®robots.txtå®äŸ
å°èŠæš¡ããžãã¹ã«ãããŠãrobots.txtã¯éèŠãªåœ¹å²ãæãããŸãããªãœãŒã¹ãéãããŠãããããæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã«ããã¯ããŒã«äºç®ãæå¹ã«æŽ»çšããå¿ èŠããããŸãã
äºäŸãšããŠã¯ãéãããããŒãžæ°ã®äžã§æãéèŠãªã³ã³ãã³ãã«ã¯ããŒã©ãŒã®æ³šæãåããããã«ãç¡é¢ä¿ãªããŒãžãã³ã³ãã³ãã®æŽæ°é »åºŠãäœãã»ã¯ã·ã§ã³ãæé€ããŠããŸãã
ãŸãããŠãŒã¶ãŒãšã¯ã¹ããªãšã³ã¹ãæãªãå¯èœæ§ã®ããäžæçãªã©ã³ãã£ã³ã°ããŒãžãããã¢ãŒã·ã§ã³ããŒãžãšãã£ãèŠçŽ ãrobots.txtã§ãããã¯ããæ€çŽ¢çµæã§ã®ãã©ã³ãã€ã¡ãŒãžãç¶æããŠããäŸããããŸãã
robots.txtã«é¢ãããããã質åãšãã®åç
robots.txtãã¡ã€ã«ã¯ãŠã§ããµã€ãã®ç®¡çè ãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã«å¯Ÿããæ瀺ãåºãããã®ãã¡ã€ã«ã§ããå€ãã®ãŠã§ããã¹ã¿ãŒã«ãšã£ãŠãã©ã®ããã«ãããSEOã«æé©ãªrobots.txtãèšå®ã§ããããäžã€ã®éèŠãªããŒãã§ããããã§ã¯ãrobots.txtã«é¢ããŠããå¯ãããã質åã«çããŠãããŸãããã
å€ãã®äººã¯robots.txtèšå®ã®åºæ¬ãç解ããŠãããã®ã®ããã®åŸ®åŠãªãã¥ã¢ã³ã¹ãé«åºŠãªå©çšæ³ã«ã€ããŠã¯ææ¡ãã¥ãããã®ã§ããäŸãã°ãç¹å®ã®ãŠãŒã¶ãŒãšãŒãžã§ã³ãã«å¯Ÿããã¢ã¯ã»ã¹èŠå¶ããã£ã¬ã¯ããªã¬ãã«ã§ã®è©³çŽ°ãªã¢ã¯ã»ã¹èšå®ã«ã€ããŠæ·±ãæãäžããŸãã
ãŸããrobots.txtãã¡ã€ã«ãçšããŠã©ã®ããã«ããŠSEOãåäžãããããšãã§ãããããã®ã¡ã«ããºã ãçåç¹ã«ã觊ããŠãããŸããèšå®ã®ãã¹ãæ€çŽ¢ãšã³ãžã³ã®ã€ã³ããã¯ã¹ã«ã©ã®ãããªæªåœ±é¿ãäžãåŸããããã®ãªã¹ã¯ãæå°éã«ãšã©ããææ³ã«ã€ããŠã解説ããŠãããŸãããã
ã¯ããŒã©ãŒããã®ããŒãžã®é衚瀺ã¯å¯èœã
ã¯ããŒã©ãŒã«ããŒãžãé衚瀺ã«ããããæ瀺ããäž»ãªæ¹æ³ã¯ãrobots.txtãã¡ã€ã«ã«ãDisallowããã£ã¬ã¯ãã£ããèšè¿°ããããšã§ããããã«ãããæå®ãããŠãŒã¶ãŒãšãŒãžã§ã³ãã¯ãã®ããŒãžãžã®ã¢ã¯ã»ã¹ãæåŠãããŸãã
ãããããDisallowããå©çšããé衚瀺æ瀺ã¯ãã¯ããŒã©ãŒãæ€çŽ¢ãšã³ãžã³ããã®æ瀺ãå°éããããšãåæã§ããå šãŠã®ã¯ããŒã©ãŒãrobots.txtã®ã«ãŒã«ã«åŸãããã§ã¯ãªããããå®ç§ãªé衚瀺ä¿èšŒã¯ãããŸãããäŸãã°ããã«ãŠã§ã¢ãæ¡æ£ããæªæãããããã¯ãæå³çã«ãããã®æ瀺ãç¡èŠããããšããããŸãã
ããã«ãããããŒãžããDisallowããããšããŠããä»ã®å€éšãµã€ãããã®ããŒãžãžã®ãªã³ã¯ãæã£ãŠããå ŽåãããŒãžã®URLãæ€çŽ¢çµæã«äžéšè¡šç€ºãããå¯èœæ§ããããŸããããã¯ãIndexed, though blocked by robots.txtããšè¡šçŸãããŸãããã£ãŠãã¯ããŒã©ãŒããå®å šã«ããŒãžãé ãæ¹æ³ãšããŠã¯ãã¡ã¿ã¿ã°ã«ãããnoindexãæ什ãããŒãžå ã«è¿œå ããããšãæå¹ã§ãã
robots.txtã®å€æŽã¯ãã€åæ ãããã
robots.txtãã¡ã€ã«ã®å€æŽã¯ä¿åããŠãŠã§ããµãŒããŒã«ã¢ããããŒãããæç¹ã§å³åº§ã«åæ ããããšèããã¡ã§ãããå®éã«ã¯ã¯ããŒã©ãŒããã®ãã¡ã€ã«ã次ã«èªã¿èŸŒãã¿ã€ãã³ã°ã§åæ ãããŸãããããã£ãŠãå€æŽã®å¹æãçŸãããŸã§ã«ã¯æéããããå ŽåããããŸãã
æ€çŽ¢ãšã³ãžã³ã«ãã£ãŠã¯ãrobots.txtãã¡ã€ã«ã®ã¯ããŒã«é »åºŠãç°ãªããŸããäžè¬çã«ãGoogleã®ãããªäž»èŠãªæ€çŽ¢ãšã³ãžã³ã§ã¯æ°æéããæ°æ¥ã§æ°ããrobots.txtãã¡ã€ã«ãæ€åºãåã¯ããŒã«ããŸããããµã€ãã®ã¯ããŒã«é »åºŠããµã€ãºã«ãå·Šå³ãããããäžæŠã«ã¯èšããŸããã
robots.txtã®å€æŽãããã«åæ ãããããšãéèŠãªå Žåã¯ãGoogle Search Consoleã®ãããªããŒã«ãå©çšããŠrobots.txtãã¹ã¿ãŒã䜿çšããå€æŽãæ£ããèªèãããŠãããããã¹ãããããšãå¯èœã§ããããã«ãããå€æŽãé©åã§ããããšã確èªããæ€çŽ¢ãšã³ãžã³ã«æ©ãã«å€æŽãèªèãããããšãã§ããŸãã
robots.txtããªãå Žåã®ããã©ã«ãã®æåã¯ïŒ
ãããŠã§ããµã€ããrobots.txtãã¡ã€ã«ãæã£ãŠããªãå Žåãæ€çŽ¢ãšã³ãžã³ã®ã¯ããŒã©ãŒã¯ãã®ãµã€ãã®å šãŠã®ããŒãžãã¯ããŒã«ããããšããã§ããããããã¯ãrobots.txtãã¡ã€ã«ãååšããªããã°ãã¯ããŒã©ãŒã¯ç¹å®ã®æ瀺ããªããããµã€ããå¶éãªãã«æ¢çŽ¢ããããšãããã©ã«ãã®æåã ããã§ãã
ãã®ç¶æ³ã¯ããŠã§ããµã€ããæ€çŽ¢çµæã«è¡šç€ºãããããšãæãå Žåã«ã¯åé¡ã§ã¯ãããŸããããã¯ããŒã©ãŒã«ããäžèŠãªãã©ãã£ãã¯ããã€ã³ããã¯ã¹åžæå€ã®ããŒãžãæ€çŽ¢çµæã«è¡šç€ºããããªã¹ã¯ããããŸãããããã£ãŠãç¹å®ã®ããŒãžãå ¬éããããªãããŸãã¯ã¯ããŒã©ãŒãã©ãã£ãã¯ãã³ã³ãããŒã«ãããå Žåã¯ãé©åã«æ§æãããrobots.txtãã¡ã€ã«ã®äœ¿çšãæšå¥šãããŸãã
robots.txtãã¡ã€ã«ããªãå Žåã®ããã©ã«ãã®æåãç解ããããšã¯ããŠã§ããµã€ãã®SEOæŠç¥ã«ãããŠã¯äžå¯æ¬ ã§ãããã®ãã¡ã€ã«ãé©åã«ç®¡çããããšã§ãæ€çŽ¢ãšã³ãžã³ãšã®é¢ä¿ãããè¯ããã®ã«ããæçµçã«ã¯ãµã€ãã®ããã©ãŒãã³ã¹ãé«ããããšãå¯èœã«ãªããŸãã
ãŸãšã
SEOã«ãããŠãrobots.txtã¯Webãµã€ãã®ã¯ããŒã©ãŒå¶åŸ¡ã«äžå¯æ¬ ã§ããrobots.txtãã¡ã€ã«ã¯ãæ€çŽ¢ãšã³ãžã³ããµã€ããã©ã®ããã«ã€ã³ããã¯ã¹ããããæ瀺ããæå¹ãªãµã€ããããã®æäŸãäœåãªã³ã³ãã³ãã®é€å€ãå¯èœã«ããŠãSEOããã©ãŒãã³ã¹ãæ倧åããŸããé©åãªæ§æã䜿ãããµã€ãæ§é ãç解ããäžã§ã®æ£ããå®è£ ãæ±ããããŸãããµããã¡ã€ã³ãåçã³ã³ãã³ããé©åã«ç®¡çããããšã§ãã»ãã¥ãªãã£ã匷åããSEOå¹æãæé©åã§ããŸããæåäºäŸããããã質åãåèã«ããŠãæ éãªãã¹ããšç£èŠãéããŠãã¯ããŒã«å¹çãæ¹åããŸããããããžãã¹ããŒãœã³ã®çãããæ£ç¢ºãªrobots.txtã®éçšã§SEOã匷åãããªã³ã©ã€ã³ã§ã®æåãç®æããŸãããã