Domain Name Forum  

Go Back   Domain Name Forum > Search Engine > Google > Google Sitemaps

Thread Tools Display Modes
Old 07-26-2009, 08:58 AM
SEO SEO is offline
Senior Member
Join Date: Jun 2009
Posts: 34,439
Default What is Robots.txt?

What is robots.txt in sitemap? What exactly are their function?
Search Engine Optimization
Domain Name Forum
Reply With Quote
Old 04-09-2012, 02:52 AM
sujith34 sujith34 is offline
Join Date: Sep 2011
Posts: 71

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from
Reply With Quote
Old 09-17-2018, 04:14 AM
richarddang911 richarddang911 is offline
Junior Member
Join Date: May 2018
Location: Vietnam
Posts: 26
Default robot.txt

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

dedicated server in Vietnam
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT -4. The time now is 05:27 PM.

| Webmaster Forum | | Domain Name Forum |

Powered by vBulletin®
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.