The Meta robot tags are seen since from long but with a couple of newly added features in recent past years has made them more useful for SEO. The Meta robot tag is an integral part of any web page which showcases different information to search engine robots. These tags are written within the section in the relevant HTML file and are known to define both or either of these actions. Earlier the main purpose of using these tags was simply to prevent the web page indexing. However, since few years a number of popular search engines have rendered additional support for the Meta Robots Tag. The below are some of the ways of using Meta robots tag in your website.
If you do not wish to see your web page in any search engine, you can simply avoid using the tag. However, all the popular search engines will index any web page which they come across by default. In fact, there are specific ways of forming Meta robot tag wherein you can use it clearly to notify different search engines to index your desired web pages. You do not require any admin action from the server end to get the desired outcome. Instead over the web pages by putting the individual directives with the head tag and you can do the needful. All these universal directives are now supported by the various popular search engines.
If you look at the Meta robots tag, it can be classified into four vital functions for the search engine crawlers. The first command is ‘Follow’ which tells the search engine crawler simply to follow different links found in that particular web page. The second command is ‘Index’ which tells the search engine crawler to simply index that particular web page. The third command is ‘Nofollow’ which tells the search engine crawler to avoid following the links found in that particular web page. The fourth and last command is ‘Noindex’ which tells the search engine crawler to simply avoid indexing that particular web page. Besides, there are few more commands for using Meta robots tags but the above discussed four are considered as the main functions.
Using the Meta Robots Tag
When you have to use these tags for multiple content values, it is recommended to put all your content values simply in one Meta tag. This will make the Meta tags quite readable and also decreases the chance for different conflicts. For example you can write in this fashion-
If you see several similar kinds of Meta tags over the web page, you are supposed to consolidate all the content values. This can be carried out by writing down the command in this fashion – and
In case if you see any conflict in these content values, you are then supposed to use the more restrictive kinds of tags. Hence if you find such tags – META NAME=”ROBOTS” CONTENT=”NOINDEX”> or , you are should follow the Noindex value.
Dealing with unnecessary kind of content values
Going by the default, the Google robots or Googlebots would index a web page and would be seen following the links associated to it. Hence for such cases you are not supposed to tag pages with the Follow or Index content values.
Directing the robots Meta tag specially at Googlebot
In order to render instructions to different search engines you are supposed to define the Meta name as Robots, whereas to give a command to only Googlebot the Meta names has to be renamed as Googlebot. In case if you are keen to give numerous instructions for a various popular search engines, it is recommended to use some specific kinds of Meta tag for each and every search engine rather using a generic type of robots Meta tag combined together.
Using robots.txt file and robots Meta tags together
If you come across some conflicts between Meta tag and robot.txt instructions in any web page the Googlebots would simply follow the most restraining kinds of commands. Hence when you block a web page with robots.txt you will never be able to see Googlebot crawling inside your website page and thus wouldn’t read the Meta tags seen over the page. Also, in case if you put robots.txt in any of your webpage and at the same time prevent it getting indexed via a Meta tag, Googlebot would though access the web page and even read the Meta tag but would certainly not index it.
The Meta Robots Tag plays an important role in search engine results and by following these tips and tricks of using the Meta Robots Tag as discussed above can help you in using them effectively.