Should I Upload to My Server Both Robots Txt and Site Map

Let The Bots Do Their Work

A articulate plan should guide all of the actions taken to bring visitors to a website. In practice, some of those deportment accept place in the clear, while others operate more in the background.

The next two posts address behind-the-scenes steps that ameliorate website indexing and the probability that site visitors can quickly find relevant content. In this post we explain using a so-called robots.txt file to give search engines directives most indexing a website.  In the post-obit post, we discuss using search engine readable site maps to further assist visitors in finding relevant content.

Even if search referrals are not the primary site traffic source, a small investment in understand how to utilize a robots.txt files provides long term payoffs in effective site indexing and an enhanced company experience. Google Analytics, and similar web analytics services can place the historical proportion of a site's traffic referred by search engines and pinpoint the most relevant search engines.

Search Engines

For academy and college websites, particularly those looking to concenter overseas students there are four search engines or indexing crawlers that are likely to exist relevant, Google, Bing, Baidu and Yandex.

By placing directives in a robots.txt file Google, Bing and other search engines are given detailed instructions about what and what not to index on a site.  In other words, search engines tin exist directed to alphabetize relevant content and ignore 'less relevant content'.

Let's parse 'less relevant content". Newsletters and calendars from 1999 are less relevant to nigh site visitors than this year'due south versions. As are files used to operate the site, access your content management system or some dynamically generated website pages.

Why not direct search engines to the good stuff and ignore the less relevant? The content is still available, as are all the links, so visitors tin can still access it. The content is but less likely to clutter upwards search results and search engines don't waste time indexing content of depression potential value to site visitors.

The mechanism for directing search engines is to place a set of instructions in a robots.txt file stored in a website's elevation-level or root directory.

The residue of this guide explains how robots.txt works, provides clarification of some common misconceptions virtually robots.txt and describes what we find, in the wild, at academy and college websites.

robots.txt or no robots.txt

Without a robots.txt file, indexing crawlers will visit every page and follow every link on a site and use the underlying indexing algorithms to determine what results to nowadays in search results. That approach is not necessarily a bad thing, at all. Why? Considering crawlers do ii things:

• They recursively follow website URLs (links) and the content at those links that is accessible by a browser.  If no robots.txt file is present every link is accessed.
• The 2d thing that crawlers do is render content that needs JavaScript or other brandish files so that this content can also exist added to a site index.  In many cases robots.txt file directives inadvertently block access to the script or style canvass rendering files. As a issue, indexing tin can be incomplete. Mayhap of more than concern, Google uses rendering in assessing a site'due south mobile friendliness: blocking the script and style canvas files counts against mobile friendliness. And mobile friendliness boosts search result ranking.  If in that location is no robots.txt file indexing will be consummate and there will be no resource blocking.

What'south the downside of not having a robots.txt file? Three things.

First, there are many directories or folders in a website containing files that are not relevant to browser access and there is no compelling reason to index these files.  At that place is besides material that becomes less relevant over time, but may remain on a site for regulatory or other reasons: grade and academic calendars, form schedules and the like.  Site visitors are better served past being directed to the electric current material than navigating through electric current and old.

The 2d reason is that the robots.txt file can exist used to tell search engines where to discover the relevant XML-format sitemap or sitemaps.

Finally a robots.txt file tin can be used to block crawlers that you don't wish to access your site. However, as complying with robots.txt directives is voluntary, malicious 'bots' will likely ignore any directives.

In preparing this guide we reviewed the main or gateway domains of about 200 (northward=206) university and college websites belonging to Canadian college education institutions to understand current practice.  We'll discuss our findings a little later, simply 20% of sites (xviii.nine% or 39 /206) do not use a robots.txt file. And, there is no harm done.

Decision-making Where Search Engine Bots Crawl

If you desire to control search engine indexing you tin can do so through robots.txt file directives. Crawlers interrogate the robots.txt file to determine any constraints on their activity.  Directives on each line or record within the file provide instructions to the crawler.

Google, Bing, Baidu and Yandex recognise iv field elements with the post-obit structure:

Field		Values		Annotate	Ascertainment
user-agent	:	[value]	#	optional comment	user-agent = crawler accessing a site
permit	:	[path]	#	optional comment	allow directive permits access
disallow	:	[path]	#	optional comment	disallow directive prohibits access
sitemap	:	[URL]	#	optional comment	Directs crawler to notice an XML file at the specified URL.  This can be on some other server if needed.

The fields tin be organised into groups, sorted by user agent for as many private user-agents (eastward.g. Googlebot, Bingbot, Baiduspider, YandexBot etc.) equally needed.

There is no limit to the number of directives or records that the robots.txt file can contain, but Google ignores whatever robots.txt content subsequently the kickoff 500KB: roughly equivalent to 9,250 or more than records. Yandex imposes a smaller file limit of 32KB and assumes if a file is larger than the limit, everything is allowed.  In our survey, nosotros constitute no robots.txt file larger than 7KB.

[value] this can either be text for a specific crawler, e.g. Googlebot or Bingbot or a wildcard '*' to denote all crawlers. Most higher education robots.txt files permit all crawlers to have site admission.

[path] path operates as a relative position indicator to the location of the robots.txt file. As a result, / indicates the elevation-most or root directory or folder. Directories or files located lower down in the bureaucracy tin be specified by their relative position to the height-most folder.

Be circumspect to spelling as [path] can be case sensitive, depending upon a server and its configuration. Further, if a server is 'case-sensitive' and content assumes it is not, this volition result in broken links (404 errors) and robots.txt directives may non have their intended result.

[URL] a complete URL rather than a relative location tells the crawler where to observe any sitemaps. In principle, the sitemaps could be located at a different domain; in practice XML-format sitemaps are unremarkably placed in the root directory.

Putting this all together a 'consummate' robots.txt might wait similar this:

# This file lists local URLs that well-behaved robots should ignore

User-agent: *

Disallow: /registrar/athenaeum # one-time stuff

Disallow: /art/culture/ # one-time stuff

Disallow: /education/coursework/ # old stuff

Disallow: /events/day.php # search engines only need one calendar view, and so hibernate the residual

Sitemap: https://world wide web.examplecollege.ca/sitemap.xml

The #'s delineate comments to be ignored by a crawler, but inserted for readability.  Crawlers ignore bare lines, but these besides better readability.

Conflicting Directives

robots.txt files tin incorporate multiple directives. Attempts to include some directories for itch while excluding others can create conflicting instructions. To resolve this, crawlers process directives based on precedence. The principle is that the most specific rule takes precedence and other directives are ignored. For example:

user-agent: *

disallow: / #disallow all indexing of the site

allow: /physics #allow indexing of the directory physics and all of its sub-directories and their contents.

A crawler encounters the directory http://www.exampleu.ca/physics.  Every bit the allow rule is more specific that the disallow rule, it takes precedence and the directory volition be indexed.

robots.txt File Location

In order to directly the crawler in the intended manner a robots.txt file must be located in the top-near or root directory for a specific host, protocol and port number.  To explicate:

Crawlers see http://example.edu/ and http://cs.example.edu/ as 2 different hosts or domains. Placing a robots.txt file at http://example.edu/robots.txt will have no effect on the http://cs.example.edu/ domain. If y'all don't want to direct how the http://cs.example.edu/ is crawled, no harm done. If you exercise desire to directly activities you need to place a divide (but possibly identical) robots.txt file at http://cs.example.edu/robots.txt

Crawlers view http://example.ac.united kingdom of great britain and northern ireland/, https://example.ac.u.k. and ftp://example.ac.uk every bit three different protocols (which they are). If those protocols use the standard ports (fourscore, 443 and 21, respectively), *and* the resulting host and content are 1 and the same, then just 1 robots.txt file is required. If, however, a non-standard port is used, and so the robots.txt file accessed this way would only use to that service and thus, the others would need a dissever applicable robots.txt file placed in each of the root directories.

The issue of locating the robots.txt file is peculiarly of import for higher education websites. Typical academy or college websites are structured as federations of sub-sites: some of these as distinct domains other times inside sub-directories.  In the old case divide robots.txt files are needed in each sub-domain in the latter case a robot.txt file in the root directory is the simply manner to enforce the desired itch behaviour.

Processing Problems for robots.txt

Crawlers try to fetch the robots.txt file from its expected location or establish that a valid file does not exist. And crawlers pay attention to the response codes received from the endeavour and may modify their behaviour.  We take summarised the potential responses in the table.

	2XX Success	3XX Redirection	4XX Not institute	5XX Server error
permit	✔︎	✔︎	✔︎	X
disallow	✔︎	X	X	✔︎
conditional	✔︎	X	Ten	X
comment	Specific processing depends on the robots.txt content	If redirection results in a 2xx, then processing will exist as described in the 2xx column, otherwise processing takes identify every bit described in the 4xx cavalcade	Assumes in that location is no robots.txt, and so all files volition be crawled	Assumes a temporary error, during which no files volition exist crawled

What Do We Notice in Practice?

Nosotros surveyed but over 200 (north=206) Canadian academy and college websites and examined the robots.txt files located in the top-level directory of the gateway domain.

Thirty nine (39) sites (18.nine%) did non have a robots.txt file. As we stated up front end this but means that these sites are crawled in their entirety – save for any pages that have page-specific meta tags specifying that the page is not to be indexed or followed.  In a subsequent blog post we will examine current higher education sitemap practices to see if there is a correlation between the absenteeism of a robots.txt file and the absence or presence of an upwards-to-date XML sitemap.

The balance of 167 sites (81.1%) tin can be divided into iii different robots.txt formulations every bit follows:

Formulation 1 – the robots.txt file directives are structured in i of two master ways

user-agent: *     or      user-agent: *

allow: /                        disallow:

The two approaches are functionally equivalent to each other and to having no robots.txt file at all.  These configurations occur virtually 5% of the time.

Formulation 2 – the robots.txt file directives are structured in ii alternatives:

user-agent: *    or       user-amanuensis: *

disallow:                     permit: /

disallow: [path]           disallow: [path]

The two approaches are functionally equivalent to each other. Disallowing naught and assuasive all and then specifying a specific location to disallow, could be achieved by simply including a disallow: [path] directive. These configurations occur 32% of the fourth dimension.

Formulation 3 – the robots.txt file directives are structured as

User-agent: *

disallow: [path1]

disallow: [path2]

disallow: [pathN]

This configuration occurs 63% of the time and, in our view, is the configuration least prone to confusion. Almost x percent of sites as well include a directive to indicate the specific location at which a sitemap or sitemaps can exist establish and all of these use Formulation 3 for their robots.txt construction.

Determination

It is perfectly OK to non have a robots.txt file: this approach just results in all directories on a website existence indexed.  On the other paw, it is very straightforward to construct a robots.txt file that carefully segregates relevant content for indexing and places less relevant content in directories that will exist ignored. Moreover, a robots.txt file tin also specify the location for a sitemap or sitemaps that can further amend indexing efficiency and thus the power of site visitors to observe the good stuff.

Blog photograph prototype: unsplash.com / pexels.com

colemanforgand.blogspot.com

Source: https://www.eqafy.com/component/content/article/49-higher-education-research/217-using-robots-txt-files-on-higher-education-websites.html?Itemid=293

Share this post