Awario Crawlers

General information about Awario crawlers.

A bot, also known as a web robot, web spider or web crawler, is a software application designed to automatically perform simple and repetitive tasks in a more effective, structured, and concise manner than any human could ever do. The most common use of bots is in web spidering or web crawling.

AwarioSmartBot and AwarioRssBot are legitimate web crawlers sent by Awario to discover and collect new and updated web data (that is further used by Internet marketers from all over the world).

The AwarioSmartBot and AwarioRssBot crawlers are identified by having user-agents of the following form:

Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)
AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)
AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com)

The AwarioSmartBot and AwarioRssBot crawlers can be identified by the user-agent above. Please do not try to identify the bots via IP as we do not use any consecutive IP blocks.

We care about your site's performance and will never hurt it!

AwarioSmartBot and AwarioRssBot are very site-friendly crawlers. We made them as "gentle" as possible when crawling sites: they make only 1 request per 3 seconds, or even less frequently, if another crawl delay is specified in your robots.txt file. The bots respect rules you specify in your robots.txt file.

If any problems arise, they may be due to peculiarities of your particular site, or a bug on another site linking to you. Therefore, we would like to ask you — if you noticed any problem with Awario bots, please report it to bots@awario.com. We will quickly make unique settings for your particular site, so that the crawling will never affect your site's performance.

Why are they crawling my site?

AwarioSmartBot and AwarioRssBot collect Web data that is further used by Awario application to let Internet marketers find who is mentioning their brand online.

If the AwarioSmartBot/AwarioRssBot crawlers have visited your page, this means that its content has never been collected and analyzed or needed to be refreshed. For this reason you will not see recurring requests from the AwarioSmartBot/AwarioRssBot crawlers to the same page.

Blocking with robots.txt

AwarioSmartBot and AwarioRssBot are:

  1. Collecting only publicly available information that can be accessed by any random visitor. In case you think the crawlers collect some sensitive information, please remove it from public access.
  2. Cannot overload your site and do any harm to it – Awario bots are designed to be very polite and they can make only 1 hit per 3 seconds max. Besides, you can easily slow the bots by directions in the robots.txt file.

However, even though Awario bots are harmless, you still may want to prevent bots from crawling your site. The easiest and quickest way to do this is to use “robots.txt”. This text file contains instructions on how a bot should process your site data.
With a robots.txt file you may block the Awario crawlers from your site or slow them down as shown in the following examples:

Block specific parts of your site:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /private/

Block entire site:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /

Slow the crawlers:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Crawl-delay: 10

More Information About Bots

For more information about bots, please refer to http://www.robotstxt.org/.

Getting Support

If you still have any questions about AwarioSmartBot or AwarioRssBot, please contact us at bots@awario.com and we will respond as soon as possible.