Limitations of the Web source
The Web source monitors many websites across the Web at large. However, this source has a number of limitations on types of web pages that can be found. Generally, a website can be monitored as long as it fits the following criteria:
- The website in question is publicly available from any location. While it doesn't necessarily have to be indexed on search engines, it needs to be directly accessible from any part of the world;
- The website in question doesn't limit crawler traffic. This means that there are no limitations for general bot traffic, or specifically Awario crawler traffic in the website's robots.txt file;
- The website doesn't adopt any special modes of access, like paid subscription, general sign up, an API token, or a full-on integration.
If you want to make sure Awario can pick up data from a particular website or web page, feel free to contact our support team and we’ll be more than happy to help. Please keep in mind that in case data from a given website proves to be unobtainable by Awario, there's nothing our team will be able to do.