At Audigent, we are committed to providing targeted, relevant advertising solutions by collecting metadata from web pages. Our technology enables us to deliver advertisements that align with user interests and enhance the overall browsing experience. By crawling websites, we gather essential metadata that helps us and our advertiser partners bid on ad space, which in turn can generate revenue for publishers.
We only crawl websites where Audigent or our advertiser partners may serve ads, ensuring that our data collection is purposeful and limited to advertising contexts.
Audigent Web Crawlers
We use two different crawlers to collect metadata:
1. AudigentAdBot
The AudigentAdBot crawler is designed to collect only the metadata included in the header of HTML pages. It does not scrape any content from the body of the page. This means that AudigentAdBot is a lightweight, focused tool for gathering information strictly necessary for advertising purposes.
If you wish to block the AudigentAdBot crawler from accessing your site, you need to explicitly disallow it in your robots.txt file using the following rule:
User-agent: AudigentAdBot
Disallow: /However, using a general wildcard rule such as:
User-agent: *
Disallow: /will not block AudigentAdBot. It needs to be specifically mentioned.
Example of full user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible;
AudigentAdBot; +https://audigent.com/bot.html) Chrome/W.X.Y.Z Safari/537.362. AudigentBot
The AudigentBot crawler has broader capabilities and can scrape any content available on the page, including content from both the header and body. Unlike AudigentAdBot, AudigentBot can be blocked using a general wildcard rule in your robots.txt file, such as:
User-agent: *
Disallow: /This ensures that AudigentBot will not crawl your site if you prefer not to allow access.
Example of full user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; AudigentBot;
+https://audigent.com/bot.html) Chrome/W.X.Y.Z Safari/537.36Why Allow Crawling by Audigent?
Allowing our crawlers to access your site can be beneficial. By enabling Audigent to gather metadata from your pages, we can better target relevant ads to your audience. In turn, this may lead to increased bids on your ad inventory and higher revenue potential for your site.
