Glossary

Key terms and concepts related to AI agents, web crawlers, bot management, and the agentic web — explained for site managers and developers.

Agent Fingerprinting

Identifying AI agents through a combination of technical signals beyond just the user-agent string.

The emerging paradigm where AI agents autonomously browse, interact with, and transact on websites.

A search platform that uses AI to generate direct answers with citations instead of traditional link results.

A web crawler that collects content to train artificial intelligence and large language models.

Techniques for identifying automated visitors versus human users on a website.

The practice of detecting, classifying, and controlling automated traffic on a website.

An AI system that controls a real web browser to browse, interact with, and complete tasks on websites.

A technique that prevents automated scripts from accessing page content by requiring JavaScript execution.

The number of pages a search engine will crawl on your site within a given time period.

A proposed standard file (like robots.txt) that provides AI language models with a site summary and key information.

A text file at a website's root that tells crawlers which pages they can and cannot access.

Machine-readable markup (like JSON-LD) that helps search engines and AI agents understand page content.

An HTTP header that identifies the software making a web request, such as a browser or crawler.

An automated program that systematically browses the web to discover and index content.

The automated extraction of data from websites, typically at scale.