robots.txt
A plain text file at the root of a site listing which bots may read what. It is a request rather than a lock — well-behaved crawlers obey it, and all the major AI crawlers do.
Related terms
OAI-SearchBot
The crawler OpenAI uses to answer live searches in ChatGPT, separate from GPTBot. Blocking one does not block the other, so a site can be absent from ChatGPT search while still being read for training.
Rich Result
A search listing with something extra — stars, prices, an image, an FAQ dropdown. Rich results come from structured data, and they take up more of the screen than a plain blue link.
Agentic Browsing
An AI navigating a website on somebody’s behalf: finding the price, checking availability, sometimes buying. Sites that rely on hover menus and unlabelled buttons are hard for an agent to use.
Computer Vision
AI that interprets images and video — reading a document, recognising a product, counting people. The part of AI that has been quietly working in industry for years.
Sentiment Analysis
Automatically judging whether text is positive, negative or neutral. Used to read reviews and messages at a scale nobody could read by hand.
Vector Database
A database built to store embeddings and find the closest matches quickly. It is the part of an AI search system that decides which of your pages is relevant to a question.