robots.txt
A plain text file at the root of a site listing which bots may read what. It is a request rather than a lock — well-behaved crawlers obey it, and all the major AI crawlers do.
Related terms
Author Authority
The weight given to content because of who wrote it. Advice attributed to a named person with a real background is treated very differently from advice signed by a company or by nobody.
Knowledge Graph
A map of things and how they relate — this company, its founder, its address, its services. Search engines and AI systems use one to know that two mentions of a name refer to the same business.
Google-Extended
The robots.txt setting that controls whether Google may use your content for Gemini and AI Overviews. It is separate from ordinary Google Search, so you can allow one and refuse the other.
Organization Schema
Structured data describing the business itself: name, logo, address, contact details, social profiles. The single most useful markup for being recognised, and the one most small sites are missing.
User-Agent
The name a browser or bot gives when it requests a page. It is how a server tells Googlebot from ChatGPT from a person, and how robots.txt rules are addressed to one and not another.
AI Bias
When a system produces unfair results because its training data reflected an unfair world. It matters commercially as well as ethically: a biased model makes confidently wrong decisions about real people.