robots.txt
A plain text file at the root of a site listing which bots may read what. It is a request rather than a lock — well-behaved crawlers obey it, and all the major AI crawlers do.
Related terms
Guardrails
The limits placed on what an AI may say or do — refusing certain requests, staying on topic, never inventing a price. Essential on anything customer-facing.
Model Drift
When a model gets worse over time because the world moved on and it did not. A pricing or demand model trained two years ago quietly stops matching how customers behave now.
AI Readiness
Whether a website can be reached, understood and quoted by AI assistants. It covers crawler access, structured data, clear headings, published contact details and speed — the unglamorous things that decide whether a model will name you.
Prompt Injection
An attack where hidden instructions in a web page or document trick an AI into ignoring its real orders. The reason a careful assistant treats page content as information, never as commands.
Deep Learning
Machine learning using neural networks with many layers. The depth is what lets a system learn complicated things — recognising a face, translating a sentence — without anyone specifying how.
Crawl Budget
The rough limit on how much of your site a crawler will fetch in a visit. Slow pages and endless near-duplicate URLs spend it on nothing, so the pages that matter get visited less often.