robots.txt
robots.txt is a file at the root of a domain that tells crawlers which paths they may request.
Why it matters
Because it is a single file governing the whole host, a bad deploy of robots.txt is one of the few changes that can affect every URL at once — and it is silent until something downstream notices.
