Bots, crawlers, and linking

Web taxonomy is the system used to classify and organise the content on a website — the set of categories, subcategories, tags, and labels that group pages by topic, type, or attribute. It's the structural layer that turns a collection of individual [...] Read more

A sitemap is a file that lists the URLs on a website to help search engines discover and crawl them efficiently. The most common format is the XML sitemap, which follows the sitemaps.org protocol and is read by Google, Bing, and other crawlers. What [...] Read more

Internal linking refers to links between pages on the same website/domain, e.g. https://www.example.com > https://www.example.com/shop/ While this is a basic function of any website, internal linking is one of the most common missed SEO [...] Read more

A nofollow attribute on a link tells search engines not to associate the linking page with its destination — meaning the link should not pass ranking signals (often referred to as "link equity" or "PageRank"). nofollow in practice On an individual [...] Read more

Pagination is the practice of splitting a long list of items across multiple pages — /blog?page=2, /products/category/page/3/, /news/2026/01/. This is best practice on blog archives, e-commerce category pages, and any large catalogue. Done well, [...] Read more

A broken link is any hyperlink that points to a destination that no longer works — most commonly a URL that returns a 404 (Not Found), but also 5xx server errors, infinite redirect loops, or DNS failures. They occur on virtually every website at [...] Read more

A permalink is the full URL of a specific web page — designed to remain stable and unchanged over time. Permalinks are how content is referenced, shared, bookmarked, and indexed. Designing them well is a one-time investment that provides a [...] Read more

Anchor text is the visible, clickable text of a hyperlink. In the HTML below, "our pricing page" is the anchor text: <a href="/pricing">our pricing page</a> It's the most direct signal a link can give about the destination. Users rely on [...] Read more

robots.txt is a plain text file that sits at the root of a website and tells web crawlers which parts of the site they are allowed to request. If your domain is example.com, the file lives at example.com/robots.txt — and nowhere else. Crawlers check [...] Read more

"noreferrer" is a link attribute value used in an anchor tag () to instruct browsers not to pass the referrer information — the URL of the originating page — to the destination site when a user clicks the link. This means the linked-to site cannot [...] Read more

For years, the web ran on a simple understanding: search engines crawled your content and sent users back to you in return. AI crawlers broke that bargain. Many consume content to generate direct answers, often without sending a visit, a citation, [...] Read more

noopener is a value for the rel attribute on a link, used alongside links that open in a new tab, that is used for security, not search. When a link uses target="_blank", the page that opens gets a JavaScript reference back to the page that launched [...] Read more