Crawl Systems

Crawl budget, robots.txt, XML sitemaps, discovery mechanics and Googlebot behaviour. The infrastructure layer that decides whether search engines find your pages in the first place.

Illustration of a character examining four floating element cards representing the loc, lastmod, changefreq, and priority elements of an XML sitemap, with each card showing a recognisable icon

AI Search Sitemap Elements

By Victor Ijomah

What Each XML Sitemap Element Does

The four XML sitemap elements have specific jobs, specific formatting rules, and…

Illustration of a character examining a structured XML sitemap document with annotation lines pointing to different elements of the file, communicating an anatomical breakdown of the structure

AI Search Sitemap Anatomy

By Victor Ijomah

The Anatomy of an XML Sitemap (with Example)

You've read about XML sitemaps. Here's what one actually looks like, with…

Illustration of a character presenting the sitemaps.org protocol document with annotated XML structure, alongside abstract search engine crawler icons representing the multi-party agreement

AI Search Sitemap Series

By Victor Ijomah

The Sitemaps.org Protocol Explained

Every XML sitemap on the web follows the same spec, agreed by…

Illustration of a character at a decision point evaluating different website types (e-commerce, blog, single-page, documentation) to determine whether each site needs a sitemap

AI Search Site Planning

By Victor Ijomah

Do I Need a Sitemap for My Website?

Most websites benefit from a sitemap, but a few genuinely don't. Here's…

Illustration comparing a sitemap file as an invitation list and a robots.txt file as a set of access rules, with a search engine crawler reading both at the root of a website

AI Search Crawler Management

By Victor Ijomah

Sitemap vs Robots.txt: What’s the Difference?

Both files sit at the root of your domain, both speak to…

Illustration showing a comparison between an XML sitemap file and an HTML sitemap page, with a character evaluating both options against a search engine crawler and a human visitor

AI Search HTML Sitemap

By Victor Ijomah

XML Sitemap vs HTML Sitemap: Which One Do You Need?

Two completely different files share the word 'sitemap', and most sites only…

Illustration of a character surveying multiple sitemap format cards representing XML, HTML, image, video, and news sitemap types

AI Search Sitemap Series

By Victor Ijomah

Types of Website Sitemaps and When Each One Matters

There's more than one kind of sitemap, even if SEO articles tend…

Illustration of a sitemap connecting pages on a website, with indexing status and crawl budget panels

AI Search Sitemap Series

By Victor Ijomah

What Is a Sitemap? Definition, Example & When You Need One

A sitemap is a list of URLs you hand to search engines.…