How a Crawler Works

A crawler usually starts with a list of known URLs and downloads their source code. From this code, it extracts new links and adds them to a queue for its next visit; over time, this creates a vast, ever-expanding map of the web. This map forms the foundation for nearly every subsequent step, from search indexes to specialized analytics tools and monitoring services.

Tip: An XML sitemap significantly speeds up this process because it provides the crawler with all the important URLs at a glance, rather than requiring it to discover them one by one via individual links.

This cycle of loading, analyzing, and tracking runs around the clock on an enormous scale, usually distributed across thousands of servers simultaneously. Even small websites are visited this way several times a day, often without the site operators even noticing, as long as no issues—such as a sudden algorithm update —come to light. It’s only by looking at the server logs that this constant bot traffic becomes truly visible, usually on a scale that surprises many site operators at first glance.

• Start using known URL lists
• Source code is being read
• New links are placed on the waiting list
• The process runs continuously and in parallel

Types of Crawlers

In addition to search engine crawlers, there are SEO tool crawlers that specifically check individual pages for errors, as well as price comparison and archive bots. Each type has its own specific purpose, even though the underlying technology remains similar. In addition, AI crawlers have now emerged that collect content for language models rather than for a traditional search index, and the boundaries between these categories are becoming increasingly blurred. This makes it more difficult for website operators to clearly distinguish between each individual type of bot.

• Search engine crawlers for the index
• SEO Tools for Error Checking
• Price Comparison Websites
• Archive Bots for Web History

Steering and Controlling Crawlers

The robots.txt file can be used to specify which areas a crawler is allowed to visit. Many online stores deliberately block important areas, such as the shopping cart system, to reduce server load and avoid duplicate content. A site’s technical SEO also plays a role in determining how efficiently a crawler can find the important sections in the first place and how much of the crawl budget is wasted on unimportant pages. A well-thought-out internal linking structure additionally directs the crawler specifically to the pages that really matter—a central component of any solid on-page SEO optimization.

• Robots.txt controls access
• The sitemap links to important pages
• Blocking Reduces Server Load
• Server logs show actual visits

Common Mistakes in Crawler Control

Entire directories are often accidentally blocked via the robots.txt file, for example after a relaunch or a new CMS installation. Such errors often go undetected for a long time because they aren’t immediately visible on the site itself, but only become apparent through declining visitor numbers and empty reports in Search Console. A quick test immediately after every major technical change usually uncovers such errors right away, rather than having to wait weeks to notice them.

• Overlooking Issues After a Relaunch
• Incorrect Paths in robots.txt
• Sitemap not kept up to date
• Errors Become Apparent Only Later

Understanding and Effectively Utilizing Crawler Budget

Every website receives only a limited amount of attention from a crawler, often referred to as a crawling budget. The number of pages visited and the time frame in which they are visited depend on the size of the website, its technical performance, and the frequency of new content. If this budget is wasted on unimportant or duplicate pages, important new content remains undiscovered for longer than necessary.

If you want to manage your budget effectively, you should consistently remove unimportant sections from the crawl structure and instead establish a clear hierarchy of category and detail pages. A well-organized structure that allows users to reach the most important page with just a few clicks is just as helpful as a regularly updated sitemap, as described in the Webmaster Tools Basics.

Even seemingly harmless URL parameters—such as those used for sorting or tracking—constantly generate new, technically distinct pages from a crawler’s perspective, thereby quietly consuming a significant portion of the available budget. A consistently applied canonical tag redirects attention back to the actual main version and prevents duplicates from unnecessarily tying up resources.

• A budget is, by its very nature, limited
• Unimportant pages waste capacity
• A clear page hierarchy provides a clear overview
• The current sitemap provides targeted redirects

Server Logs as a Tool for Crawler Monitoring

Server logs show exactly when a particular crawler visited a particular page, regardless of what browser analytics tools later display. This raw data thus provides a perspective that traditional web analytics tools naturally cannot capture, because they focus on human visitors, as described in the fundamentals of analytics in marketing.

It’s especially worth checking these logs regularly after technical changes—such as a relaunch—because they immediately reveal whether crawlers are encountering new error pages or whether important sections are suddenly being visited less frequently. Those who neglect this check often don’t notice the problems until weeks later, when they see a drop in visitor numbers.

If you want to take a closer look, you should also check whether a supposed bot actually comes from the specified provider, because some malware programs falsely pose as well-known crawlers to circumvent blocks. A simple comparison of the IP address against the official ranges of the respective provider can quickly clarify this.

• Bot visits can be clearly identified in the log
• Effectively complements traditional web analytics
• Especially important after technical changes
• Early Warning System for Crawling Issues

What AI Crawlers Do

Like any bot, an AI crawler downloads a page’s source code and analyzes the text. Unlike traditional crawling, however, this is rarely done for ranking factors, but rather to gather raw data for a language model or to provide an immediate, specific answer to a user’s question.

Warning: Many AI crawlers only partially respect robots.txt or use multiple user agents simultaneously; therefore, a single entry is often not enough to maintain complete control.

Some of these bots collect training data exclusively for future model versions, while others retrieve fresh content with every user query to generate an up-to-date response. This difference also determines how quickly your own content appears in responses and how long outdated information remains there without being automatically corrected when the page is refreshed.

• GPTBot collects data for OpenAI
• PerplexityBot retrieves content in real time
• Google Extended Affects Gemini Training
• ClaudeBot crawls for Anthropic models

Difference from Traditional Search Engine Crawlers

Googlebot crawls to include a page in an index for the search results. AI crawlers, on the other hand, draw either from a training corpus or a single, instantly generated response, without maintaining their own search index in the traditional sense. This lack of an index structure makes it even more difficult to reliably measure a site’s visibility to AI crawlers at all.

For website operators, this means double the work when it comes to management: A robots.txt rule for Google does not automatically apply to all AI crawlers, since each provider uses its own user agents and rules, which can also change on an ongoing basis. A list of allowed or blocked bots, once created, must therefore be regularly reviewed and updated—a task that has now become an integral part of every GEO agency’s work.

• Googlebot populates a search index
• AI crawlers feed models or responses
• Every bot needs its own rules
• Control requires ongoing maintenance

Control Visibility to AI Crawlers

If you want to appear in AI responses, you must actively allow AI crawlers; if you don’t want that, you must specifically block them. Both options are possible via the robots.txt file, but they require a conscious decision rather than a default setting, because a blanket block automatically excludes your own visibility in AI responses—even if that wasn’t your intention at all.

• Enabling Greater AI Visibility
• Blocking to Protect Your Own Content
• Regularly Check the robots.txt File
• New bots are constantly appearing

Identifying AI Crawlers in Server Logs

Server logs reliably show which AI crawlers are actually visiting a site, regardless of what is allowed or blocked in the robots.txt file. Regularly reviewing these logs reveals whether new bots are appearing or whether known crawlers have changed their visit frequency—which is often an early indicator of growing AI visibility. Without this monitoring, any assessment of your own AI visibility ultimately remains mere speculation.

• Identifying the User-Agent in the Log
• Track Visit Frequency Over Time
• Conducting Targeted Research on Unknown Bots
• Adjust the rules as needed

Step-by-Step Guide to Configuring an AI Crawler in robots.txt

If you want to control AI crawlers in a targeted manner, you shouldn’t limit yourself to a single blanket rule; instead, you should create a separate entry for each relevant bot. First, it’s worth taking stock: Which user agents actually appear in your server logs, and which of them should be granted access in the future? Only then should you proceed with the actual configuration of the robots.txt file, using individual blocks for each bot.

It’s also important to actually test the changes after they’re published—for example, using the testing tools in Search Console or by checking the logs again after a few days. This is the only way to determine whether a bot is actually following the new rule and whether its behavior has changed as intended.

It also makes sense to apply more granular restrictions on a per-directory basis rather than a single rule for the entire domain—for example, if you want to specifically allow access to a blog section while consistently blocking an internal customer portal. Some AI crawlers also ignore the “crawl-delay” field, which is why the actual access frequency can only be reliably monitored via the logs, not via robots.txt alone.

• Check the server log for existing bots
• Create a separate block for each user agent
• Test Rules After Publication
• Check the results after a few days

AI Crawlers and Your Own Sitemap Working Together

A well-maintained XML sitemap not only helps traditional search engines but also makes it easier for some AI crawlers to find up-to-date content—assuming the bot in question takes it into account at all. If you’re not yet familiar with setting up a sitemap, you’ll find the technical basics in an introduction to Webmaster Tools Basics. It’s also helpful to understand what it means when a page has been crawled, as this fundamental concept also helps in interpreting AI-specific bot behavior.

If there is no up-to-date sitemap or if it contains outdated URLs, AI crawlers will also waste resources on pages that are no longer relevant, instead of quickly indexing new or updated content. Regularly maintaining your sitemap therefore pays off twice over—both for traditional search rankings and for your visibility in AI-generated responses.

One detail that’s often overlooked is the date of the last update in the sitemap: If this field is filled out correctly, bots can more quickly identify which pages have actually changed since their last visit, rather than having to check every URL in its entirety again. This small technical adjustment is especially worthwhile for glossary pages that are updated frequently.

• The updated sitemap makes it easier to find what you’re looking for
• Outdated URLs waste resources
• Maintenance Benefits SEO and AI
• Understanding the Basics of Crawling

How a Page Is Crawled

Googlebot follows links from one URL to the next, downloading the complete source code of each page in the process. This visit alone is called crawling, regardless of what happens to the collected data later. The bot also keeps track of when a page was last visited and schedules its next visit accordingly, usually based on the URL’s previous frequency of updates.

Warning: An incorrectly configured robots.txt rule or a missing noindex tag will prevent crawling, even if the page’s content is flawless and it functions without technical issues.

How often a page is visited depends on what’s known as the crawl budget. Pages with regular updates and strong internal linking are given priority, while neglected subpages are crawled less frequently, meaning updates aren’t visible until some time later. Especially for large websites with thousands of subpages, this budget has a noticeable impact on which sections are visited promptly and which are visited only occasionally.

• The bot follows internal and external links
• Source code is loading completely
• Robots.txt can block access
• Crawl budget controls the frequency of visits

Crawled does not necessarily mean indexed

Crawling and indexing are often treated as the same thing, but they are two separate steps in the same process. A page may have been visited without subsequently appearing in search results—for example, due to thin content, duplicate content, or conflicting instructions in the source code that signal to the bot that it should ignore the page.

Search Console shows, for each URL individually, whether it has been crawled but is not currently indexed, and usually provides the reason right away. If you check this report regularly, you’ll spot problems much sooner than by simply monitoring rankings, since drops in rankings usually don’t become apparent until after a noticeable delay.

• Crawling is always the first step
• Indexing will follow afterward
• Thin content slows down indexing
• Each URL has its own status

What an XML Sitemap Contains

Technically, a sitemap is an XML file that provides additional information for each URL, such as the last modification date, the frequency of updates, or a relative priority. Search engines like Google use this information to identify new or updated pages more quickly, rather than relying solely on internal links to navigate the site.

A sitemap does not guarantee indexing; it is merely an invitation to the crawler to visit certain pages.

Especially for very large websites with thousands of subpages or for newly launched domains without many backlinks, a well-maintained sitemap significantly speeds up the process of discovering new content. If important pages are missing from the file or if it contains incorrect URLs, valuable crawl budget is wasted.

• Last Modified Date by URL
• Relative priority of individual pages
• Refresh Rate as a Guide
• Preventing Duplicate Content

Why Sitemaps Are Important for Rankings

A sitemap is no substitute for good internal linking or on-page optimization; rather, it complements both of these measures. Google Search Console allows you to see how many of the submitted URLs have actually been indexed and where discrepancies occur. Such reports often provide the first indications of technical issues such as faulty redirects, blocked directories, or orphaned pages without internal links.

• Check for Discrepancies Between the Sitemap and the Index
• Detect Faulty Redirects Early
• Find orphaned pages without internal links
• Check indexing reports regularly

The loading speed of the sitemap file itself also plays a role, especially if it contains several thousand URLs and is served by a slow server. Crawlers can read a compressed, well-structured file more reliably and quickly in its entirety than a disorganized version several megabytes in size that lacks a clear structure.

Sitemaps in Technical SEO Practice

Creating a sitemap is closely linked to other technical factors, such as crawling and page load time. For very large domains, it’s recommended to split the sitemap into multiple sitemap files with a parent index so that search engines can understand the structure more quickly. In the event of a relaunch, the sitemap must also be updated and resubmitted; otherwise, it will continue to point to old URLs that no longer exist.

• Split Large Domains into Multiple Files
• Create a Parent Sitemap Index
• Resubmit after each relaunch
• Consistently Remove Old URLs

An Overview of Different Types of Sitemaps

In addition to the standard sitemap for text pages, there are specialized versions for specific content types. An image sitemap helps search engines specifically index graphics on a page; a video sitemap provides additional information such as duration or a thumbnail; and a news sitemap is relevant for editorial teams with highly time-sensitive content because it is parsed particularly quickly. For most small and medium-sized websites, a single, well-maintained sitemap covering all pages is entirely sufficient; larger portals with diverse content formats, on the other hand, benefit from a clear separation by type. Which option makes sense depends on the content focus of the respective website and can usually be expanded incrementally without having to completely rebuild the existing structure.

• Image Sitemap for Graphics and Photos
• Video Sitemap with Duration and Thumbnail
• News Sitemap for Recent Editorial Articles
• A sitemap is sufficient for smaller websites

Regardless of the type you choose, the following applies: A sitemap should contain only real, accessible pages and be updated automatically on a regular basis so that it does not become outdated over time and send crawlers to dead links.

Sitemap: Common Mistakes in Practice

Over time, many sitemaps come to contain URLs that no longer exist because pages have been deleted or renamed without the file being updated accordingly. Redirects also frequently end up in the sitemap unnoticed, even though it should contain only the final, actually accessible destination addresses. A content management system that automatically generates sitemaps can reduce these errors, but it is no substitute for regular manual checks.

Consistently maintaining the URL structure also has a direct impact on the sitemap: If you change permalinks retroactively without cleaning up old entries, you’ll end up with permanently broken links.

• Remove Deleted Pages from the Sitemap
• Enter only final destination addresses
• Don’t blindly trust automatic generation
• Consistently update permalink changes

Combine Sitemap Data with Analytics Tools

A well-maintained sitemap only provides a complete picture when combined with actual user data. Google Analytics can be used to check whether URLs from the sitemap are actually receiving visitors or whether certain sections are rarely visited despite being indexed.

This combination often reveals more quickly than crawling reports alone which pages should be revised or removed from the site structure because they are not relevant to either search engines or visitors.

• Compare Sitemap Data with Actual Visitor Numbers
• Identify areas with low foot traffic
• Prioritize revisions based on data
• Crawling reports alone are not enough

What the robots.txt file actually does

The file is located in the root directory of a domain and consists of simple text rules. Each rule targets a specific crawler via the “User-agent” entry and uses “Disallow” or “Allow” to specify which paths the crawler is permitted to visit. Google, Bing, and other search engines read this file before each crawl and generally adhere to the specifications reliably.

A single incorrect “Disallow” entry in the root directory can be enough to remove the entire domain from Google’s index.

That’s why this file is one of the most sensitive technical settings on a website. Even a single forgotten slash or a path that’s too broad can block sections that should actually be visible. Anyone who makes changes to the file should always test the results afterward before the new version goes live.

• Selectively Allow Access for Individual Bots
• Exclude Entire Directories from Crawling
• Set the path to the sitemap
• Allocate the crawl budget to important pages
• Protect Duplicate Content from Being Indexed

Controlling Traditional Search Engine Crawlers

Googlebot, Bingbot, and similar search engine crawlers re-read the robots.txt file on every visit. Specific user-agent lines can be used to define a separate rule for each bot—for example, to allow Bing to access different sections than Google. Google Search Console offers a dedicated testing tool for this purpose, which allows you to check individual URLs against the current file before a change actually takes effect. Especially for large online stores with filter pages or shopping cart paths, a clean configuration prevents valuable crawl budget from being wasted on irrelevant pages.

• Set Custom Rules for Each Search Engine
• Exclude filter and shopping cart pages
• Use the testing tool before making any changes
• Define the crawl delay as needed

AI Crawlers and the New Bot Landscape

In addition to traditional search engines, a growing number of AI crawlers now read the robots.txt file, including GPTBot, ClaudeBot, Google-Extended, and CCBot. By configuring custom user-agent entries, you can specify whether these systems are allowed to collect content for training data or use it for live responses. This control has become an integral part of a sound technical visibility strategy, as is also assessed in our SEO/GEO audit. Even during a relaunch, reviewing the robots.txt file is one of the first steps to ensure that no unintended blocks are carried over to the new site. If you also want to keep an eye on structured ranking factors, you’ll find further insights in our article on on-page optimization.

• Explicitly Allow GPTBot and ClaudeBot
• Set up Google Extended separately for AI training
• Check the rules before every relaunch
• Regularly audit technical visibility

Properly Maintain and Test Your robots.txt File

This file is not a one-time task; it must be updated whenever there is a major change to the site structure. New directories, renamed paths, or a change in the content management system often alter which areas should actually be crawled. Regularly checking Search Console reveals whether Google is encountering blocks that are no longer intended or whether important areas are inadvertently remaining blocked. External technical analysis tools also help highlight differences between the current and a previous version of the file before they turn into a real visibility problem.

• Check the file after every structural change
• Check Search Console alerts regularly
• Check Old Restrictions for Relevance
• Document changes before going live

Robots.txt: Common Mistakes in Practice

The most common mistake is a Disallow entry that is too broad, which inadvertently blocks entire sections of a website even though only a single directory was intended. Conflicting rules—where a later line overrides an earlier allowance—also frequently lead to unclear crawling behavior. Especially when switching content management systems, the file is often carried over unchanged, even though the path structure has completely changed.

Another common issue involves URL structures that change over time: Old “Disallow” rules then point to paths that no longer exist, while new, genuinely sensitive areas remain unprotected.

• Don’t make “Disallow” entries too broad
• Consistently resolve conflicting rules
• Re-check the file after a system change
• Regularly Remove Outdated Paths

Robots.txt in Conjunction with Tag Management

Tools like Google Tag Manager often load external scripts from additional domains, which may themselves have their own crawling rules. If you’re only focusing on your own robots.txt file, it’s easy to overlook the fact that an embedded script on another domain has entirely different specifications.

For more complex setups involving multiple integrated services, it is therefore a good idea to regularly take stock of all the domains involved, not just your own main domain.

• External scripts come with their own rules
• Don’t Just Focus on the Main Domain
• List the domains involved on a regular basis
• Update the setup for new services

Those who view robots.txt, the sitemap, and the technical structure as an integrated system rather than as separate, individual tasks will spot errors more quickly and avoid a situation where a change in one place triggers unnoticed side effects in another place that don’t become apparent until weeks later.