How to Check Whether AI Crawlers Actually Reach Your Links

As today's 41-day test showed, a link injected by JavaScript can look perfectly normal to a human visitor while being effectively invisible to most AI crawlers. The catch is that you can't tell just by looking: your browser shows you the page after JavaScript has run, so what a crawler actually receives has to be checked separately.

There are three checks worth running: whether the link is in the raw HTML, whether each bot is actually visiting your site, and whether robots.txt is blocking it. None require a developer or special tooling, and working through them in order shows whether the problem is your link structure or a blocking rule.

Check whether the link is in the raw HTML

Start by checking whether the link exists in the raw HTML before JavaScript runs. Two ways to do this.

One is to turn off JavaScript in your browser and try clicking through the site's key paths — from a category list to a detail page, a "load more" control, or pagination at the bottom of a list. If a click does nothing, or the list itself is empty, that link only exists after JavaScript executes.

The other is to open "view source" on the page and look for the visible links as actual <a href="..."> tags in the raw source. If the screen shows a full list but only a few of those show up in the source, the rest were filled in by JavaScript afterward. This gap tends to be largest on category or tag pages with many entries.

Look for bot names in your server logs

AI crawlers identify themselves with a user-agent string when they hit your server. Open your server's access logs and check whether names like GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, GoogleOther, Bingbot, Meta-ExternalAgent, Amazonbot, and ChatGPT-User actually show up.

How you view access logs depends on your host or server tooling. If you run your own server, you can grep the log files for these names; with a separate logging tool or a CDN dashboard, filter by user agent instead.

Two things to look for: whether the bot visits your site at all, and whether it also shows up on URLs newly published or previously reachable only through JavaScript links. If a bot visits your list pages often but rarely touches the detail pages linked from them, the link between the two may be the thing it can't find. If it doesn't show up on the list pages either, the issue isn't the link — the bot isn't reaching the site at all, which points to the next check.

Google's own crawlers need to be told apart by name. In today's test, it was mostly GoogleOther that followed JavaScript-injected links, while Googlebot — the crawler that actually builds the search index — found far fewer of the same pages. Seeing "Google" show up often in your logs doesn't tell you which one it was; you have to check the exact user-agent string to know whether it was the crawler that feeds Search or a different one.

Check whether robots.txt is blocking it

If a bot never shows up in your logs at all, check robots.txt before assuming it's a link problem. Opening /robots.txt on your domain shows exactly which bots are allowed and which are blocked.

Some setups lump training crawlers (GPTBot, ClaudeBot, etc.) together with search-answer bots (OAI-SearchBot, PerplexityBot, etc.) under one blanket rule. If you want to show up in AI-generated answers, check whether these are split into separate rules. Since robots.txt is a convention bots follow voluntarily, if a bot is absent from your logs despite not being blocked there, check your firewall, CDN settings, or any bot-blocking service you run too.

Put all three together

If the link is missing from raw HTML but robots.txt is open, the fix is server-side rendering of the link. If robots.txt is blocking it, the fix is the rule itself. If the link is in the raw HTML and robots.txt is open but the bot still never shows up in logs, look outside robots.txt — a firewall or CDN rule is the likely cause. Looking at only one signal — "this bot doesn't show up" — can point you at the wrong cause.

After switching a link to raw HTML or fixing a robots.txt rule, check the logs again. In today's test, different bots returned at very different speeds — GPTBot re-crawled within two days of the fix, while others took over a week. Watch the logs over a few days rather than calling it a failure after a day or two.

This isn't a one-time check

In today's test, bot behavior kept changing across all 41 days. Meta-ExternalAgent didn't show up at all early on, then started crawling systematically only by day 41; Amazonbot went the other way and left the site entirely partway through. There's no guarantee that a result you confirmed once still holds after a site redesign changes your link structure, after you edit a robots.txt rule, or once a new bot name shows up.

Rather than checking once and assuming it stays fixed, it's safer to redo the same three checks — raw HTML, logs, robots.txt — whenever you add a new section or make a major structural change to the site. That matters especially for a frontend framework swap or a shift in rendering from server-side to client-side (or the reverse): check first whether your links still land in the raw HTML.