A 41-Day Test Finds AI Crawlers Miss Links Injected by JavaScript

Vinicius Stanula, Associate Director of SEO at LOCOMOTIVE, ran a 41-day test on a Brazilian business-classification directory site he owns, and found that most AI crawlers largely fail to discover links injected by JavaScript.

The site has about 2,400 pages in total; the test covered 1,062 pages reachable through 21 top-level sections and their sub-categories. Eleven sections kept hard-coded child links in raw HTML (748 pages), while ten sections had child links injected by JavaScript after the page loaded (293 pages). Sitemaps were made to return 404s and breadcrumbs and hierarchy panels were removed, so there was no alternate discovery path.

Server logs tracked Googlebot and GoogleOther, OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Bingbot, Meta-ExternalAgent, Amazonbot, ChatGPT-User, and PerplexityBot.

Through day 27, of the 293 JavaScript-injected pages, Googlebot found 7 (2%) and GoogleOther found 142 (48%). GPTBot, ClaudeBot, Meta-ExternalAgent and Amazonbot found none (0%), and Bingbot found only 3 (1%). The picture reversed for the 748 hard-coded HTML pages: GPTBot and ClaudeBot found all 748 (100%), GoogleOther found 495 (66%), and Googlebot found only 35 (5%).

"Only Google executed the JavaScript and followed the injected links," Stanula wrote, "and almost none of that work came from Googlebot, the crawler that actually builds the search index."

Starting on day 27, Stanula converted the JavaScript-injected links in those sections to hard-coded HTML. GPTBot swept up 250 previously unreachable pages within two days, and Bingbot added 212 more within a week. Meta-ExternalAgent didn't begin crawling systematically until day 41. ClaudeBot only recrawled pages it already knew about and found no new ones, and Amazonbot had already left the site before the conversion. Google's crawlers barely moved either — by day 41, Googlebot had found only one new page in the converted sections.

Stanula argues that "Google renders JS" is only half the picture: "Rendering happened, but the crawler that feeds Search reached 2% of my JS-linked pages." His recommendations: audit what the raw HTML actually contains (disable JavaScript and click through critical paths, or diff the raw source against the rendered DOM), and prioritize server-side rendering for links specifically. Fixes also paid off quickly — GPTBot re-swept within 48 hours of the conversion, and Bingbot within a week.

Why it matters

This is a single test on one Brazilian directory site, and crawl behavior can vary by site structure and traffic, so it shouldn't be generalized wholesale. Still, the bot names in the test (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others) are the same ones that show up in any site's own server logs, so whether your category and detail pages sit in raw HTML — and whether those bots actually show up in your logs — is something you have to check directly.

Source: Search Engine Land, published August 19, 2026 (by Vinicius Stanula). Checked September 20, 2026. Read the original article