AI Wants To Index The Web
Conversational Search Sets The Stage For Generative Experiences
We’re entering a critical next step in the evolution of the Web, and that is a combination of transparency and formality in how AI engines such as Open AI’s ChatGPT will crawl it. Earlier this week, Search Engine Journal reported on the announcement of “GPTBot”—Open AI’s initiative to train future versions of its GPT models based on GPT indexable content. Open AI, as detailed in Search Engine Journal, allows operators of sites to either opt in or out of the AI Web crawling with simple lines of code.
What’s worth noting is how this mirrors significant developments in the early days of the Web, before AI. One of the perks of being this advanced in my career years is that I got to work through all of the significant advancements of the Web, starting from “brochureware” Websites to where we are now with AI (See above visual). LLMs such as ChatGPT/Bing and Bard are at the same stage as early search engines, which took a chaotic and massive network of information and made things findable. That’s where we are with AI Web crawling today—anyone who manages a Web property and wants to be highly visible or less visible has several options to make it so.
But in the age of AI, we have to ask ourselves why we’d want to be highly visible or somewhat invisible to the AI engines of the world. It probably boils down to one of two things:
1. You believe people will use platforms like ChatGPT/Bard/Bing in addition to or in place of search engines and want to be found
2. You don’t trust AI to represent your Web information, and/or you don’t want your content used to train AI systems
To that second point, a recent piece on Business Insider highlighted the notable trust issues associated with AI and described not opting out of being crawled by GPTbot as “self-sabotage.” From Business Insider:
It's self-sabotage to let OpenAI's GPTbot crawl your website. This realization is spreading pretty swiftly among online communities. The Verge, a digital publication that competes with Insider, looks like it took steps to block GPTbot already.
I believe that history is repeating itself, and much like any Web-related venture is highly dependent on the search results that come back in Google., this AI indexing of the Web will net out similar results. GPTBot and any other AI Web crawlers are part of the next chapter of SEO (Search Engine Optimization) battles to be fought. However, unlike Google and to the point raised by Business Insider, the benefit to those being crawled isn’t as straightforward (at least in the GPTBot model) as links to source materials are replaced by more conversational responses to prompts—the value exchange for those providing the source material isn’t as tangible.
Still, in a future filled with AI, is it better to be visible than invisible?
Toward A Future of Generative Experiences
Regardless of how some of these trust issues get resolved (an opportunity for Bard here), the Web is steadily moving toward conversational interfaces, and these conversational interfaces, when combined with the innovation we are seeing with Generative AI, means we’re going to be getting used to more and more generative experiences. Adobe Firefly empowering you to prompt your way through Photoshop? That’s a Generative experience, as are all of the AI apps, features, interfaces, and services to come. Conversational search (Bing/Bard etc.) is just the starting point.
I always respect your ability to synthesize complex things down to a simple visuals.