How are users using chat bots to search?

Now I’ll admit this is something I’m less familiar with from a user-perspective. I’m not someone who uses many AI tools and instead prefers using search engines to find information online. I can’t give an in-depth breakdown of every popular AI search engine and how they behave. I do, however, understand the user story of how users of these AI search engines interact with my website.

The general user stories for these platforms, are that people are asking a question, and getting an answer that provides an in-depth breakdown. Similar to what reading a blog article provides but constructed by an LLM. These systems do link to their sources generally (more on that later!), but from my experience they do not click on the sources for the most part. While some users do, a majority are content with what they’ve been given by the AI bot and move on. Some might ask further clarifying questions if they have them, but they rarely appear to actually leave the AI interface.

What do these AI search engines appear to be doing?

Using Cloudflare’s AI Audit tools, I’m able to identify traffic from various AI bots, assistants, and crawlers. I’m not sure if this is due to me being on a free account, but my data appears to be automatically deleted after 7 days, so I only ever have the last 7 days to go off. The specific AI Assistant I want to focus on in this UI is the ChatGPT-User bot, which is the bot that ChatGPT uses to query information based on a user’s search request. This means theoretically, that each hit from this bot implies a user has asked a question where the requested article is at least seemingly relevant.

The other tool I want to focus on, although to a lesser extent due to the lack of analytics, is the automated Google AI Overview that sometimes shows up in search results. For this case, I’m mostly comparing it to the prior rich search results that showed instead, as it fills the same area and theoretically is meant to fill the same use case.

I have a fair few articles at this point, mostly in the Minecraft space, that are steadily the number one spot in the Google rankings. Specifically as an example, I’m using my article on what Minecraft mod platform someone should use. Asking ChatGPT the leading question of that article gives me a fairly interesting result. It gives me a nicely formatted response, with section headers that look like they’ve been put through my Emojify tool. In general, it looks like a summary of my article. This would make sense, I get ~1467 hits a week from the ChatGPT-User bot according to CloudFlare.

What is odd though, is that my website is not listed as a source in most instances. Depending on my specific phrasing, I’m able to get it citing my site at the very bottom of the list. As far as I can tell, it’s using one article (in this case mine) as a structural article to build off of, and then finding individual sources for each point that it extracts. This allows it to show a variety of sources and show more details per point. However, in doing so, it’s managed to instead mostly cite websites that are using these SEO leads to sell their server hosting services, or spammy clickbait sites.

The Google AI Overview is similar, although ironically, it’s using the header image from my article without actually citing my article anywhere.

To clarify, I am not accusing these services of plagiarising my posts. My concerns are that these platforms are creating a negative feedback loop that encourages lower quality sources; by not citing the base sources they use as a structure, and not offboarding users to the actual sites for the information.

What does this mean for blogs?

These systems in their current state, are inherently taking viewers away from blogs. While the information from articles might still come across, it destroys a few core parts of blogging. You can’t build an audience. You can’t interconnect articles to help people expand their knowledge. You can’t demonstrate that you truly are an authority on the subject, to build trust with your audience.

While the weekly ChatGPT-User hits for my most popular articles have been skyrocketing, my actual search hits and page views (as per Google Analytics) have been slowly going down. On the surface, it looks like my site is getting significantly fewer hits. The articles are still high up in Google search results, and there’s still verifiable interest in the topics they cover. The site hasn’t lost popularity, there are just fewer clicks in search, and fewer page views.

If you add these AI search engine statistics from Cloudflare on top of my Google Analytics results though, you can extrapolate a growth curve that follows on from what I was seeing years ago. My content appears to be growing at the same rate it always was; my writing is still fulfilling its intended purpose of informing people.

The only difference is, they’re not visiting my site to read it.