Use Google Analytics to Understand if Your Customers are Using Bard & Other AI-Search Tools

AI-Driven Search Traffic in Google Analytics

For AI-driven search tools like chat-gpt.org, Microsoft Bing Chat, Google Bard, Neeva (before shutting down), and Writesonic; we can now see traffic showing up in our Google Analytics - or analytics tool of choice - alongside our other traffic channel groupings:

How a user gets from an AI tool to your website

Use cases for AI chat are growing daily.

Sometimes folks are trying to figure out where to start on their newfound love of baking...

or trying to track down the best bakeries to go to for inspiration:
Either way, links are slowly beginning to trickle into results in platforms like Chat GPT, Bard, and Bing Co-Pilot.

And our analytics team at Seer Interactive figured out a way to see some of that data in Google Analytics.

Read on to see what we found when we looked in aggregate across a handful of clients.

 

7 brands analyzed for ai-driven search traffic

Monthly sessions peaked at 177 sessions in aggregate

While we believe it's worth setting up the tracking to make sure you can understand if and when AI-driven search traffic starts to ramp for your business.

But, from initial findings, we aren't finding any brands at this time where AI-driven traffic is becoming a significant part of their traffic.

Neeva drove 64% of ai-driven search traffic

Across these 7 brands - Neeva drove the majority of the ai-driven search traffic that we're able to identify at this time.

The major drop from May to June 2023 is the deprecation of Neeva.

404s increased across all brands

All of the brands seem to be experiencing a mixture of outdated URLs and hallucinations as seen by the "null" and "(not set)" landing pages across these 7 brands.

'Login' also showed up for one of our SAAS clients.

 

How to track AI-driven search traffic

Copy Seer's 'AI Signal' Looker Dashboards (2-Minutes)

As AI continues to influence search marketing, it's important to understand how your digital presence is affected.

[Pro Tip] Traffic to your site from AI tools is not the only data input you need to understand AI's impact on how users find your business. (jump to additional AI analyses)

But it is a quick and easy start to understanding if there's instances of your brand being linked out from those tools.

This is where our free Looker Dashboard(s) come into play.

The views will allow you to track AI-search traffic landing on your website directly in Google Analytics.

Create a copy of Looker Dashboard for Google Analytics 4 Accounts:

Create a copy of Looker Dashboard for Google Analytics 'Universal Analytics' Accounts:

Dashboards use referrer traffic data to capture trends

What is referral traffic?

Referral traffic comes from another website, social media platforms, or any other assets that link to your site.

Another way to think of referral traffic is that it is traffic that comes to your site from sources other than search engines and paid media.

Tracking varies by analytics tools and websites.

What AI-Tools are driving referral traffic?

Our Analytics Development team surveyed several AI tools and found:

• 3 platforms that suppress referral information on outbound links
• 5 platforms that provide referral information on outbound links
• 1 platform that was deprecated (but previously providing referral info)

Withheld referrer data came in two ways:

1. Same-origin Referrer Policy:

1. • What it is: A policy set by websites to control the referrer information sent when users navigate from one page to another.
   • Effect: When a user clicks a link on a website with this policy, the referrer information (i.e., the originating page's URL) is only sent if the destination page is on the same website. If the user is directed to a different website, no referrer information is sent.
   • Example: If you're on chat.openai.com and click a link to another page on the same site, the referrer information is sent. But if you click a link leading to a different website, no referrer data is shared.

2. rel="noreferrer":

   • What it is: An attribute added to specific links in HTML code.
   • Effect: When a user clicks a link with this attribute, no referrer information is sent to the destination page, regardless of whether it's on the same website or a different one. Additionally, it behaves as if another attribute, 'noopener', is also specified, which means the new page won't have access to the previous page's window object.
   • Example: If a website uses "rel=noreferrer" on a link, and you click it, the destination page won't know which page you came from.

In essence, while both methods prevent the sharing of referrer data in certain situations, the "Same-origin Referrer Policy" is a broader website-wide setting, whereas "rel=noreferrer" is applied to individual links.

Making decisions with this data

We have 3 considerations for you before using this data to make decisions.

1. This is not holistic data - don't rely on raw numbers

As noted above, not all AI-search tools pass referrer information to your analytics tools, so this is not a holistic look of all ai-driven search traffic.

Instead, look at this as a percentage of total traffic and set some baselines. As that percentage rises, create a milestone where you'll start to investigate it more heavily and create strategy around how to handle and harness this traffic.

[Pro Tip] Set-up a AI-Powered Search channel in your analytics account. Jump to setion on how to create a Channel.

2. Run additional AI-impact analyses

Zero-click searches will become more common in AI-driven search, as it removes friction for the user in providing the answer they need without additional action(s) the user needs to take (i.e. a click and often website friction like pop-ups and un-needed supplemental information.)

We've created two analyses that attempt to understand more at the impression level, what will be impact by AI-driven search first:

• What keywords will be most disrupted by chat-driven search

• How to see if you're getting brand impressions in ChatGPT

In the following sections, we'll dive deeper into how AI-driven search traffic is tracked and what it means for your website and business.

404 increases are another signal to watch

We identified some nuances about ai-search driven traffic through our research that can result in spikes in your 404 pages. Here's the quick hitter list:

• Some traffic to ChatGPT traffic will be to non-existent pages 
• Large Language Models (LLMs) may have old URLs in them
• LLMs may also generate URLs from educated guesses.

[Pro Tip] Monitor 404s tab available in Looker dashboards.

404 increases led us to "chat.openai.com" referrer

Recently, we worked with a client to better understand a significant jump in 404 “page not found” errors that occurred on January 29th, 2023. 

What we saw was an increase of traffic coming from chat.openai.com. This must be the day this brand's links started appearing in the chatbot's results.

We also saw that traffic abruptly drop off on February 18th, 2023. If you're reading this post top to bottom, you'll recall from above that "chat.openai.com" is one of the tools that now blocks a referrer from being passed to Google Analytics. Our best guess is that is around the time chat.openai.com implemented their same-origin referrer policy, blocking the referrer string.

Here’s a screenshot of that in Google Analytics:

Non-existent URLs from AI tools were increasing 404s

We discovered two main reasons for non-existent URLs to be showing up in AI chat:

1. Outdated URLs

Outdated URLs in Large Language Model (LLM) training data are a natural consequence of how these algorithms work.

ChatGPT 3 was trained on books, Wikipedia and other web crawls coming from June 2020, while ChatGPT 3.5 and  ChatGPT 4 are only aware of historical phenomena prior to September 2021.

ChatGPT cannot access updates to URLs prior to those time frames.

2. The Extrapolation Effect (AKA “Hallucinations”)

Multiple reports have come out about ChatGPT chat responses that are factually incorrect.

In one example, the British dailymail.com reported multiple requests for articles that never existed. They found out that some article citations from ChatGPT had been made up.

This effect has been called “hallucination”, and stems from the foundational mechanics of AI algorithms: they are great at choosing “which word comes next”. However, this sometimes leads to wrong choices, and wrong answers, including URLs.

Use 404 page increases as an opportunity to delight

No matter the reason for the traffic, it presents an opportunity to capture these users’ attention, delight them, and convert them.

If you are experiencing a rise in 404 “page not found” traffic, we believe there are some ways to turn a “sorry, we don’t have what you want” message into a “you came to the right place!” message.

Below are some ideas for re-capturing this traffic.

Let Your 302 Redirects Live a Little Longer

Since LLMs have crawl data from History Past, links in answers have a chance of being old.

If you were considering removing redirects from recent content migrations, consider leaving them in place for a bit longer to ensure that users will get to the content you want them to see.

Improve your 404 page User Experience

It’s not often that you encounter a 404 page “in the wild”, but while working on this article, we encountered several.

And we can definitely say that a 404 page is like running into a brick wall from a user experience perspective.

Provide Search Functionality on 404 Pages

Make it obvious to the user that they can easily find what they’re looking for.

Want to take it to the next level? Do some of the work for them: extract significant words from the URL by removing stop words and preemptively search your own site, displaying your best content to users looking for more.

Provide other content-finding or product-finding tools

Showcase your Featured Articles, Top Content, Frequently Asked Questions…whatever it takes to help reorient someone who may be lost, and engage them!

Set up AI-search traffic channel in your analytics account (<10 minutes)

It's going to feel good to get ahead on something for once, won't it? Take the 10-minutes. Your boss will thank you later.

Google Analytics 4

1. Open Google Analytics 4
2. Navigate: Admin > Data Settings > Channel Groups
   
3. Enter 'Group Name' - we used "Channels 2023"
4. Enter 'Channel Name' - we used "AI-Driven Search"
5. For conditions use "matches regex" and copy / paste this regex formula:
   
   (.*gpt.*|(.*google.*bard.*|.*bard.*google.*)|.*neeva.*|.*writesonic.*)

6. Save channel

PiwikPRO:

1. Open PiwickPRO
2. Navigate: Settings > Custom channel grouping > Add a custom channel grouping

Adobe Analytics

1. Open Adobe Analytics
2. Navigate: Admin > Report Suite Manager > Marketing Channels > Marketing Channel Manager