How to use Chrome’s Googlebot mode to view a website

Unsurprisingly, the technical world of SEO can be complicated. It’s not all as simple as posting blog posts and a web server picking them up to display in a search engine’s results.

Often, a web developer or team of web developers are called in to utilise developer tools that help with the process of auditing websites to determine how well they’re being detected by search engines and what different things are having an impact on SEO.

An emulator for Googlebot is one of those developer tools they might scan your webpage with. We get it though… Googlebot? Emulators? The FirstPage New Zealand team is here to help make this all user friendly.

What is Googlebot?

Googlebot is the web crawling robot used by Google to discover and index new and updated web pages for their index. Googlebot visits web pages, follows links, and adds new pages it finds to the index. When a user searches on Google, the search engine uses this to find relevant pages. Googlebot is the driving force behind web servers detecting your page or article and indexing it for search engines to display.

So, wouldn’t it be cool to be able to take the wheel? Imagine having access to the same data on your web page and being able to use that to optimise your page for Google indexing… Adding this to your list of SEO tools could be a game changer.

The benefits of reviewing websites as Googlebot

There are several benefits to reviewing websites as Googlebot:

1. Allows you to see how Google views your website, which can help identify issues affecting search rankings.
2. Allows you to see which pages on your website are being indexed, and which ones aren’t.
3. Helps identify crawl issues that may be preventing Google from accessing and indexing your website.
4. Helps optimise your website’s content and structure for better search engine visibility.
5. Helps you understand how Google is interpreting your website’s content and ranking your pages.
6. If you’re wanting to run a campaign that targets Mac OS users and not Linux users, or users of specific web browsers, you’ll want to change the user agent with a user agent string to see if you’re doing things correctly.

Can I see what Googlebot sees?

“The Googlebot tool employs a headless version of the Chrome browser to display web pages. While it is possible to adjust certain settings to gain a better understanding of how Googlebot views a website, it is not possible to completely replicate its behaviour.

Though we can never perfectly replicate Googlebot via an emulator, we can get pretty damn close!

How to set up your Googlebot browser

For a user to emulate Googlebot with a Chrome extension in the browser, you’ll need to use a tool such as the “User-Agent Switcher Extension for Chrome”. This extension is a type of software that allows you to change the user agent of your Chrome browser to any fake user agent you choose, including Google bots. This is also called user agent spoofing.

A user-agent is like a name tag and Googlebot has one. The web browser sends this name tag when it makes any http requests of a webpage to identify itself. So, if you change your user agent, you’ll alter network conditions so that your http request identifies itself as Googlebot rather than an average user agent!

1. Download and install Chrome

   Before you can change user agent, you’ll need to install the browser extension from the Chrome Web Store. Head to the web store and search the browser extensions available. You’re looking for the ‘User-Agent Switcher for Chrome’ or something like it.

   Install this user agent switcher extension in your Google Chrome browser. It’s important to note that you need to use the Chrome browser, and browsers like Microsoft Edge or Mozilla won’t work with a browser extension from Chrome. Consider changing your browser default.

2. Utilise extensions with user agent strings in your browser

   Once the extension is installed, you can start to change user agent details and alter network conditions. Click on the extension icon in the Chrome toolbar of the browser (up top). If you have more tools installed from Chrome extensions, it will be listed alongside. Like we mentioned, you can’t see Chrome extensions in different browsers!

   Next, you’ll want to click on the “Add” button to add new user agents. Then, in the “Description” field, enter a name for the user agent (e.g., “Googlebot”). Finally, in the “User Agent” field, you’ll be able to enter a custom user agent string. Right click to copy and paste the Googlebot user agent string: “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot. html)”. Click on the “Add” button to add the user agent string.

   You now have the ability to change user agent when viewing web pages!

3. Configure browser settings to emulate Googlebot user agent

   Now you can select the “Googlebot” user agent from the list of user agents available. You can also add another user agent string to be able to emulate a different browser’s user agent.

   Next, navigate to the website you want to view as Googlebot. The website will now be displayed as if it is being viewed by Googlebot! You can switch back to your normal user agent at any time by selecting it from the list of user agents in the extension of your browser.

4. Start scanning!

   Now you can look through the files and different content you have on your website and determine where you might need to utilise different language or make other technical improvements. You might play around and test changes to a single article first.

Can I view websites as other user agents?

Yep! As mentioned, you can’t use other browsers with Chrome extensions, however you can use strings for a user agent with the user-agent Chrome extension to connect with and emulate these different browsers, operating systems and engine bots on varying devices.

• Google Chrome: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3”
• Mozilla Firefox: “Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0”
• Microsoft Edge: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (K HTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134”
• Apple Safari: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7”
• Googlebot: “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
• Bingbot: “Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”
• Yahoo! Slurp: “Mozilla/5.0 (compatible; Yahoo! Slurp https://help.yahoo.com/kb/search-for-desktop/SLN22600.html)”

What more can I do?

Being able to utilise a custom user agent is an extremely useful SEO tool. A custom user agent doesn’t only help you emulate the way browsers scan your websites, but also helps emulate the different devices that they do this on. These devices range from the IPhone to Windows laptop devices.

Get in touch!

If the idea of changing a user agent makes you anxious, we don’t blame you. All this technical talk can be intimidating. You’re not alone if you want to skip the trial and test phase of all of this and get straight to the results of these tools.

Connect with the experts at FirstPage New Zealand today and we’ll take the concern, worry and challenge off your shoulders, instead placing results in the palm of your hand!