How can Javascript code can be executed in order to get useragent?

Question

I have an email marketing website with user-tracking capability and this is normally what I do. I ask to my customers to add this code to their websites in order to track behavior of their customers.

var _ssprt=('https:'==document.location.protocol?'https':'http');
var ig = navigator.userAgent.toLowerCase().indexOf('googlebot') > 0;
document.write('<img height="1" width="1" src="'+_ssprt+'://www.myurl.com/system/sitecode.php?t='+document.title+'&adres='+document.location.href+'&ua='+ig+'" hspace="0" />');

Normally if one enters to the website through a browser, I can detect its user-agent easily.

However, if this is GoogleBot , since it executes the website as source code, it wouldn't send any datas to my main url. It cannot execute the php as well. Thus, I cannot see if any googlebots enter to the website.

I use this code in order to get the user-agent

var ig = navigator.userAgent.toLowerCase().indexOf('googlebot') > 0;

I thought I can redirect my sitecode.php to js via htaccess. So it will behave as sitecode.js and I will include it with script src code

I am wondering if I had done this, googlebot would have executed that JS.

I am trying to do this with this piece of code but I couldn't succeed. Also, I am not sure if Google would execute this and send me the user-agent data.

RewriteEngine on
RewriteRule ^sitecode.js$ sitecode.php [QSA,L]

Answer 1

I found the answer on this website which answers to my question.

http://searchengineland.com/is-googlebot-skewing-google-analytics-data-22313

Postscript: Google Analytics posted a response in the comments:

“The official Google bot does not execute Google Analytics JavaScript. We’re not sure what it is exactly, it could be anyone’s bot, some intern’s experiment, or other such traffic.” I agree with this comment in that the official Googlebot reads JavaScript but does not execute it. Besides, it does not store and send cookies, which means that Paves/Visit would be exactly 1 and time on site exactly 0. Lastly, If the officiall Googlebot did execute JavaScript, we would have seen massive ammounts of visits.

It is also important to note that although we used Google Analytics as an example, we mean all JavaScript based solutions, including Omniture, Yahoo Web Analytics, WebTrends and others.

Please note that this issue requires additional investigation both in regards to Google Analytics and to how Google Search uses the Googlebot.

Answer 2

My interpetation is that you do want to detect GoogleBot hits on your page? Or at least detect them so you can filter them out in your own code?

GoogleBot can interpet some JavaScript, but it does not execute it as a browser. And Google is quiet about what GoogleBot is doing when it comes to interpeting scripts. The same problem exists when the user has disabled JavaScript, then you will not see their visits.

There are ways to make AJAX content crawlable by GoogleBot, but it will also require some server code.

Unfortunatly, the safest way to make sure you track all visitors is to use server side code.

Optionally - and I suggest this with some reservation as I have not tried it myself - you could try adding an img or a hidden link to your php-page on your side and then check the user agent and referer to get the page that the user was visiting - but I'm not entirely sure it works that way and that GoogleBot will send the refered-header. Maybe someone else have tried this out?

Answer 3

navigator.userAgent is available only in a browser environment. Googlebot just does a HTTP fetch, it does not run client side JavaScript. Its like fetching a page with wget or curl - you just download the page content (source/HTML) but not execute the scripts within.

To track googlebot accesses, you'll need to put in some server side solution, but depending on what server side technology your customers use, you'll need to provide snippets to support multiple server side technologies.