How to noindex AJAX-loaded pages without a head tag?

Question

On a client's website there are a series of tutor popups that are located on separate pages.

http://launcheducation.com/

The thing is that they do not need to be noindexed, they're just popups on the main page. If you load the links directly they aren't styled correctly. They don't have proper html or head tags, because they are merely loaded through AJAX.

Because they don't have a head tag, I can't add meta tags to it. I'm an SEO n00b, I only know the basics. But I certainly don't think I can add meta tags without a head tag. So I added them to robots.txt instead.

I thought that would be the end of it, but my client is using the SEOmoz Toolbar which says that they are still being followed. I don't think its an issue at all, I think its just that the toolbar doesn't check robots.txt, but he keeps on repeatedly bringing it up.

I don't know what to do, so I want to clarify if putting the links in robots.txt will work. Here's my robots.txt:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-content/themes/launcheducation/tutor-ajax/*?
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669

Yes, I did start off with just "Disallow: /wp-content/themes/launcheducation/tutor-ajax/*?" When the plugin still didn't acknowledge the change I added them all. The robots.txt is also clearly loadable in the root directory:

http://launcheducation.com/robots.txt

Any help would be appreciated, even if its only to affirm that my statement is correct and its just the plugin that doesn't check robots.txt. If I am wrong, of course, I want to know how I can fix this.

Thanks in advance.

Answer 1

It looks like you're on the right track. I'm not sure about the SEOMoz tool, but what you should use is the Google Webmaster Tool. Within the tool, there's a feature that lets you crawl any given URL using Googlebot. That will tell you definitively whether or not your AJAX partials are being properly blocked with robots.txt

If there are any edge cases where a URL pattern matches something you want crawlable, you can always send a NOINDEX command via the http header since you don't have the ability to add the meta tag. The X-Robots-Tag http header is intended for this purpose, mostly where PDFs and Images aren't able to render HTML ... it will work in your case.

The full specs are here: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

As an aside, 3rd party tools like SEOMoz are great, but nothing beats the tools that the engines themselves provide. :)