Strange RoR problem; googlebot gets error, human sees page

Question

I maintain a Ruby-on-Rails website (actually running JRuby 1.5.5, Rails version 2.3.10) and am seeing something fairly strange. A certain controller action is causing an exception in the logs, when accessed by a search engine bot:

ActionView::TemplateError (can't convert nil into String) on line #14 of app/vie
ws/scenarios/show_send_message.rhtml:
11:   <% if ! is_logged_in? %>
12:     <p>Your email (optional, used to contact you if necessary):<br /><%= tex
t_field_tag 'user_email', @user_email || '', :size => 50 %>
13:     <% if ! is_human? %>
14:       <%= show_simple_captcha %>
15:     <% end %>
16:   <% end %>
17:   <p><%= submit_tag 'Send' %>

... However, when I view the page as a human, it works fine. "show_simple_captcha" is a method from a plugin, and it can't return nil.

Does anyone have any idea why RoR would respond differently to a bot than it would to a real browser? I'm not so much looking for the answer to the problem as I am a way to proceed with debugging, but I'll gladly accept the former.

Answer 1

When I tried to view the page, I got 500 too. There's a big chance something is wrong with your code. For example, you expect session or cookie variable to be set, when it's not.

Answer 2

It is hard to tell from the above error message and code snippet, but is it possible that this action is making the assumption that you are logged in to the site?

My first thought is that the application has some type of a state (cookie, session, etc) and that when you are visiting the website you already have established your state with the app and therefore you do not experience any problems. The Google Bot on the other hand, may just be jumping right to this page and will lack any state your site might assume was already setup.

Another thing that pops to my mind, is the captcha control could be making some rendering decisions based on user agent. Perhaps the user agent the Google Bot is using isn't liked by your captcha control. Try getting one of those Firefox plugins that lets you change your user agent, and set it to the value you see in your logs. Hit your website and see what happens.

Answer 3

If you can emulate a bot, start debugging the plugin. My guess is that it's a bug with the plugin itself and not your app.

Answer 4

I too am facing the same problem. On investigating the plugin, I could understand that while rendering the page on which there's captcha, the 'simple_captcha_controller' is called for rendering the captcha-image (look at the 'src' attribute of the captcha image). In this url, the param 'simple_captcha_key' is also passed. The key is calculated and is passed as a param to simple_captcha_controller'. The key-value pair is stored in the table 'simple_captcha_date'. So, when a human is viewing the form, the key is pre-calculated(and has a value) and sent to the 'simple_captcha_controller' and hence we don't see the problem. However, when a bot is 'viewing', the mechanism of which I don't quite understand, the key passed to the 'simple_captcha_controller' doesn't have a value in the 'simple_captcha_data' table and returns nil. ('SimpleCaptchaData.get_data(key).value' in simple_captcha_config.rb returns nil) The error can be reproduced if arbitrary keys are passed.