This is a partial answer because I don't know precisely how to solve your problem, but I can at least tell you where it is and hopefully give you an idea so you can figure out the rest.
Based on the stacktrace, it looks like you're using the CanCan
gem. Apparently, it is intercepting the request before it gets to your action, as evidenced by this portion of the stacktrace:
...
activerecord (3.2.14) lib/active_record/querying.rb:5:in `find'
cancan (1.6.10) lib/cancan/model_adapters/abstract_adapter.rb:20:in `find'<- here
cancan (1.6.10) lib/cancan/controller_resource.rb:116:in `find_resource'
...
Unfortunately, I am not familiar with this library. However, after browsing their wiki page, I was able to find this information regarding conditionally checking authorization.
According to that page, you might be able to get away with something like this:
skip_authorization_check :only => [:add_speaker, :remove_speaker]
I assume doing this will suppress CanCan
's authorization check, thus preventing your error.