I'd say:
either you declare the observer in all config files but the test one
or you load it conditionally:
config.active_record.observers = your_list if Rails.env.test?
Btw, it seems weird you do not need to spec it too. Anyway, I know you can't sum up your app logic there, just curious.