I struggled with the same question myself, here is how I was able to solve it:
MyModel.update_all([%(data = data || hstore(?,?)), 'a', 'new_value']))
The core fix for this was wrapping the update_all action in a [] and %(). I am still struggling to figure out how the %() defines the SET in the Postgre SQL so if anyone has an explanation that would be uber-helpful.
In my case I was actually removing a key too (really I wanted to update the key name but keep the value). So if anyone has that issue the code looks like:
MyModel.update_all([%(data = delete("data",?)), 'a'])
I was hoping to do both actions in the same call but that was creating a really odd command in the SQL where the second action was added as part of the WHERE clause not the SET. Still a bit of black magic for me but hopefully this helps...