Ok, turns out you need to install the gems that Jammit uses for each compressor, otherwise it will silently default back to its default compressor.
So, to get it to work with closure, i ran:
gem install closure-compiler --no-rdoc --no-ri
And poof, closure file started compressing down to 84k. That was a frustrating hour...