Actually, I just figured it out right after posting the question. It's quite obvious in retrospect, but the LOCK_UN has to work on the same file handle to work, of course. Like so:
$ irb
>> f = File.open('/tmp/asdf', 'r+')
=> #<File:/tmp/asdf>
>> f.flock(File::LOCK_EX)
=> 0
>> f.flock(File::LOCK_UN)
=> 0
>> f = File.open('/tmp/asdf', 'r+')
=> #<File:/tmp/asdf>
>> f.flock(File::LOCK_EX)
=> 0