After a day of messing around I'm answering my own question.
Turns out if we create a blogger blog (with a custom domain in my case) and add that blogger blog to the list of contributed blog in our Google Plus, rel=publisher of that Google Plus id is automatically added. Once I removed the link from my Google Plus, the code in the header:
<link href='MY_GOOGLE_PLUS_URL' rel='publisher'/>
Dissappears !!