If I understand your goal correctly (correct me if I'm wrong)
Your toggle classes are just going to arbitrarily remove the class and then re-add it. This is not necessary. You shouldn't need to be doing any JavaScript with the bootstrap radio buttons...only modify the CSS. Bootstraps handlers will handle adding and remove the active class. You, then, just need to define what the base color and active colors are in CSS:
.btn-primary { /* color when not active */ }
.btn-primary.active { /* color when radio is active */ }
When I have custom radio buttons that are only in a specific location and are different from the defaults, then I create a custom class that has different colors, e.g.
.btn.btn-radio { background-color:#FFF;border-color:#CCC;color:#333; }
.btn.btn-radio:hover { background-color:#EBEBEB;border-color:#ADADAD;color:#333; }
.btn.btn-radio.active { background-color:#9BB4C9;border-color:#285E8E;color:#000; }
.btn.btn-radio .badge { background-color:#333;color:#FFF; }