Create a function in the page that you put in the iframe. This function will return a reference to the button.
var btn = iframe.contentWindow.getMyRadButton();
btn.click();
where getMyRadButton is something like
function getMyRadButton() {
return $find("<%=RadButton1.ClientID%>");
}