You can get the siteVisibility state of a site.
Take a look at the SiteService Wiki page.
Something like this should work:
if (document.properties["st:siteVisibility"] != "PRIVATE"){
<your email action here>
}
Be sure you have selected type is st:site in your rule, otherwise add an extra check on that.