You can use the UrlReferrer to check if the call is comming from your site. One very simple working example:
if( !context.Request.UrlReferrer.Contains("site.com/")) )
{
context.Response.End();
return;
}
In some rare cases that users overwrite the Referrer, this fails.