set the Response.Result. If the result is not null it will not execute the action. the exact syntax is escaping me right now, but it's as simple as
if(actionContext.ModelState.IsValid == false)
{
var response = actionContext.Request.CreateErrorResponse(...);
actionContext.Response = response;
}