The ASP Custom Validator will fire only when the user attempts to submit the webform via a button click, etc. If you want to run the validation during a client-side event, you'll have to use Javascript, not an ASP server control.
See this JS Fiddle Demo for an example of what you could do. The code is below.
HTML
<input type="text" class="password" id="mytextbox"></input><span class="errormsg">Password length invalid</span>
JQuery / Javascript
$(function(){
// This is to add the "On Change" event to your textbox.
$('#mytextbox').change(function(){
if(!validateFunction($(this))){
$(this).next('.errormsg').fadeIn({duration: 400});
} else {
$(this).next('.errormsg').fadeOut({duration: 100});
}
});
})
// This is the function that validates your checkbox. It returns true or false.
function validateFunction(obj){
if(obj.hasClass('password')){
var txt = obj.val();
return (txt.length > 5 && txt.length <= 30);
}
};