Unfortunately, I think this is impossible. The main issue with what you are trying to do is actually getting the IP once the properties dialog is displayed. There is no way (AFAIK) of affecting the dialog before it is posted, and no way of getting the instruction pointer after. If you could figure that out, you could SubClass the DlgProc and intercept the OkToCommit call to do your validation.
What you could do instead though would be to validate the properties, and if they are not appropriate re-post the dialog immediately using 'actionMan.executeAction 0 "40022" -- Edit: Properties'. The user would see the dialog dissappear, and reappear immediately. Its not perfect, but it's something.