All portal properties can not be overridden with hook feature.
You can find out supported properties those can be overriden in dtd file :
liferay-portal-src-6.2.0-ce-ga1\definitions\liferay-hook_6_2_0.dtd
To achieve captcha on message board edit screen, you have to provide
captcha.check.portlet.message_boards.edit_message=true
in portal-ext.properties file. You have to keep portal-ext.properties file parallel to tomcat folder.
I have verified this, its working fine.
Regards,