The root cause is related to exception handling mode in visual c++ compiler. I use the lua function (such as luaL_error) with extern "C" modifier to prevent compiler from name-mangling. And the default exception handling mode is /EHsc which assume extern "C" function don't throw exception. So, the exception can't be catched. The solution is change /EHsc to /EHs.
For more information please refer to http://msdn.microsoft.com/en-us/library/1deeycx5.aspx.