I have a strong feeling that VSTGUI does not support 10.8 yet. You should set the target SDK in your Xcode build settings to 10.7 (or maybe 10.6), which probably involves downloading a huge DMG from Apple via Xcode.
Also it probably wouldn't hurt to contact the VSTGUI developers to seek support from them.