This isn't a ctypes issue.
You can't alert from that context. alert is a method on window. So you can do one of two things:
Get the most recent window and run alert there.
Components.utils.import('resource://gre/modules/Services.jsm');
Services.wm.getMostRecentWindow(null).alert('hello world');
Use prompts service: example at mdn
Components.utils.import('resource://gre/modules/Services.jsm');
Services.prompts.alert(null, 'Hello World TITLE', 'hello world message');
Also if you are using addon sdk you dont have access to Components
so you can't do Components.utils.import
you will have to put at top of your main.js this const {Cu} = require('chrome');
then you can do Cu.import('blah')