Your dialog proc is fundamentally modal, since it returns a value. It therefore blocks until the user responds, because it can't return its value until the user gives it one.
To make it modeless, build it to just create itself and return. The buttons all then need to call procs (either global or with some fully qualified name) that will set the user value in some place you're waiting for it, then destroy the dialog properly.
This means that your "return" value must be global, the window ID variable must be global, the handler proc/procs must be global, and you'll need to trigger whatever processing you want the value for in some way that's too application specific for me to guess. It's a fair bit of work, but it's easier than trying to incorporate the threading library.