If your binaries are standalone apps, as in they're not plugins or dylibs for other programs, then there's nothing wrong with using 32-bit binaries. You're in good company; if you look at Google's Chrome browser, that also uses 32-bit binaries.
The advantage of 64-bit is that you can address more memory, if required and I think there are a few assembler instructions that are faster with 64-bit instructions, but overall, if it works for you, I don't believe it's an issue.
If you find yourself creating a dylib that must work with both 32 and 64 bit applications, that would be a reason for creating a Universal Binary, to save having to create one of each and work out which you need at runtime; with a Universal Binary OSX will take care of that for you.