USB is host-driven. This means the device doesn't send any messages except as replies to messages from the host (Mac/PC). So to elicit a message from it, you will first need to send it a valid message.
I assume from your question that you don't actually have any documentation/specification for the device in question? This means you'll need to obtain that information, either by asking the manufacturer for it, or via reverse engineering it.
The easiest way to reverse engineer it is to listen in on the messages generated by an existing driver, e.g. some Windows software that does something similar to what you want to do. Then you need to listen in on the data. In the past, I've had success running the Windows software in a VM on Linux and passing through the USB device in question to the VM, and using Linux' USB debugging features to log the output. You should be able to do the same on OSX using the "USB Prober" application with the "IOUSBFamily Log Release" available from Apple's developer download area.