An alternative to my original mathematical answer also occurred to me. If you can afford the memory and processing power of drawing all your UI elements twice then you can get good results by using a secondary buffer.
The principle involves having an off-screen graphics buffer (e.g. using PGraphics). It must be exactly the same size as the main display, and have anti-aliasing disabled. Draw all your interactive UI elements (buttons etc.) to this buffer. However, instead of drawing them the normal way, give each one a unique colour which it uses for fill and stroke (don't add any text or images... just solid colours). For example, one button might be entirely red, and another entirely green. Any other RGB value works, as long as each item has a unique colour. Make sure the background has a unique colour too.
The user never sees that buffer, so don't draw it to the screen (unless you're debugging or something). When you want to detect what item the mouse is over, just lookup the mouse position on that off-screen buffer. Get the pixel colour at that location, and match it to the UI element.
After you've done all that, just go ahead and draw everything to the main display as normal.
It's worth noting that you can cut-down the processing time of this approach a lot if your UI elements never (or rarely) move. You only need to redraw the secondary buffer when something appears/disappears, animates, or changes size/position.