I have used an Octree before for this sort of thing and it seemed to be my best option. Every object can be placed in the corresponding node and tested through the usual Octree protocol. The real benefits come from offsetting the ray cast to the end of a spatial node when the current node is empty. This will surely give you significant gains over brute-forcing it.
The following link should give you a base with Octree's and how it is set up: Introduction to Octrees
While the link is only giving you a base, the use of the Octree is simply for reducing necessary checks, so judging by your question, you seem to already have the knowledge required to perform the actual picking.