glibc's dlopen handles are based on link_map. I can iterate all link_map's and see if one has l_addr equal to the one given by dladdr. The address of that link_map is the handle.
dietlibc also uses link_map's as handles.
In the case of uClibc there is another struct - dyn_elf. It contains a pointer to a link_map. The global variable _dl_symbol_tables has a pointer to the head of a linked list of dyn_elf's.
bionic uses its own struct which contains struct link_map as a member (not a pointer).