I just found a workaround which seems to have helped:
I put my library in /usr/lib/ and ran ldconfig --> everything works
I remove my library from /usr/lib --> everything keeps working.
Maybe ldconfig tries to be too smart; and didn't notice any changes in /usr/lib/test/ since it wasn't tracking this folder before. By changing something in /usr/lib/ I might have forced him to do a real refresh?