In order to do that, you need to go into your target's "Build Settings" pane, find the "Other Linker Flags" setting, open the setting details by clicking on the arrow on the left side, and put different values in the "Debug" and "Release" lines.
The syntax for the option you want is -l<library name without "lib" prefix or extension>
, so for instance:
-lboost_serialization-mt
in the Release setting will link with libboost_serialization-mt.dylib-lboost_serialization-mtd
in the Debug setting will link with libboost_serialization-mtd.dylib
You may need to also tell it where your libraries are located, using -L</path/to/your/boost/libs>
in the same setting, e.g. -L/opt/local/lib/
You probably also need to remove boost libraries from the "Link binary With Libraries" phase altogether.
You can also bypass the GUI for this by using .xcconfig settings files. See This question for details.