You've got two issues here:
1. How to remember to publish the correct settings each time you publish:
One way to deploy such settings is by using web.config transformations in Visual Studio. This is pretty easy to set up and means that you do not have to remember to update the settings each time you publish.
As well as debug and release environments, you can also create transforms for "UAT", "Staging", "Beta" or whatever other configs you might need.
You might find these articles useful: here, here and here.
For example, here is a transform for a Release environment:
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<connectionStrings>
<add name="MyDB"
connectionString="Data Source=ReleaseSQLServer;Initial Catalog=MyReleaseDB;Integrated Security=True"
xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
</connectionStrings>
<system.web>
<compilation xdt:Transform="RemoveAttributes(debug)" />
<customErrors defaultRedirect="GenericError.htm"
mode="RemoteOnly" xdt:Transform="Replace">
<error statusCode="500" redirect="InternalError.htm"/>
</customErrors>
</system.web>
</configuration>
Above, you will see that the transform for Release mode sets the attributes of the MyDB connection string (xdt:Transform="SetAttributes"), removes the debug attribute from the compilation section and replaces the customErrors section with a new version.
It's a simple, yet very powerful technique.
2. How to get your assemblies to pick up the settings in the web.config
If your libraries have been written in the usual way, they should be retrieving their connection strings by simply accessing the [Web]ConfigurationManager.ConnectionStrings property. Like @Bob Horn says, they should then pick up the settings from the host process's config file (in this case the web.config of your web app).
However, sometimes you might find that a library is getting its settings from a .Settings file in the project, in which case things get a little more complicated. You will need to copy the settings section of the app.config in to the web.config. (You can also do this using the transforms technology described above.)
If you have access to the source code of the other assemblies, find the part of the code that retrieves connection strings. If it's not accessing the ConfigurationManager class, then that might explain why it's not picking up the web.config file.