Your approach is interesting, but it is not necessary for you to devise a working system because it has already been done, and luckily, you are only one step away from the solution !
Using CMake, it is easy to automate the building and linking of external source code, using the ExternalProject module.
See http://www.kitware.com/media/html/BuildingExternalProjectsWithCMake2.8.html for useful information.
This approach has several advantages:
- you do not have to include the library's source code in your repository
- you can point to the specific version/git tag of the library that you know works with your software OR the latest release if you are certain it will not break compatibility
- you do not have to write a complete CMakeLists.txt file to build a possibly complex code base
- you can eventually configure the external project to build as a static library so you will not have to distribute shared libraries
- you can even completely bypass this if not necessary, by trying to detect a working version of the library on your system with the usual find_package call, and only fall back to building it as an external project if not found