Evaluating third-party libraries

Question

At our company, we frequently use third-party libraries and frameworks in our products. Often, we are faced with the task of evaluating one or more libraries that full fill similar requirements (in a specific case we had to choose between some encryption libraries). We often do a simple prototype to try to figure out, if the library somehow works for us. However, often the prototypes get really complex and when we are at a point where we figure out, that the library is not suited for our purposes, lots of time was spend, that we could have used to write the desired functionality ourselves.

My question would be, do you have or practice any guidelines when it comes to evaluating third-party libraries? More specifically, do you have something like a maximum time that will be spend on prototypes and do you have general requirements for libraries you use (license issues, support, documentation, ...)?