Does project architecture have a strong relation to used libraries?

Question

Programmers, when it comes to talk about popular patterns in enterprise applications, preach that you should code against interfaces to remove strong relations between components; doing this will aid in changing concrete types while minimizing changes in other areas of the application. The underlying assumption is that interfaces are all about abstractions. When it comes to real life it is almost impossible. Lets use an example:

Lets say you currently use an ORM framework in your application, which you abstracted by applying the repository pattern. Now, can you really change ORM framework seamlessly?

public interface IRepository<T>
{        
    T GetById(int id);      
    void Add(T entity);
    void Remove(T entity);
    void Update(T entity);
    IQueryable<T> Query(Expression<Func<T, bool>> filter);                                
} 

Theoretically, this implementation can be applied to the rest of the application, but many questions remain about the actual application of this pattern:

1. How does this pattern shield me from the internal process of libraries?
2. How does the abstraction deal with the variation in exceptions thrown by each library?
3. If I have selected an ORM which provides a caching mechanism internally and I later decide to switch to an alternative ORM with no such support, how does the pattern handle this?

So, from this, I believe that the project architecture is strongly influenced on the libraries that are used. Is there anything wrong with this? Am I assuming too much from this pattern?