In direct/uncached lookup, Informatica fires a select query on the database for each and every record. So if a million records are coming from source, it will hit the database a million times. This takes time on database end as well as to move the data over the network. Moving small chunks of data repeatedly over the network is costlier than sending it as a whole.
But in a cached lookup, Informatica fetches the whole set of records once and caches it in its native server. Also, the cache is sorted and indexed based on your condition columns. Now, when a lookup needs to performed, it just searches (there are very efficient searching algorithms available like binary search) the input fields in the cache, which is a lot faster.