I think there will be no measurable performance impact. Even if we consider two testing scenarios: one query on large number of rows vs large number of small batches, the function will be fetched only once from another database, and then stay cached. It happens that I have same assemblies installed on each db, so just in case i just ran such tests on 100M rows. No difference.
You can read more about it here: Appdomain
However, there are some contraindications for using clr functions from other databases. For instance, all the databases depend on the one that have assemblies installed. You can't take it offline, and if it crashes, some processes would derail, too. Also, you can't create computed columns using function from another database. For me that was enough to install the assemblies on each db separately.