I assume that you used CreateRemoteThread with a start address set to LoadLibrary, and that you start a thread in the DllMain of the injected DLL.
First, in DllMain DLL_PROCESS_ATTACH save in a global variable the HMODULE of the DLL.
Second, pass this HMODULE to FreeLibraryAndExitThread when you want your thread to exit and unload the Dll.
Beware! you must NOT have "living code" left behind you, that is, no callback address passed to whatever API, if the callback is trigered after the unload, that will be immediate crash (or worse).