How to extend an in-program function using Detours in c++ without going in an infinite loop?

Question

I am trying to learn to use detours to modify and extend functions in a program. In this case I am trying to modify the InsertDateTime function in Windows Notepad 32 bit.

I am using Winject to inject the dll which I create to modify the function. The DLL is injected correctly and the function gets detoured to the new function that I have specified. However, the new function to which I detour InsertDateTime() is supposed to call the original function InsertDateTime() at the end of the new function called MyInsertDateTime().

However, the problem is, that when the new function gets called and in the end tries to call the old function. This one, of course, also gets redirected/detoured in turn in an infinite loop. So the original function can never be called!

Somehow I guess I need to detach the new function in order to call the old one. But when I do that I get error? This might however be some fundamental misstake. How should I do this properly?

The code can be seen below:

#include <iostream>

#include <windows.h>
#include "detours.h"
#pragma comment(lib, "detours.lib")
//

using namespace std;

static int(__stdcall* InsertDateTime)(int);

int MyInsertDateTime(int x) //Our function
{
MessageBox(NULL, TEXT("Detoured Function Call"), TEXT("Min funktion"), MB_OK);
return InsertDateTime(x); //Return the origional function
}

BOOL APIENTRY DllMain(HANDLE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
    InsertDateTime = (int(__stdcall*)(int))(reinterpret_cast<DWORD>(GetModuleHandleA("notepad.exe")) + 0x978A);
    switch (ul_reason_for_call) //Decide what to do
    {
    case DLL_PROCESS_ATTACH: //On dll attach
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourAttach((PVOID*)(&InsertDateTime), (PVOID)MyInsertDateTime);
        DetourTransactionCommit();
    break;
    case DLL_THREAD_ATTACH: //On thread attach
        break;
    case DLL_THREAD_DETACH: //On thread detach
        break;
    case DLL_PROCESS_DETACH: //on process detach
        DetourDetach((PVOID*)(reinterpret_cast<DWORD>(GetModuleHandleA("notepad.exe")) + 0x978A), InsertDateTime);
    break;
    }
    return TRUE;
}

Finally the modified MyInsertDateTime() where I try to detach the detour. Perhaps I am doing something wrong when detaching?

int MyInsertDateTime(int x) //Our function
{
//Messagebox
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach((PVOID*)(reinterpret_cast<DWORD>(GetModuleHandleA("notepad.exe")) + 0x978A), InsertDateTime);
DetourTransactionCommit();
MessageBox(NULL, TEXT("InsertDateTime Just Got Called"), TEXT("Min funktion"), MB_OK);
return InsertDateTime(x); //Return the origional function
}

Answer 1

You should only remove the Detour in DllMain when the DLL is unloading.

DetourAttach() is giving you the function pointer to the original function.

See here:

http://www.codeproject.com/Articles/30140/API-Hooking-with-MS-Detours

DetourAttach(&(PVOID&)pSend, MySend);

int WINAPI MySend(SOCKET s, const char* buf, int len, int flags)
{
  fopen_s(&pSendLogFile, "C:\\SendLog.txt", "a+");
  fprintf(pSendLogFile, "%s\n", buf);
  fclose(pSendLogFile);
  return pSend(s, buf, len, flags); // Calling original function obtained via the DetourAttach call, this will NOT cause MySend to be called again.
}

You might have an issue because:

InsertDateTime = (int(__stdcall*)(int))(reinterpret_cast<DWORD>(GetModuleHandleA("notepad.exe")) + 0x978A);

Might be the "wrong" address causing the jmp to be in the wrong place.