Patching out CALLL by replacing with NOPs works in user space but not in kernel space

Question

I have a device driver I want to patch. This device driver calls IOLog and I want to get rid of the logging.

If I replace the CALLL to IOLog with (the corresponding number of) NOPs inside the device driver (kext), the kernel crashes with what looks like a smashed stack ("Backtrace terminated-invalid frame pointer 0").

The same technique however, works fine in user-space (e.g. NOPping NSLogs inside an OS X binary).

What am I missing here?

Answer 1

You didn't explain whether you do cold patching (driver on disk) or hot patching (driver in memory). For in-memory patching, all kinds of issues might exist, such as the driver being executed when you patch it, the CPU(s) having cached portions of the code, etc. See the Intel manual section on self-modifying code.

For on-disk patching, it might be that you have a relocation record for the target address. So when the driver is loaded, the dynamic module loader will fixup the address of IOLog in the code, replacing it with the real address. This will overwrite your nop instructions.