How to guess next instruction to execute in assembly with dynamic memory

Question

A friend of me challenged me to hack a small program he coded. Basically, it is an exe file that displays an image but in order to do so you need a key file with a password.

I started to reverse engineer it with ollydgb and I was able to figure out that a file with name key.txt needs to be present and contain the password. Another thing I realized is that my friend uses the password to calculate a memory adress and call it. So, if you have the wrong password, the application will crash since it will jump to a random address, probably causing a violation.

He basically stores de password in EBX. He puts a fixed value EAX. Then he does ADD EAX, EBX and finally CALL EAX.

So, knowing all this, if I know which address be executed next, I could just substract the address from the fixed value stored in EAX and I will get the HEX value corresponding to the password.

My problem is, how in the world can I know which should be the next address to be executed? I am fairly new to cracking...

I try to point to the next address after the CALL but it does not work. I also checked the libraries it is using and I certainly see opengl32 but I am not sure if I have to jump somehow to that library.

My question is, how can I figure out which is the next address to be executed?

Answer 1

One of the attributes of the X86 instruction set is that despite being variable length (instructions are encoded using a variable number of bytes), instruction sequences will converge rather quickly. This means if you randomly pick a location as the start of a function and start disassembling from that point forward you won't get too screwed up. A few of the instructions you disassemble will be wrong, but eventually (and rather quickly) you will end up with real instructions.

That means you can do things like pick random addresses in the executable, and start looking for common patterns, like the prologue and epilogue of a function. Those can give you hints as to where the code is.

That, however, assumes that you friend isn't hand writing the assembly or otherwise obfuscating the code.

However, no matter what he does, he will have some calls to methods in the import address table, or system call instructions. These will either run the code to draw the image, or will modify the page table of the process so that obfuscated code can be deobfuscated. Those patterns would be more reliable. So a good place to start is by dumping the iat (import address table), and looking for calls to the interesting functions. If that doesn't work, try looking for system call patterns. Exactly what to look for depends on the particular OS you are on.

A few things to keep in mind:

1. If your friend didn't turn off ASLR (address space layout randomization), then the code might not always work even if you have the right password.

2. I'm not too familiar with olydbg, but I going to assume it uses recursive descent disassembly. If it does, it may not include your target function in the disassembly because it won't have any statically resolvable calls to it.

To solve 2 you will probably want to write code to do the disassembly your self, decoding the instructions. The best place to start would be to scan through the exe looking for indirect call instructions. After that try system calls.

Update

One more thing: It is possible for there to not be system calls in the image if the exe is configured to make the exe section writable or the data section executable (or to turn off the nxbit). Its a good idea to take a look at the section table in the image and see if there is anything unusual as a sanity check before you start.

Answer 2

Unless the program is tiny (and by that I mean just a few KB of code), you can't figure it out easily and need to perform some code analysis, make hypotheses and confirm or disprove them by trial and, inevitably, error.

The first thing that comes to my mind is finding functions that are present in the code but there are no call/jump instructions transferring control to them. You can narrow down the search by first identifying those functions that will have to be called in order for the picture to be drawn (the ones that call the relevant OpenGL/DirectX/GDI functions). And then you can work backward from there.

It may also be useful to see if there are any function pointers/addresses in the data section of the executable, the reason being that the compiler may remove functions that aren't referenced by any other functions or pointers and so for the function of interest to still be present in the compiled executable, there must be some references to it somewhere (unless, of course, the code is compiled with all optimizations off).

Either way, I'm afraid, you're going to have to plow through a lot of code.

Another thing to look at is the the executable itself and its sections (e.g. their names, sizes, locations or contents). That might give you some clues.