How long is your project and how many people are you ? To me what you try to do is quite some work. Do you have good knowledge about low level behavior ?
To answer your question, the main problem here is that the cog simulator is not maintained in the pharo vm fork. This is because no one in the pharo crew use the simulator. We only use external debugging from gdb. In fact the pharo folks work mostly on VM plugins, the core of the VM is mainly maintained and developed by Eliot Miranda which works on Squeak. Therefore we report to him when there's a bug in the VM core.
For your project you would have to split it in at least 2 steps:
step 1: make the object table work with the stack VM
step 2: make the JIT work with your object table
Note that for step 2 I would recommend not to change the way an object access its header, therefore having a VW-like object table where you have the fixed size header on the one in the the object table, and the fields of the objects (and maybe header extensions) in the heap.
So use the StackVMSimulator and build the StackVM first. When everything will work (including context), you can think about hacking the JIT. Recently Guillermo Polito ported the Stack VM to the build process (see PharoSVMBuilder instead of PharoVMBuilder), a guy reported problems with this builder but you could hack it a bit to make it work.
Now to make the simulator work on Pharo 2.0 (which is the Pharo version of the generator image you have), you have to open the monticello browser and merge from Eliot's branch the Cog package (repo MCHttpRepository location: 'http: //source. squeak. org/VMMaker'), but not the latest Cog, the one at around the same date as the current VMMaker package of pharo-vm because the latest Cog and VMMaker of Eliot's branch are not stable.
The alternative being to start from Eliot's build image and merge things from the pharo branch. Here are infos about how to build the squeak development image (http://www.mirandabanda.org/cogblog/build-image/).
Then Eliot gave me this script once:
| cos |
cos := CogVMSimulator newWithOptions: #(Cogit SistaStackToRegisterMappingCogit).
cos desiredNumStackPages: 8.
cos openOn: 'my/favourite.image'.
cos openAsMorph; toggleTranscript; halt; run
You don't need the SistaStackToRegisterMappingCogit option. I guess some similar script with the StackVMSimulator should work.
Lastly there are some documentation about the simulator but it is only for the CogSimulator (these documentations expects you already knows how the StackSimulator works, and just give you hints about how to use it with the JIT): http://www.mirandabanda.org/cogblog/2008/12/12/simulate-out-of-the-bochs/ and in one of the video named "Cog VM (part x)", x being from 1 to 6, Eliot shows how he uses the simulator to disassemble x86, print the stack and inspect the heap.
Another tip, ask your questions on the pharo mailing list (pharo users or pharo dev), because here no one may notice your question (fortunately someone pointed me out your question this time).
And tell on the pharo mailing list if you managed to run the simulator in Pharo 2.0, some people (as me) are very interested in it. I was planning to do it at some point.
Good luck ! Nice project anyway.