SWT uses plugin fragments to control this - there is a 32-bit fragment and a 64-bit fragment (as well as fragments for Mac, Linux, etc). Make sure your launch configuration is set to use the proper fragment depending on which JVM you're running it on.
As a note, just including "both" plugin fragments will not work, as then they will both attempt to load and cause problems.
Also, this related question may be useful.
Another idea would be to dive into the loadLibrary
function - from what I remember of how that works, it looks for a cached version of the SWT dll from a temp location somewhere, and so it might be using a cached 64-bit dll instead of properly unpacking the 32-bit one from the jar. In that case, deleting the cached one may fix the issue.
The easiest solution, though, may be just to have two separate instances of Eclipse: one setup to run 32-bit, one setup to run 64-bit. This way you don't have to worry about toggling back and forth, or futz around with configurations as much.