Assuming you are not making extensive use of large mmaps or the long long type the performance penalty will be a few percent at most, likely nothing significant at all. The 64 bit version will also likely be larger which could be a disadvantage anyway when considering the likelihood of causing memory paging on low resource systems. A few more registers will be available but in most programs this still has little impact on speed.
The instruction set will not be reduced as such, it will simply not use 64 bit mode of the CPU and thus not use the 64 bit extended versions of instructions.
Look here for more information on the performance implications: http://en.wikipedia.org/wiki/64-bit_computing#32-bit_vs_64-bit