cross-compile of Python's greenlet and gevent on Linux x86_64 for PowerPC

StackOverflow https://stackoverflow.com/questions/11587635

Question

On my Linux x86_64 host, I am trying to cross-compile some additional Python modules for my PowerPC target, specifically, greenlet, gevent, and gevent-websockets. Currently, I am stuck just trying to cross-build the greenlet module.

Using info from this site:

http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html

I was able to cross-compile Python 2.7.2 using this setup for my build environment

# Undo variables for cross-compile environment
unset ROOT
unset SDKDIR
unset KLIBDIR
unset NFSDIR
unset CONFIG
unset CONFIGURED
unset ARCH
unset OS
unset TOOLCHAIN_BASE
unset TOOLCHAIN_BIN
unset CROSS_COMPILE
unset c
unset KERNEL_DIR
unset AS
unset LD
unset CC
unset AR
unset STRIP
unset SSTRIP
unset OBJCOPY
unset OBJDUMP
unset MAKE
unset CFLAGS

# Set cross-compile variables:
export TOOLCHAIN=/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin/powerpc-none-linux-gnuspe-
export CC=${TOOLCHAIN}gcc
export CXX=${TOOLCHAIN}g++
export AR=${TOOLCHAIN}ar
export RANLIB=${TOOLCHAIN}ranlib
export BLDSHARED="${TOOLCHAIN}gcc -shared"
export LDSHARED="${TOOLCHAIN}gcc -shared"
export RFS="../../ltib/rootfs"
export CFLAGS="-save-temps -Wall -I${RFS}/usr/include -I${RFS}/include/python2.7 -L${RFS}/usr/lib -L${RFS}/lib"
export LDFLAGS="-I${RFS}/usr/include -I${RFS}/include/python2.7 -L${RFS}/usr/lib -L${RFS}/lib"
export CROSS_COMPILE=ppc-linux
export CROSS_COMPILE_TARGET=yes
export HOSTARCH=ppc-linux
export BUILDARCH=x86_64-linux-gnu

Configuring my environment with the above script and then trying to build the greenlet module yields:

$ python ./setup.py build
running build
running build_ext
building 'greenlet' extension
creating build
creating build/temp.linux-x86_64-2.7
/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin/powerpc-none-linux-gnuspe-gcc -I../../../ltib/rootfs/usr/include -L../../../ltib/rootfs/usr/lib -L../../../ltib/rootfs/lib -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-x86_64-2.7/greenlet.o
In file included from /usr/include/python2.7/Python.h:58,
                 from greenlet.h:8,
                 from greenlet.c:5:
/usr/include/python2.7/pyport.h:849:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
error: command '/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin/powerpc-none-linux-gnuspe-gcc' failed with exit status 1

Why is setup.py pulling from /usr/include/python2.7 on my host system? I can't find that dir on my target. How can I create it for my target?

Any suggestions?

Thanks!

Trevor

UPDATE #1:

My relative references to my host's copy of the target's rootfs were incorrect. Correcting it and rerunning yields:

$ python ./setup.py build
running build
running build_ext
building 'greenlet' extension
creating build
creating build/temp.linux-x86_64-2.7
/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin/powerpc-none-linux-gnuspe-gcc -save-temps -Wall -I../../ltib/rootfs/usr/include -I../../ltib/rootfs/include/python2.7 -L../../ltib/rootfs/usr/lib -L../../ltib/rootfs/lib -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-x86_64-2.7/greenlet.o
greenlet.s: Assembler messages:
greenlet.s:832: Error: syntax error; found `(' but expected `,'
greenlet.s:832: Error: junk at end of line: `(31),1'
error: command '/opt/freescale/usr/local/gcc-4.3.74-eglibc-2.8.74-dp-2/powerpc-none-linux-gnuspe/bin/powerpc-none-linux-gnuspe-gcc' failed with exit status 1

At least it is finding more of my target's include libraries, but now I am really stumped! :(

Any more suggestions?

Thanks!

UPDATE #2:

By adding the -save-temps flag to the compiler (updated error above), I was able to save and examine the intermediate assembler code that was mentioned in the above error message. The broken lines are:

#APP
 # 52 "platform/switch_ppc_linux.h" 1
    mr 8(31), 1
 # 0 "" 2

The MR (move register) op is fairly simple, accepting only 2 args (mr to-reg, from-reg). I don't know how the parenthesis with additional register number were added. FWIW, here is the referenced macro in the above header file:

#define STACK_REFPLUS 1

#ifdef SLP_EVAL

#define STACK_MAGIC 3

/* !!!!WARNING!!!! need to add "r31" in the next line if this header file
 * is meant to be compiled non-dynamically!
 */
#define REGS_TO_SAVE "r13", "r14", "r15", "r16", "r17", "r18", "r19", "r20", \
       "r21", "r22", "r23", "r24", "r25", "r26", "r27", "r28", "r29", \
       "cr2", "cr3", "cr4"
static int
slp_switch(void)
{
    register int *stackref, stsizediff;
    __asm__ volatile ("" : : : REGS_TO_SAVE);
    __asm__ ("mr %0, 1" : "=g" (stackref) : );
    {
        SLP_SAVE_STATE(stackref, stsizediff);
        __asm__ volatile (
            "mr 11, %0\n"
            "add 1, 1, 11\n"
            "add 30, 30, 11\n"
            : /* no outputs */
            : "g" (stsizediff)
            : "11"
            );
        SLP_RESTORE_STATE();
    }
    __asm__ volatile ("" : : : REGS_TO_SAVE);
    return 0;
}

#endif

I am starting to wonder if this is a bug in the compiler, because the macro seems simple enough! Any suggestions? ... Thanks!

Was it helpful?

Solution

Maybe you should have asked a new question, because there really are (at least) two completely independent problems here. But, looking at your second problem:

__asm__ ("mr %0, 1" : "=g" (stackref) : );

This is wrong. I'll explain why below, but first, the following change will likely fix it:

__asm__ ("mr %0, 1" : "=r" (stackref) : );

You may also need to change the "g" (stsizediff) below to "r" (stsizediff).

So, what's wrong with the existing version? First, look at how stackref is defined:

register int *stackref, stsizediff;

The register is a hint to the compiler, saying you think it might make things faster or better if it allocated a register for stackref instead of using a stack location, not a requirement. If stackref ends up in R12, great; if it ends up 8 bytes into the stack frame, that's fine too. Either is perfectly legal, as long as it doesn't violate any constraints.

So, what constraints are there on stackref? The only one is in that asm block quoted above. You've got "=g" (stackref) as an output operand. The = means it's a write-only constraint, and the g means it must be in a register, memory location, or immediate value.

So the compiler's not doing anything wrong. It allocates stackref 8 bytes off the stack, which matches the constraint (that's a memory location), then it substitutes that value in for the "%0", and you get:

mr 8(31), 1

Nothing wrong with that—until you try to assemble it, and the assembler notices that you're trying to use 8(31) with an opcode that only takes registers. But the problem isn't the compiler, or the assembler, it's the code. You asked it to use stackref as an operand for mr, and didn't force stackref to be a register, so you got what you asked for.

Anyway, changing the "=g" to "=r" changes the constraint from "any register, memory location, or immediate value" to "any general register". That means the compiler has to put stackref in a general register. Or, if it can't for some reason, it will fail and tell you why, instead of generating assembly that won't assemble.

So, why did this work for the original author? Well, he probably got lucky, and stackref got allocated to, say, R12, instead of 8 bytes into the stack frame, so he ended up with mr 12, 1, which assembles just fine.

Or, one more possibility. Looking at the git tree, it looks like the code was developed on Mac OS X, then ported to AIX (by people who are primarily Mac developers) a decade ago, then copied verbatim from AIX to linux (even leaving the description "Port for AIX on PowerPC"), and not touched significantly since. Both OS X and AIX only had gcc 3 back then. So, maybe that's the reason it worked for everyone at the time, and isn't working for you. And maybe just getting an older cross-compiler will solve your problem. But I'd try fixing the code first.

OTHER TIPS

Why is setup.py pulling from /usr/include/python2.7 on my host system?

It's not. The /usr/include/python2.7/pyport.h:849 refers to the source used to build your host Python, which may or may not actually be on your system.

I can't find that dir on my target. How can I create it for my target?

I'm not sure you want to. I think you're off on a red herring.

Here's the key to your problem:

$ python ./setup.py build

You're using your host's native Python to build an extension, and you're not doing anything to tell it you want the extension cross-compiled. So, as far as it knows, you're trying to build greenlet for it, not for the other Python. Hence things like this:

creating build/temp.linux-x86_64-2.7

But of course you're giving it the ARM cross-compiler, which isn't going to be able to compile extensions for your x86_64 host python, hence this:

/usr/include/python2.7/pyport.h:849:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

Your host Python was built with LONG_BIT set for a 64-bit LP system, but it's trying to build code with a compiler for a 32-bit system.

The blog post http://kynesim.blogspot.co.uk/2012/06/cross-compiling-python-for-arm-with.html (linked from the one you cited) shows how to build third-party C extension modules. As you can see, it's not entirely trivial, and may need a bit of experimentation to get it to work, but it looks like it's doable.

python ./setup.py build

We are trying to cross build an extension, but system uses python-config and wrong pyconfig.h. See the difference between x86_64 and arm pyconfig. Do not try to use i686 container to fix LONG_BIT issue, this is the wrong way, you will receive more complex issues later.

There are several python extensions that creates virtual env for python-config. But there is one 100% reliable way without any extension.

  1. Create amd64 container and install base system (buildah or docker).
  2. Install target toolchain into this container (crossdev).
  3. Install all required software except python extensions inside target root folder using target toolchain.
  4. Add patch for container's python that forces python-config to return target include and library pathes and rebuild container's python.
  5. Install all required python extensions using target toolchain into target root folder.
  6. Extract target root folder from container and remove it.
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top