Cross compiling for Linux on Windows (linker output file won't run as executable on linux and has undefined symbols)

StackOverflow https://stackoverflow.com/questions/23234795

Question

I'm trying to set up a cross compile for linux ELF files on Windows using clang and a version of ld which has been compiled to have elf64 support. The clang compile part is fine, it outputs ELF obj files that work when linked on linux. My test case cpp is just main containing a printf statement.

To attempt a link on windows I copied over all of the libraries from my ubuntu install and specified the ones that are needed as linker arguments in the right order (as they appear on the ld invocation performed by g++). The problem is, the output file is slightly different than the one produced on linux and it won't run as an executable. I noticed, for example, that in the version linked on windows there is the undefined symbol "printf" which appears to have become "puts" instead in the linux version.

Anyway I'm wondering if anyone knows what is happening here.

My actual linking arguments are like this:

LFLAGS = --build-id --eh-frame-hdr --oformat elf64-x86-64 -m elf_x86_64 --hash-style=gnu --as-needed -z relro --verbose

LIBDIRS = -LI:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1 -LI:\\Linux\\libs\\lib\\x86_64-linux-gnu -LI:\\Linux\\libs\\lib -LI:\\Linux\\lib\\x86_64-linux-gnu -LI:\\Linux\\lib

LIBS = I:\Linux\libs\lib\x86_64-linux-gnu\\ld-2.17.so I:\\Linux\\lib\\x86_64-linux-gnu\\crt1.o I:\\Linux\\lib\\x86_64-linux-gnu\\crti.o I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\crtbegin.o Test.o I:\Linux\lib\x86_64-linux-gnu\\libstdc++.so.6.0.18 I:\Linux\libs\lib\x86_64-linux-gnu\\libgcc_s.so.1 I:\Linux\gcc\x86_64-linux-gnu\4.8.1\\libgcc.a I:\Linux\libs\lib\x86_64-linux-gnu\\libc.so.6 I:\Linux\lib\x86_64-linux-gnu\\libc_nonshared.a I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\crtend.o I:\\Linux\\lib\\x86_64-linux-gnu\\crtn.o

all:    $(MAIN)
    ./ld-new.exe $(LFLAGS) $(LIBDIRS) $(LIBS) -o Test

$(MAIN):
    clang $(CFLAGS) $(SYSINCLUDES) -o $(OBJS) -x c++ $(SRCS)

Linking on linux:

==================================================
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o succeeded
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o succeeded
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o succeeded
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o
attempt to open Test.o succeeded
Test.o
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.so succeeded
-lstdc++ (/usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.so)
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libm.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libm.a failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libm.so succeeded
-lm (/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libm.so)
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so succeeded
-lgcc_s (/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so)
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.a succeeded
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libc.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libc.a failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libc.so succeeded
opened script file /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libc.so
opened script file /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libc.so
attempt to open /lib/x86_64-linux-gnu/libc.so.6 succeeded
/lib/x86_64-linux-gnu/libc.so.6
attempt to open /usr/lib/x86_64-linux-gnu/libc_nonshared.a succeeded
(/usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
attempt to open /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 succeeded
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so succeeded
-lgcc_s (/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so)
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.a succeeded
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o succeeded
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o
attempt to open /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o succeeded
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o
ld-linux-x86-64.so.2 needed by /lib/x86_64-linux-gnu/libc.so.6
found ld-linux-x86-64.so.2 at /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

nm output on linux: 

k@system:/shared$ nm ./Test
0000000000601040 B __bss_start
0000000000601040 b completed.6992
0000000000601030 D __data_start
0000000000601030 W data_start
0000000000400470 t deregister_tm_clones
00000000004004e0 t __do_global_dtors_aux
0000000000600e18 t __do_global_dtors_aux_fini_array_entry
0000000000601038 D __dso_handle
0000000000600e28 d _DYNAMIC
0000000000601040 D _edata
0000000000601048 B _end
00000000004005e4 T _fini
0000000000400500 t frame_dummy
0000000000600e10 t __frame_dummy_init_array_entry
0000000000400718 r __FRAME_END__
0000000000601000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000004003e0 T _init
0000000000600e18 t __init_array_end
0000000000600e10 t __init_array_start
00000000004005f0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000600e20 d __JCR_END__
0000000000600e20 d __JCR_LIST__
                 w _Jv_RegisterClasses
00000000004005e0 T __libc_csu_fini
0000000000400550 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
000000000040052d T main
                 U puts@@GLIBC_2.2.5
00000000004004a0 t register_tm_clones
0000000000400440 T _start
0000000000601040 D __TMC_END__

Linking on Windows:

==================================================
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\ld-2.17.so succeeded
I:\Linux\libs\lib\x86_64-linux-gnu\ld-2.17.so
attempt to open I:\Linux\lib\x86_64-linux-gnu\crt1.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crt1.o
attempt to open I:\Linux\lib\x86_64-linux-gnu\crti.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crti.o
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtbegin.o succeeded
I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtbegin.o
attempt to open Test.o succeeded
Test.o
attempt to open I:\Linux\lib\x86_64-linux-gnu\libstdc++.so.6.0.18 succeeded
I:\Linux\lib\x86_64-linux-gnu\libstdc++.so.6.0.18
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\libm-2.17.so succeeded
I:\Linux\libs\lib\x86_64-linux-gnu\libm-2.17.so
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\libgcc.a succeeded
attempt to open I:\Linux\lib\x86_64-linux-gnu\libc.so succeeded
opened script file I:\Linux\lib\x86_64-linux-gnu\libc.so
attempt to open /lib/x86_64-linux-gnu/libc.so.6 succeeded
/lib/x86_64-linux-gnu/libc.so.6
attempt to open /usr/lib/x86_64-linux-gnu/libc_nonshared.a succeeded
(I:/Linux/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
attempt to open /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 succeeded
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
attempt to open I:\Linux\lib\x86_64-linux-gnu\libc_nonshared.a succeeded
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtend.o succeeded
I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtend.o
attempt to open I:\Linux\lib\x86_64-linux-gnu\crtn.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crtn.o
ld-linux-x86-64.so.2 needed by I:/Linux/libs/lib/x86_64-linux-gnu/libc-2.17.so
found ld-2.17.so at I:\Linux\libs\lib\x86_64-linux-gnu\ld-2.17.so

nm output of windows generated executable file 

k@system:/shared$ nm ./Test
0000000000601040 B __bss_start
0000000000601040 b completed.6992
0000000000601030 D __data_start
0000000000601030 W data_start
0000000000400470 t deregister_tm_clones
00000000004004e0 t __do_global_dtors_aux
0000000000600e18 t __do_global_dtors_aux_fini_array_entry
0000000000601038 D __dso_handle
0000000000600e28 d _DYNAMIC
0000000000601040 D _edata
0000000000601048 B _end
00000000004005f4 T _fini
0000000000400500 t frame_dummy
0000000000600e10 t __frame_dummy_init_array_entry
0000000000400728 r __FRAME_END__
0000000000601000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000004003d8 T _init
0000000000600e18 t __init_array_end
0000000000600e10 t __init_array_start
0000000000400600 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000600e20 d __JCR_END__
0000000000600e20 d __JCR_LIST__
                 w _Jv_RegisterClasses
00000000004005f0 T __libc_csu_fini
0000000000400560 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
0000000000400530 T main
                 U printf@@GLIBC_2.2.5
00000000004004a0 t register_tm_clones
0000000000400440 T _start
0000000000601040 D __TMC_END__
k@system:/shared$ ./Test
bash: ./Test: No such file or directory

EDIT- more information

By won't run I mean ubuntu gives the message:

bash: ./Test: No such file or directory

Here is the result of file/ldd 

k@system:/shared$ file ./Test
./Test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0xee999db5a0e77d05f50d8fd78a27fc3ac52584b1, not stripped
k@system:/shared$ ldd ./Test
    linux-vdso.so.1 =>  (0x00007ffff2bfe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e2b60c000)
    /lib/ld64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f5e2b9eb000)

EDIT 2-

I've since tried to get closer to replicating the exact linking process on linux. 

LFLAGS = --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker  I:\\Linux\\libs\\lib\\x86_64-linux-gnu\\ld-2.17.so -z relro --verbose

LIBDIRS = -LI:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1 -LI:\\Linux\\libs\\lib\\x86_64-linux-gnu -LI:\\Linux\\libs\\lib -LI:\\Linux\\lib\\x86_64-linux-gnu -LI:\\Linux\\lib

LIBS_BEGIN = I:\\Linux\\lib\\x86_64-linux-gnu\\crt1.o I:\\Linux\\lib\\x86_64-linux-gnu\\crti.o I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\crtbegin.o 
LIBS_END = I:\\Linux\\lib\\gcc\\x86_64-linux-gnu\\4.8.1\\libstdc++.so I:\\Linux\\libs\\lib\\x86_64-linux-gnu\\libgcc_s.so.1 I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\libgcc.a  I:\\Linux\\lib\\x86_64-linux-gnu\\libc_nonshared.a I:\\Linux\\libs\\lib\\x86_64-linux-gnu\\ld-linux-x86-64.so.2 I:\\Linux\\libs\\lib\\x86_64-linux-gnu\\libc.so.6 I:\\Linux\\libs\\lib\\x86_64-linux-gnu\\libgcc_s.so.1 I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\libgcc.a I:\\Linux\\gcc\\x86_64-linux-gnu\\4.8.1\\crtend.o I:\\Linux\\lib\\x86_64-linux-gnu\\crtn.o

all:    $(MAIN)
    ./ld-new.exe $(LFLAGS) -o Test  $(LIBS_BEGIN) $(LIBDIRS) $(OBJS) $(LIBS_END)

Updated linking on windows: 

==================================================
attempt to open I:\Linux\lib\x86_64-linux-gnu\crt1.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crt1.o
attempt to open I:\Linux\lib\x86_64-linux-gnu\crti.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crti.o
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtbegin.o succeeded
I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtbegin.o
attempt to open Test.o succeeded
Test.o
attempt to open I:\Linux\lib\gcc\x86_64-linux-gnu\4.8.1\libstdc++.so succeeded
I:\Linux\lib\gcc\x86_64-linux-gnu\4.8.1\libstdc++.so
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\libgcc_s.so.1 succeeded
I:\Linux\libs\lib\x86_64-linux-gnu\libgcc_s.so.1
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\libgcc.a succeeded
attempt to open I:\Linux\lib\x86_64-linux-gnu\libc_nonshared.a succeeded
(I:\Linux\lib\x86_64-linux-gnu\libc_nonshared.a)elf-init.oS
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\ld-linux-x86-64.so.2 succeede
d
I:\Linux\libs\lib\x86_64-linux-gnu\ld-linux-x86-64.so.2
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\libc.so.6 succeeded
I:\Linux\libs\lib\x86_64-linux-gnu\libc.so.6
attempt to open I:\Linux\libs\lib\x86_64-linux-gnu\libgcc_s.so.1 succeeded
I:\Linux\libs\lib\x86_64-linux-gnu\libgcc_s.so.1
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\libgcc.a succeeded
attempt to open I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtend.o succeeded
I:\Linux\gcc\x86_64-linux-gnu\4.8.1\crtend.o
attempt to open I:\Linux\lib\x86_64-linux-gnu\crtn.o succeeded
I:\Linux\lib\x86_64-linux-gnu\crtn.o
ld-linux-x86-64.so.2 needed by I:\Linux\libs\lib\x86_64-linux-gnu\libc.so.6
found ld-linux-x86-64.so.2 at I:\Linux\libs\lib\x86_64-linux-gnu\ld-linux-x86-64
.so.2

Updated nm run on the file "Test"

0000000000601040 B __bss_start
0000000000601040 b completed.6992
0000000000601030 D __data_start
0000000000601030 W data_start
0000000000400490 t deregister_tm_clones
0000000000400500 t __do_global_dtors_aux
0000000000600e18 t __do_global_dtors_aux_fini_array_entry
0000000000601038 D __dso_handle
0000000000600e28 d _DYNAMIC
0000000000601040 D _edata
0000000000601048 B _end
0000000000400614 T _fini
0000000000400520 t frame_dummy
0000000000600e10 t __frame_dummy_init_array_entry
0000000000400748 r __FRAME_END__
0000000000601000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000004003f8 T _init
0000000000600e18 t __init_array_end
0000000000600e10 t __init_array_start
0000000000400620 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000600e20 d __JCR_END__
0000000000600e20 d __JCR_LIST__
                 w _Jv_RegisterClasses
0000000000400610 T __libc_csu_fini
0000000000400580 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
0000000000400550 T main
                 U printf@@GLIBC_2.2.5
00000000004004c0 t register_tm_clones
0000000000400460 T _start
0000000000601040 D __TMC_END__

Result of llvm-readobj on the file:

I:\LLVM_BUILD\VC12\64\Release\bin>llvm-readobj -file-headers Test

File: Test
Format: ELF64-x86-64
Arch: x86_64
AddressSize: 64bit
LoadName:
ElfHeader {
  Ident {
    Magic: (7F 45 4C 46)
    Class: 64-bit (0x2)
    DataEncoding: LittleEndian (0x1)
    FileVersion: 1
    OS/ABI: SystemV (0x0)
    ABIVersion: 0
    Unused: (00 00 00 00 00 00 00)
  }
  Type: Executable (0x2)
  Machine: EM_X86_64 (0x3E)
  Version: 1
  Entry: 0x400460
  ProgramHeaderOffset: 0x40
  SectionHeaderOffset: 0x11C0
  Flags [ (0x0)
  ]
  HeaderSize: 64
  ProgramHeaderEntrySize: 56
  ProgramHeaderCount: 9
  SectionHeaderEntrySize: 64
  SectionHeaderCount: 30
  StringTableSectionIndex: 27

And the actual linked, ostensibly executable file itself:

https://dl.dropboxusercontent.com/u/1735585/Test
Was it helpful?

Solution

The "No such file or directory" error can be a bit confusing if the file is there and executable. You will also get that error on executing a file if the ELF interpreter (the program that performs the shared library linking) isn't found.

That is what happens here. The relevant lines from readelf -l Test:

INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
               0x000000000000002e 0x000000000000002e  R      1
    [Requesting program interpreter: I:\Linux\libs\lib\x86_64-linux-gnu\ld-2.17.so]

Obviously that interpreter will not be found on a Linux system. For some reason ldd shows a file mapping to the correct interpreter for that entry.

I can't tell you how to fix that in your Windows build environment, but with a correct interpreter ldd won't show a mapping. For example:

$ ldd /bin/true
    linux-vdso.so.1 (0x00007fff1bbff000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f94ff99f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f94ffd73000)
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top