Why do shebangs and commands from certain scripts (python, perl, libtool) are wrongly interpreted by bash?

Question

EDIT : this problem is still there : I rebuilt my entire system and the chroot I use for it on a sane host works like a charm. But once I boot the same target system, the problem appears again: shebang for perl and python are wrongly interpreted...

I've been building and administrating my own Linux system successfully for about 4 years but I'm kinda stuck now and I'd appreciate very very much to get directions to fix that hellish problem:

Some Python, Perl scripts or even commands from within 'configure' scripts are misinterpreted: I've identified python-config, xscreensaver-text and intltool-merge (Perl scripts) whose Shebangs are not respected or are only interpreted by the shell.

We've been messing around with locales, encondings, considerations about UTF-8, checking env, $LANG, $TERM and such, switching Linux kernels (3.1.5, 3.5.4, 3.8.1, 3.11.1), but still no success.

Here is the strace output from xscreensaver-text, first with a simple call form the CLI then a second call with explicit Perl. Its shebang is:

#!/bin/perl -w

root@poopy:~# strace /usr/bin/xscreensaver-text
execve("/usr/bin/xscreensaver-text", ["/usr/bin/xscreensaver-text"], [/* 32 vars */]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
exit_group(1)                           = ?
+++ exited with 1 +++

root@poopy:~# strace perl /usr/bin/xscreensaver-text
execve("/usr/bin/perl", ["perl", "/usr/bin/xscreensaver-text"], [/* 32 vars */]) = 0
brk(0)                                  = 0x601000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd74eb1d000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=198969, ...}) = 0
mmap(NULL, 198969, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd74eaec000
close(3)                                = 0
open("/usr/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868472, ...}) = 0
mmap(NULL, 3981888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd74e52f000
mprotect(0x7fd74e6f1000, 2097152, PROT_NONE) = 0
mmap(0x7fd74e8f1000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7fd74e8f1000
mmap(0x7fd74e8f7000, 16960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd74e8f7000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd74eaeb000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd74eae9000
arch_prctl(ARCH_SET_FS, 0x7fd74eae9740) = 0
mprotect(0x7fd74e8f1000, 16384, PROT_READ) = 0
mprotect(0x7fd74eb1e000, 4096, PROT_READ) = 0
munmap(0x7fd74eaec000, 198969)          = 0
brk(0)                                  = 0x601000
brk(0x622000)                           = 0x622000
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fd74eae9a10) = 16061
wait4(16061, poopy
Linux 3.11.1

ven. 27 sept. 2013 14:06:27 CEST

up  2:17,  1 user
load average: 0,03, 0,06, 0,12

[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 16061
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=16061, si_status=0, si_utime=2, si_stime=0} ---
exit_group(0)                           = ?
+++ exited with 0 +++

My PATH is:

/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib64/jdk/bin:/usr/lib64/jdk/jre/bin

It is on an x86_64 system with i686 multilib capabilities. My locale is fr_FR.utf8 ($LANG and $LC_ALL). I will add any info you guys will need. My shell in '/etc/passwd' is '/bin/bash', it exists and works flawlessly. Tried with normal users and root, no difference. Bash 4.2, linux 3.11.1, glibc 2.17. Tried in xterms, xfce-terminal and ttys, same problem.

The 'python-config' or 'intltool-merge' (and many others) have exactly the same behaviour. I even saw once a 'configure' script failing with the message "ac_cv_somevariable: command not found". Any idea where to look for debugging this?

Thanks.

Answer 1

Since Linux 3.10 a new option called CONFIG_BINFMT_SCRIPT must be set to Y if you want your kernel to handle interpreted scripts.

More explainations here : http://cateee.net/lkddb/web-lkddb/BINFMT_SCRIPT.html

If you compiled it as a module, try to modprobe the binfmt_script module and then test your scripts.

Answer 2

Did you try

#!/usr/bin/perl

instead of

#!/bin/perl

?

Answer 3

RESOLVED!

The kernel has a new option since 3.10: CONFIG_BINFMT_SCRIPT. It was compiled as a module here, so non-loaded by default! Loading the module with modprobe binfmt_script or building the kernel with CONFIG_BINFMT_SCRIPT=y fixes the problem. Thanks!

Answer 4

Note: This analysis is WRONG. od -x outputs "short converted (2 byte) ints". I thought it was outputting od -xC which would be all bytes in network order.

Quick analysis of the binary dump:

2123 752f 7273 622f 6e69  // hex bytes
! #  u /  r s  b /  n i

!#u/rsb/ni     // condensed
#!/usr/bin     // 2-flipped
#!/bin/perl -w // question line

My guess would be the problem isn't on your system, but with encoding and interpretation on one of the systems between yours and the public repository on-line for each of these files.

Sniffing the files as they're downloaded might help? (hopefully you can find something unencrypted, you could also check a binary dump at each machine hop)

EDIT: Vim keys should be fixed by a corrected /etc/inputrc in most cases. If not (and you're not using zsh) then it could be related.