Ruby installation on RHEL works, and then starts segfaulting

Question

We have server which came with RHEL 5.8, unregistered - so no yum. We compiled and installed Ruby directly. It works - and then a few days later ruby programs start segfaulting. Can anyone suggest where to start looking for this error?

/usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1161: [BUG] Segmentation fault
ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0008 p:---- s:0021 b:0021 l:000020 d:000020 CFUNC  :require
c:0007 p:1596 s:0017 b:0017 l:000016 d:000016 CLASS  /usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1161
c:0006 p:0079 s:0015 b:0015 l:000014 d:000014 TOP    /usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:114
c:0005 p:---- s:0012 b:0012 l:000011 d:000011 FINISH
c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC  :require
c:0003 p:0021 s:0006 b:0006 l:000005 d:000005 TOP    <internal:gem_prelude>:1
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:000d18 d:000d18 TOP   

-- Ruby level backtrace information ----------------------------------------
<internal:gem_prelude>:1:in `<compiled>'
<internal:gem_prelude>:1:in `require'
/usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:114:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1161:in `<module:Gem>'
/usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:1161:in `require'

-- C level backtrace information -------------------------------------------
/usr/bin/ruby [0x525545] vm_dump.c:796
/usr/bin/ruby [0x569b96] error.c:258
/usr/bin/ruby(rb_bug+0xb3) [0x569d03] error.c:277
/usr/bin/ruby [0x4b1749] signal.c:633
/lib64/libpthread.so.0 [0x3a1700ebe0]
/usr/bin/ruby(st_lookup+0x26) [0x4b8136] st.c:333
/usr/bin/ruby [0x51c723] vm_method.c:374
/usr/bin/ruby [0x44bcef] object.c:2062
/usr/bin/ruby(rb_convert_type+0x5a) [0x44e87a] object.c:2083
/usr/bin/ruby(rb_string_value_ptr+0x27) [0x4c02c7] string.c:810
/usr/bin/ruby [0x56cabe] load.c:156
/usr/bin/ruby(rb_require_safe+0x20f) [0x56d6cf] load.c:550
/usr/bin/ruby [0x522a94] vm_insnhelper.c:404
/usr/bin/ruby [0x514697] insns.def:1018
/usr/bin/ruby [0x51ad48] vm.c:1236
/usr/bin/ruby(rb_iseq_eval+0x152) [0x51b292] vm.c:1464
/usr/bin/ruby [0x56c486] load.c:310
/usr/bin/ruby(rb_require_safe+0x6c6) [0x56db86] load.c:620
/usr/bin/ruby [0x522a94] vm_insnhelper.c:404
/usr/bin/ruby [0x514697] insns.def:1018
/usr/bin/ruby [0x51ad48] vm.c:1236
/usr/bin/ruby(rb_iseq_eval+0x152) [0x51b292] vm.c:1464
/usr/bin/ruby(ruby_process_options+0x325) [0x4b06c5] ruby.c:1116
/usr/bin/ruby(ruby_options+0x90) [0x4178b0] eval.c:73
/usr/bin/ruby [0x414985] main.c:38
/lib64/libc.so.6(__libc_start_main+0xf4) [0x3a1601d994]
/usr/bin/ruby [0x414889]

-- Other runtime information -----------------------------------------------

* Loaded script: /usr/bin/ruby

* Loaded features:

    0 enumerator.so
    1 /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
    2 /usr/local/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
Segmentation fault

Answer 1

A wild guess.

RHEL systems are by default configured to prelink shared libraries to a random address, once every 2 weeks.

Perhaps your copy of ruby doesn't work correctly when it is prelinked?

It starts working after a fresh build, then prelink is performed on all shared libraries, and it stops working until next fresh install.

You can find out whether the library is prelinked or not by running:

readelf -l encdb.so | grep LOAD | head -1

Un-prelinked library will have first load segment at 0. Prelinked library will have it as non-zero.

Update: this link suggest that Ruby indeed may not work with prelinking. Solution from there (copied for posterity):

Add -b /usr/bin/ruby to your /etc/prelink.conf file, this tells prelink to not touch the ruby interpreter.