reproducing/debugging some multithreaded hell

https://stackoverflow.com/questions/21672532

09-10-2022
|

题

I have a binary file which runs fine 99% of the time. And from time to time, for no reason I can understand, It crashes with this output. After what I just relaunch and everything works fine.

The binary is a template system. it read some templates with placeholders like {%foo%} or {%include:foo%} whose respective for are:

to look for "foo" in the current context (std::map )
to include right there a subtemplate called "foo" ( as many times as a given stored procedures returns lines ) overwriting/adding to current std::map context.

Each template generator gets its own copy of its contexts and then execute in its own thread, where it saves the generated output to a file and send it to an MQ. Once every generators has returned, everything is cleaned up and the exe returns 0.

I do not even use a doubly linked list ( or any form or linked list anywhere ). I do not understand this crashes even though they seem to be related to some multithread heap corruption.

I would welcome any hint:

How could I reproduce this more often so I can debug it?

Where/what should i look for?

*** glibc detected *** /Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march: corrupted double-linked list: 0x08da3d18 ***

======= Backtrace: =========
/lib/libc.so.6[0x8dc5b6]
/lib/libc.so.6[0x8de3bd]
/lib/libc.so.6(__libc_malloc+0x7b)[0x8e03ab]
/opt/quest/lib/libvas.so.4(vassym_sqlite3Malloc+0x1e)[0xb65eb15e]
/opt/quest/lib/libvas.so.4(vassym_sqlite3pager_open+0x10a)[0xb65dc4aa]
/opt/quest/lib/libvas.so.4(vassym_sqlite3BtreeOpen+0x42)[0xb65c5a42]
/opt/quest/lib/libvas.so.4(vassym_sqlite3BtreeFactory+0x35)[0xb65d8cb5]
/opt/quest/lib/libvas.so.4[0xb65d953d]
/opt/quest/lib/libvas.so.4(vassql_init+0xea)[0xb6644d2e]
/opt/quest/lib/libvas.so.4(libvascache_misc_db_init+0xc4)[0xb66226ce]
/opt/quest/lib/libvas.so.4[0xb662b3b7]
/opt/quest/lib/libvas.so.4(vascache_miscinfo_get+0x4b)[0xb662b4cb]
/opt/quest/lib/libvas.so.4(vascache_init+0x1c1)[0xb6622ac1]
/lib/libnss_vas3.so.2[0xb66b9970]
/lib/libnss_vas3.so.2[0xb66ba52e]
/lib/libnss_vas3.so.2(_nss_vas3_getXXent_get_tsd+0x8c)[0xb66ba6b0]
/lib/libnss_vas3.so.2[0xb66bdf28]
/lib/libnss_vas3.so.2(_nss_vas3_getpwuid_r+0x1d)[0xb66bed85]
/lib/libc.so.6(getpwuid_r+0xa3)[0x9055a3]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5[0xb671c60f]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5[0xb670e049]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5[0xb670e233]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5[0xb670d499]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5(tdsdbopen+0x26c)[0xb66e15ef]
/Soft/fox_dev/redhat/dist/fxa/exe/sqldrivers/libqsqltds.so[0xb67495ce]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtSql.so.4(_ZN12QSqlDatabase4openEv+0x41)[0xb76eee81]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x806446b]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x8065e27]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x8065d30]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x8066298]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x8066528]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x80690cb]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x80699a4]
/Soft/fox_dev/redhat/env/Dev_FXA/exe/fxa_march[0x809db9c]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtCore.so.4[0xb6e5fe0f]
/Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtCore.so.4[0xb6e6c5f2]
/lib/libpthread.so.0[0x9ef49b]
/lib/libc.so.6(clone+0x5e)[0x94642e]
======= Memory map: ========
00356000-00365000 r-xp 00000000 fd:00 461183         /lib/libresolv-2.5.so
00365000-00366000 r-xp 0000e000 fd:00 461183         /lib/libresolv-2.5.so
00366000-00367000 rwxp 0000f000 fd:00 461183         /lib/libresolv-2.5.so
00367000-00369000 rwxp 00367000 00:00 0
00385000-00465000 r-xp 00000000 fd:00 597951         /usr/lib/libstdc++.so.6.0.8
00465000-00469000 r-xp 000df000 fd:00 597951         /usr/lib/libstdc++.so.6.0.8
00469000-0046a000 rwxp 000e3000 fd:00 597951         /usr/lib/libstdc++.so.6.0.8
0046a000-00470000 rwxp 0046a000 00:00 0
00495000-00497000 r-xp 00000000 fd:00 461203         /lib/libcom_err.so.2.1
00497000-00498000 rwxp 00001000 fd:00 461203         /lib/libcom_err.so.2.1
007b8000-007ba000 r-xp 00000000 fd:00 458824         /lib/libkeyutils-1.2.so
007ba000-007bb000 rwxp 00001000 fd:00 458824         /lib/libkeyutils-1.2.so
007e4000-007e8000 r-xp 00000000 fd:00 461202         /lib/libgthread-2.0.so.0.1200.3
007e8000-007e9000 rwxp 00003000 fd:00 461202         /lib/libgthread-2.0.so.0.1200.3
00857000-00871000 r-xp 00000000 fd:00 461149         /lib/ld-2.5.so
00871000-00872000 r-xp 00019000 fd:00 461149         /lib/ld-2.5.so
00872000-00873000 rwxp 0001a000 fd:00 461149         /lib/ld-2.5.so
00875000-009b3000 r-xp 00000000 fd:00 461180         /lib/libc-2.5.so
009b3000-009b5000 r-xp 0013e000 fd:00 461180         /lib/libc-2.5.so
009b5000-009b6000 rwxp 00140000 fd:00 461180         /lib/libc-2.5.so
009b6000-009b9000 rwxp 009b6000 00:00 0
009bb000-009bd000 r-xp 00000000 fd:00 461185         /lib/libdl-2.5.so
009bd000-009be000 r-xp 00001000 fd:00 461185         /lib/libdl-2.5.so
009be000-009bf000 rwxp 00002000 fd:00 461185         /lib/libdl-2.5.so
009c1000-009e6000 r-xp 00000000 fd:00 461182         /lib/libm-2.5.so
009e6000-009e7000 r-xp 00024000 fd:00 461182         /lib/libm-2.5.so
009e7000-009e8000 rwxp 00025000 fd:00 461182         /lib/libm-2.5.so
009ea000-009fd000 r-xp 00000000 fd:00 461188         /lib/libpthread-2.5.so
009fd000-009fe000 r-xp 00012000 fd:00 461188         /lib/libpthread-2.5.so
009fe000-009ff000 rwxp 00013000 fd:00 461188         /lib/libpthread-2.5.so
009ff000-00a01000 rwxp 009ff000 00:00 0
00a03000-00a15000 r-xp 00000000 fd:00 597527         /usr/lib/libz.so.1.2.3
00a15000-00a16000 rwxp 00011000 fd:00 597527         /usr/lib/libz.so.1.2.3
00a18000-00a1f000 r-xp 00000000 fd:00 461189         /lib/librt-2.5.so
00a1f000-00a20000 r-xp 00006000 fd:00 461189         /lib/librt-2.5.so
00a20000-00a21000 rwxp 00007000 fd:00 461189         /lib/librt-2.5.so
00a23000-00a39000 r-xp 00000000 fd:00 461192         /lib/libselinux.so.1
00a39000-00a3b000 rwxp 00015000 fd:00 461192         /lib/libselinux.so.1
00a3d000-00a78000 r-xp 00000000 fd:00 461191         /lib/libsepol.so.1
00a78000-00a79000 rwxp 0003a000 fd:00 461191         /lib/libsepol.so.1
00a79000-00a83000 rwxp 00a79000 00:00 0
00a85000-00b22000 r-xp 00000000 fd:00 461196         /lib/libglib-2.0.so.0.1200.3
00b22000-00b23000 rwxp 0009c000 fd:00 461196         /lib/libglib-2.0.so.0.1200.3
00b25000-00b2e000 r-xp 00000000 fd:00 461186         /lib/libcrypt-2.5.so
00b2e000-00b2f000 r-xp 00008000 fd:00 461186         /lib/libcrypt-2.5.so
00b2f000-00b30000 rwxp 00009000 fd:00 461186         /lib/libcrypt-2.5.so
00b30000-00b57000 rwxp 00b30000 00:00 0
00c8a000-00c95000 r-xp 00000000 fd:00 461190         /lib/libgcc_s-4.1.2-20080825.so.1
00c95000-00c96000 rwxp 0000a000 fd:00 461190         /lib/libgcc_s-4.1.2-20080825.so.1
00c98000-00cab000 r-xp 00000000 fd:00 461184         /lib/libnsl-2.5.so
00cab000-00cac000 r-xp 00012000 fd:00 461184         /lib/libnsl-2.5.so
00cac000-00cad000 rwxp 00013000 fd:00 461184         /lib/libnsl-2.5.so
00cad000-00caf000 rwxp 00cad000 00:00 0
08048000-080b2000 r-xp 00000000 00:26 181700         /Soft/fox_dev/redhat/dist/fxa/exe/fxa_march
080b2000-080b3000 rwxp 0006a000 00:26 181700         /Soft/fox_dev/redhat/dist/fxa/exe/fxa_march
08a80000-09f31000 rwxp 08a80000 00:00 0                  [heap]
af800000-af828000 rwxp af800000 00:00 0
af828000-af900000 --xp af828000 00:00 0
afa00000-afc00000 rwxp afa00000 00:00 0
afc00000-afd00000 rwxp afc00000 00:00 0
afe00000-aff00000 rwxp afe00000 00:00 0
b0000000-b0100000 rwxp b0000000 00:00 0
b0100000-b0200000 rwxp b0100000 00:00 0
b0200000-b0400000 rwxp b0200000 00:00 0
b0400000-b0600000 rwxp b0400000 00:00 0
b0600000-b0700000 rwxp b0600000 00:00 0
b0800000-b0a00000 rwxp b0800000 00:00 0
b0a00000-b0b00000 rwxp b0a00000 00:00 0
b0b00000-b0c00000 rwxp b0b00000 00:00 0
b0c00000-b0e00000 rwxp b0c00000 00:00 0
b0e00000-b0ff1000 rwxp b0e00000 00:00 0
b0ff1000-b1000000 --xp b0ff1000 00:00 0
b1000000-b1100000 rwxp b1000000 00:00 0
b1200000-b12f7000 rwxp b1200000 00:00 0
b12f7000-b1300000 --xp b12f7000 00:00 0
b1300000-b1400000 rwxp b1300000 00:00 0
b1400000-b14e5000 rwxp b1400000 00:00 0
b14e5000-b1500000 --xp b14e5000 00:00 0
b1500000-b15e9000 rwxp b1500000 00:00 0
b15e9000-b1600000 --xp b15e9000 00:00 0
b1600000-b16f6000 rwxp b1600000 00:00 0
b16f6000-b1700000 --xp b16f6000 00:00 0
b1700000-b17f8000 rwxp b1700000 00:00 0
b17f8000-b1800000 --xp b17f8000 00:00 0
b1800000-b18dd000 rwxp b1800000 00:00 0
b18dd000-b1900000 --xp b18dd000 00:00 0
b1a00000-b1c00000 rwxp b1a00000 00:00 0
b1c00000-b1e00000 rwxp b1c00000 00:00 0
b1e00000-b1ee5000 rwxp b1e00000 00:00 0
b1ee5000-b1f00000 --xp b1ee5000 00:00 0
b1f00000-b2000000 rwxp b1f00000 00:00 0
b2100000-b2200000 rwxp b2100000 00:00 0
b2200000-b2300000 rwxp b2200000 00:00 0
b2300000-b2400000 rwxp b2300000 00:00 0
b2474000-b2475000 --xp b2474000 00:00 0
b2475000-b2e75000 rwxp b2475000 00:00 0
b2e75000-b2e76000 --xp b2e75000 00:00 0
b2e76000-b3876000 rwxp b2e76000 00:00 0
b3876000-b3877000 --xp b3876000 00:00 0
b3877000-b4277000 rwxp b3877000 00:00 0
b4277000-b4278000 --xp b4277000 00:00 0
b4278000-b4c78000 rwxp b4278000 00:00 0
b4c78000-b4c79000 --xp b4c78000 00:00 0
b4c79000-b5679000 rwxp b4c79000 00:00 0
b5679000-b567a000 --xp b5679000 00:00 0
b567a000-b607a000 rwxp b567a000 00:00 0
b607a000-b60fb000 rwxp b641f000 00:00 0
b60fb000-b63ed000 r-xp 00000000 fd:00 461150         /lib/libnss_ldap-2.5.so
b63ed000-b6405000 rwxp 002f2000 fd:00 461150         /lib/libnss_ldap-2.5.so
b6405000-b6414000 rwxp b6405000 00:00 0
b6430000-b645e000 r-xp 00000000 fd:00 40163          /opt/mqm/lib/amqcctca_r
b645e000-b6460000 rwxp 0002e000 fd:00 40163          /opt/mqm/lib/amqcctca_r
b6460000-b6461000 rwxp b6460000 00:00 0
b6461000-b6496000 r-xs 00000000 fd:03 65542          /var/db/nscd/hosts
b6496000-b66ad000 r-xp 00000000 fd:00 989554         /opt/quest/lib/libvas.so.4.5.1
b66ad000-b66b7000 rwxp 00217000 fd:00 989554         /opt/quest/lib/libvas.so.4.5.1
b66b7000-b66c3000 r-xp 00000000 fd:00 989556         /opt/quest/lib/nss/libnss_vas3.so.2
b66c3000-b66c4000 rwxp 0000b000 fd:00 989556         /opt/quest/lib/nss/libnss_vas3.so.2
b66c4000-b66cd000 r-xp 00000000 fd:00 458792         /lib/libnss_files-2.5.so
b66cd000-b66ce000 r-xp 00008000 fd:00 458792         /lib/libnss_files-2.5.so
b66ce000-b66cf000 rwxp 00009000 fd:00 458792         /lib/libnss_files-2.5.so
b66da000-b673f000 r-xp 00000000 00:26 182147         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5
b673f000-b6742000 rwxp 00064000 00:26 182147         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5
b6742000-b674d000 r-xp 00000000 00:26 181921         /Soft/fox_dev/redhat/dist/fxa/exe/sqldrivers/libqsqltds.so
b674d000-b674e000 rwxp 0000b000 00:26 181921         /Soft/fox_dev/redhat/dist/fxa/exe/sqldrivers/libqsqltds.so
b674e000-b6750000 r-xp 00000000 fd:00 689014         /usr/lib/gconv/UTF-16.so
b6750000-b6752000 rwxp 00001000 fd:00 689014         /usr/lib/gconv/UTF-16.so
b6752000-b6759000 r-xs 00000000 fd:00 689043         /usr/lib/gconv/gconv-modules.cache
b6759000-b675a000 r-xp 01738000 fd:00 594281         /usr/lib/locale/locale-archive
b675a000-b6760000 r-xp 016bb000 fd:00 594281         /usr/lib/locale/locale-archive
b6760000-b6794000 r-xp 01682000 fd:00 594281         /usr/lib/locale/locale-archive
b6794000-b6994000 r-xp 00000000 fd:00 594281         /usr/lib/locale/locale-archive
b6994000-b6997000 rwxp b6994000 00:00 0
b6997000-b6a13000 r-xp 00000000 fd:00 40235          /opt/mqm/lib/libmqiz_r.so
b6a13000-b6a16000 rwxp 0007c000 fd:00 40235          /opt/mqm/lib/libmqiz_r.so
b6a16000-b6a1a000 rwxp b6a16000 00:00 0
b6a1a000-b6ac9000 r-xp 00000000 fd:00 598559         /usr/lib/libstdc++.so.5.0.7
b6ac9000-b6ace000 rwxp 000ae000 fd:00 598559         /usr/lib/libstdc++.so.5.0.7
b6ace000-b6ad3000 rwxp b6ace000 00:00 0
b6ad3000-b6c0b000 r-xp 00000000 fd:00 38366          /opt/mqm/lib/libmqmcs_r.so
b6c0b000-b6c6f000 rwxp 00137000 fd:00 38366          /opt/mqm/lib/libmqmcs_r.so
b6c6f000-b6c7d000 rwxp b6c6f000 00:00 0
b6c7d000-b6ca0000 r-xp 00000000 fd:00 40239          /opt/mqm/lib/libmqic_r.so
b6ca0000-b6ca3000 rwxp 00023000 fd:00 40239          /opt/mqm/lib/libmqic_r.so
b6ca3000-b6cbc000 r-xp 00000000 fd:00 40173          /opt/mqm/lib/3.2/libimqb23gl_r.so
b6cbc000-b6cc1000 rwxp 00019000 fd:00 40173          /opt/mqm/lib/3.2/libimqb23gl_r.so
b6cc1000-b6cc2000 rwxp b6cc1000 00:00 0
b6cc2000-b6cc8000 r-xp 00000000 00:27 70166          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libintl.so
b6cc8000-b6cc9000 rwxp 00006000 00:27 70166          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libintl.so
b6cc9000-b6d1d000 r-xp 00000000 00:27 70165          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libcomn.so
b6d1d000-b6d26000 rwxp 00053000 00:27 70165          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libcomn.so
b6d26000-b6d2a000 rwxp b6d26000 00:00 0
b6d2a000-b6d56000 r-xp 00000000 00:27 70160          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libsybtcl.so
b6d56000-b6d5a000 rwxp 0002b000 00:27 70160          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libsybtcl.so
b6d5a000-b6d5b000 rwxp b6d5a000 00:00 0
b6d5b000-b6d67000 r-xp 00000000 00:27 70168          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libcs.so
b6d67000-b6d68000 rwxp 0000b000 00:27 70168          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libcs.so
b6d68000-b6dc5000 r-xp 00000000 00:27 70169          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libct.so
b6dc5000-b6dd1000 rwxp 0005c000 00:27 70169          /Soft/sybase/V12.5.1_32_14757/OCS-12_5/lib/libct.so
b6dd1000-b6dd2000 rwxp b6dd1000 00:00 0
b6dd2000-b6ded000 r-xp 00000000 fd:00 40245          /opt/mqm/lib/3.2/libimqc23gl_r.so
b6ded000-b6df3000 rwxp 0001a000 fd:00 40245          /opt/mqm/lib/3.2/libimqc23gl_r.so
b6df3000-b6df4000 rwxp b6df3000 00:00 0
b6df9000-b6dfa000 rwxs 00000000 00:09 0                  /SYSV83868006 (deleted)
b6dfa000-b6dfb000 r-xp 00000000 fd:00 459612         /lib/libfipscheck.so.1.0.0
b6dfb000-b6dfc000 rwxp 00000000 fd:00 459612         /lib/libfipscheck.so.1.0.0
b6dfc000-b6dfd000 r-xp 00000000 fd:00 688966         /usr/lib/gconv/ISO8859-1.so
b6dfd000-b6dff000 rwxp 00000000 fd:00 688966         /usr/lib/gconv/ISO8859-1.so
b6dff000-b70af000 r-xp 00000000 00:26 184836         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtCore.so.4
b70af000-b70b8000 rwxp 002b0000 00:26 184836         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtCore.so.4
b70b8000-b70fe000 r-xp 00000000 00:26 186498         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtXml.so.4
b70fe000-b7100000 rwxp 00045000 00:26 186498         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtXml.so.4
b7100000-b723f000 r-xp 00000000 00:26 187416         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtNetwork.so.4
b723f000-b7244000 rwxp 0013e000 00:26 187416         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtNetwork.so.4
b7244000-b76bd000 r-xp 00000000 00:26 185561         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtXmlPatterns.so.4
b76bd000-b76de000 rwxp 00478000 00:26 185561         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtXmlPatterns.so.4
b76de000-b76df000 rwxp b76de000 00:00 0
b76df000-b771d000 r-xp 00000000 00:26 183945         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtSql.so.4
b771d000-b771e000 rwxp 0003e000 00:26 183945         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtSql.so.4
b771e000-b771f000 rwxp b771e000 00:00 0
b771f000-b79c1000 r-xp 00000000 00:26 185457         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtScript.so.4
b79c1000-b79cc000 rwxp 002a1000 00:26 185457         /Soft/fox_dev/redhat/dist/fx/libs/Qt/libQtScript.so.4
b79cc000-b79dc000 rwxp b79cc000 00:00 0
b79dc000-b7d38000 r-xp 00000000 00:26 578147         /Soft/fox_dev/xerces-c-3.1.1-x86-linux-gcc-3.4/lib/libxerces-c-3.1.so
b7d38000-b7dcc000 rwxp 0035c000 00:26 578147         /Soft/fox_dev/xerces-c-3.1.1-x86-linux-gcc-3.4/lib/libxerces-c-3.1.so
b7dcc000-b7dcd000 rwxp b7dcc000 00:00 0
b7dcd000-b7eb1000 r-xp 00000000 00:26 514553         /Soft/fox_dev/boost/boost_bin/lib/libboost_regex.so.1.54.0
b7eb1000-b7eb4000 rwxp 000e3000 00:26 514553         /Soft/fox_dev/boost/boost_bin/lib/libboost_regex.so.1.54.0
b7eb4000-b7ebb000 r-xp 00000000 00:26 514381         /Soft/fox_dev/boost/boost_bin/lib/libboost_chrono.so.1.54.0
b7ebb000-b7ebc000 rwxp 00006000 00:26 514381         /Soft/fox_dev/boost/boost_bin/lib/libboost_chrono.so.1.54.0
b7ebc000-b7ebf000 r-xp 00000000 00:26 514342         /Soft/fox_dev/boost/boost_bin/lib/libboost_system.so.1.54.0
b7ebf000-b7ec0000 rwxp 00002000 00:26 514342         /Soft/fox_dev/boost/boost_bin/lib/libboost_system.so.1.54.0
b7ec0000-b7ec1000 rwxp b7ec0000 00:00 0
b7ec1000-b7ec5000 r-xp 00000000 00:26 515549         /Soft/fox_dev/boost/boost_bin/lib/libboost_timer.so.1.54.0
b7ec5000-b7ec6000 rwxp 00003000 00:26 515549         /Soft/fox_dev/boost/boost_bin/lib/libboost_timer.so.1.54.0
b7ec6000-b7eca000 r-xp 00000000 00:26 180293         /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so
b7eca000-b7ecb000 rwxp 00004000 00:26 180293         /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so
b7ecb000-b7f30000 r-xp 00000000 00:26 167289         /Soft/fox_dev/redhat/dist/fx/lib/libfx.so
b7f30000-b7f32000 rwxp 00064000 00:26 167289         /Soft/fox_dev/redhat/dist/fx/lib/libfx.so
b7f32000-b7f33000 rwxp b7f32000 00:00 0
b7f33000-b7f34000 r-xp b7f33000 00:00 0                  [vdso]
bfc3e000-bfc53000 rwxp bffe8000 00:00 0                  [stack]
bfc53000-bfc55000 rw-p bfffd000 00:00 0
/Soft/fox_dev/redhat/env/Dev_FXA/bin/fxa_march.sh: line 28:  8322 Aborted                                 ${FXA_ESP}/exe/fxa_march -b $1 -t $2 -p $3 -o $4 -m $5 -d $6 -e $7
Fin du fichier de fxa_march.sh : 10/02/14 09:24:19

the lines about database open and Qtds scares me. Because there is very few I can do about that. It seems this driver/plugin is not very robust.

EDIT

this is my valgrind (partial) output on a simplified batch:

    ("/Soft/fox_dev/Qt-4.7.4/plugins", "/Soft/fox_dev/redhat/dist/fxa/exe")
    ==24089== Syscall param semctl(arg) points to uninitialised byte(s)
    ==24089==    at 0x947DDE: semctl@@GLIBC_2.2 (in /lib/libc-2.5.so)
    ==24089==    by 0x59B82B0: nss_vas3_semctl (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59B91A4: get_sem_id (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59B935A: get_tsd (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59B96AF: _nss_vas3_getXXent_get_tsd (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59BCF27: internal_nss_vas_getpwuid_r (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59BDD84: _nss_vas3_getpwuid_r (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x9055A2: getpwuid_r@@GLIBC_2.1.2 (in /lib/libc-2.5.so)
    ==24089==    by 0x597A60E: tds_get_homedir (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596C048: tds_get_home_file (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596C232: tds_read_conf_file (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596B498: tds_read_config_info (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==  Address 0xbe93d0a0 is on thread 1's stack
    ==24089==
    ==24089== Thread 2:
    ==24089== Syscall param semctl(arg) points to uninitialised byte(s)
    ==24089==    at 0x947DDE: semctl@@GLIBC_2.2 (in /lib/libc-2.5.so)
    ==24089==    by 0x59B82B0: nss_vas3_semctl (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59B9411: get_tsd (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59B96AF: _nss_vas3_getXXent_get_tsd (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59BCF27: internal_nss_vas_getpwuid_r (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x59BDD84: _nss_vas3_getpwuid_r (in /opt/quest/lib/nss/libnss_vas3.so.2)
    ==24089==    by 0x9055A2: getpwuid_r@@GLIBC_2.1.2 (in /lib/libc-2.5.so)
    ==24089==    by 0x597A60E: tds_get_homedir (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596C048: tds_get_home_file (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596C232: tds_read_conf_file (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x596B498: tds_read_config_info (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==    by 0x593F5EE: tdsdbopen (in /Soft/fox_dev/redhat/dist/fx/libs/Qt/libsybdb.so.5)
    ==24089==  Address 0x7532d50 is on thread 2's stack
    ==24089==
    ==24089== Conditional jump or move depends on uninitialised value(s)
    ==24089==    at 0x5286AA3: rriCopyChannelExits (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52BEA8B: ConvertMQCD (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AF797: DoConnect (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AE5EC: zstMQCONNX (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x4FFE7C6: MQCONNX (in /opt/mqm/lib/libmqic_r.so)
    ==24089==    by 0x4E9C212: ImqMgr::connect() (in /opt/mqm/lib/3.2/libimqc23gl_r.so)
    ==24089==    by 0x4075E07: MQConnection::connect(MQAccessType, std::string&) (in /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so)
    ==24089==    by 0x80614BA: reports::sendToQueue() (reports.cpp:611)
    ==24089==    by 0x8068152: reports::run() (reports.cpp:761)
    ==24089==    by 0x80A9644: reportGroup::run() (reportgroup.cpp:16)
    ==24089==    by 0x4C2AE0E: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==    by 0x4C375F1: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==
    ==24089== Conditional jump or move depends on uninitialised value(s)
    ==24089==    at 0x5286AF0: rriCopyChannelExits (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52BEA8B: ConvertMQCD (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AF797: DoConnect (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AE5EC: zstMQCONNX (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x4FFE7C6: MQCONNX (in /opt/mqm/lib/libmqic_r.so)
    ==24089==    by 0x4E9C212: ImqMgr::connect() (in /opt/mqm/lib/3.2/libimqc23gl_r.so)
    ==24089==    by 0x4075E07: MQConnection::connect(MQAccessType, std::string&) (in /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so)
    ==24089==    by 0x80614BA: reports::sendToQueue() (reports.cpp:611)
    ==24089==    by 0x8068152: reports::run() (reports.cpp:761)
    ==24089==    by 0x80A9644: reportGroup::run() (reportgroup.cpp:16)
    ==24089==    by 0x4C2AE0E: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==    by 0x4C375F1: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==
    ==24089== Conditional jump or move depends on uninitialised value(s)
    ==24089==    at 0x5286B37: rriCopyChannelExits (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52BEA8B: ConvertMQCD (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AF797: DoConnect (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AE5EC: zstMQCONNX (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x4FFE7C6: MQCONNX (in /opt/mqm/lib/libmqic_r.so)
    ==24089==    by 0x4E9C212: ImqMgr::connect() (in /opt/mqm/lib/3.2/libimqc23gl_r.so)
    ==24089==    by 0x4075E07: MQConnection::connect(MQAccessType, std::string&) (in /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so)
    ==24089==    by 0x80614BA: reports::sendToQueue() (reports.cpp:611)
    ==24089==    by 0x8068152: reports::run() (reports.cpp:761)
    ==24089==    by 0x80A9644: reportGroup::run() (reportgroup.cpp:16)
    ==24089==    by 0x4C2AE0E: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==    by 0x4C375F1: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==
    ==24089== Syscall param write(buf) points to uninitialised byte(s)
    ==24089==    at 0x9F644B: ??? (in /lib/libpthread-2.5.so)
    ==24089==    by 0x5270CCF: ccxSend (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52A8BA0: rriInitSess (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AF968: DoConnect (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AE5EC: zstMQCONNX (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x4FFE7C6: MQCONNX (in /opt/mqm/lib/libmqic_r.so)
    ==24089==    by 0x4E9C212: ImqMgr::connect() (in /opt/mqm/lib/3.2/libimqc23gl_r.so)
    ==24089==    by 0x4075E07: MQConnection::connect(MQAccessType, std::string&) (in /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so)
    ==24089==    by 0x80614BA: reports::sendToQueue() (reports.cpp:611)
    ==24089==    by 0x8068152: reports::run() (reports.cpp:761)
    ==24089==    by 0x80A9644: reportGroup::run() (reportgroup.cpp:16)
    ==24089==    by 0x4C2AE0E: ??? (in /Soft/fox_dev/Qt-4.7.4/lib/libQtCore.so.4.7.4)
    ==24089==  Address 0x65113af is 159 bytes inside a block of size 32,810 alloc'd
    ==24089==    at 0x4007100: malloc (vg_replace_malloc.c:270)
    ==24089==    by 0x50837AB: xcsGetMem (in /opt/mqm/lib/libmqmcs_r.so)
    ==24089==    by 0x527C174: cccGetMem (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x7D427FB: cciTcpAllocMem (in /opt/mqm/lib/amqcctca_r)
    ==24089==    by 0x5276698: ccxAllocMem (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52A883C: rriInitSess (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AF968: DoConnect (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x52AE5EC: zstMQCONNX (in /opt/mqm/lib/libmqiz_r.so)
    ==24089==    by 0x4FFE7C6: MQCONNX (in /opt/mqm/lib/libmqic_r.so)
    ==24089==    by 0x4E9C212: ImqMgr::connect() (in /opt/mqm/lib/3.2/libimqc23gl_r.so)
    ==24089==    by 0x4075E07: MQConnection::connect(MQAccessType, std::string&) (in /Soft/fox_dev/redhat/dist/fx/lib/libMQ.so)
    ==24089==    by 0x80614BA: reports::sendToQueue() (reports.cpp:611)
    ==24089==
    ==24089== Thread 1:
    ==24089== Invalid free() / delete / delete[] / realloc()
    ==24089==    at 0x4006B0D: free (vg_replace_malloc.c:446)
    ==24089==    by 0x985D2D: free_mem (in /lib/libc-2.5.so)
    ==24089==    by 0x9858A6: __libc_freeres (in /lib/libc-2.5.so)
    ==24089==    by 0x4001468: _vgnU_freeres (vg_preloaded.c:62)
    ==24089==    by 0x9065A3: _Exit (in /lib/libc-2.5.so)
    ==24089==    by 0x88AE93: (below main) (in /lib/libc-2.5.so)
    ==24089==  Address 0x4fc07a0 is not stack'd, malloc'd or (recently) free'd

Basically My mq lib from ibm and my sybase TDS driver are doing crap ? I checked "my calls" nearly everything is build on the stack locally inside one function or another.

解决方案

If you're accessing the database driver from multiple threads, then the problem is likely in the driver, and there are two solutions:

Protect all accesses to the driver by a mutex. Ideally, create a wrapper class for database access that enforces this.
Create multiple instances of the driver, one per each thread.

A connection can only be used from within the thread that created it. Moving connections between threads or creating queries from a different thread is not supported.

This means that you cannot use cloneDatabase in one thread, and use the cloned connection in another thread. You must create and then open a fresh, independent connection, by code running in the thread where you wish to use that connection.

You must be also very careful that whatever implicitly shared classes you pass to the threads are in fact separate instances. It is a bug if they are references, const references, or pointers. See this answer for details.

其他提示

I'd say

Syscall param semctl(arg) points to uninitialised byte(s)

matches this part:

I checked "my calls" nearly everything is build _on the stack locally inside one function or another_.

^ That leads to problems if a thread outlives the scope in which it was created. Specifically, if you create a semaphore on the stack, it would lead to errors like the one valgrind reported

#include <semaphore.h>
#include <time.h>
#include <assert.h>
#include <errno.h>
#include <signal.h>

void foo()
{
    sem_t sem;
    pthread_t th;
    pthread_attr_t attr;
    int s = pthread_attr_init(&attr);
    // error handling
    s = pthread_create(&th, &attr, &thread_start, static_cast<void*>(&attr));

} // oops

I know this is probably not what your code looks like, just sketching the picture :)

Because there is very few I can do about that. It seems this driver/plugin is not very robust.

See The First Rule of Programming: It's Always Your Fault

许可以下： CC-BY-SA 和归因

不隶属于 StackOverflow