How to debug random crash?

Added by 3togo over 5 years ago

The linux version of the etlegacy hanged randomly with freezing sound and without any keyboard nor mouse response. It happened randomly. I have removed compiz and fglrx. Is there any pointers or wiki to demonstrate how to debug the crash. I speculate that it is the SDL causing the crash but how can I prove it? /var/crash doesn’t pick up any logs.

Please advise.


Replies (22)

RE: How to debug random crash? - Added by IR4T4 over 5 years ago

Which version is used?

RE: How to debug random crash? - Added by 3togo over 5 years ago

IR4T4 wrote:

Which version is used?

etlegacy-linux-2.71rc3-full-installer.sh

Anyhow, I found the problem was server-related. The server etet.drdr.org triggered a random crash which is pretty annoying. But I think it is good to trace why but how?

RE: How to debug random crash? - Added by Radegast over 5 years ago

That server is offline at the moment, so I cannot connect to it. You would have to compile ET:L with build type set to Debug, execute ET:L under a debugger and connect to that server. It would then tell you what part of code caused the crash. It’s pretty simple once you know how to do it

RE: How to debug random crash? - Added by 3togo over 5 years ago

What debugger will you recommend?

RE: How to debug random crash? - Added by 3togo over 5 years ago

Sorry for typo mistakes
the server is :
etet.drtk.com

Radegast wrote:

That server is offline at the moment, so I cannot connect to it. You would have to compile ET:L with build type set to Debug, execute ET:L under a debugger and connect to that server. It would then tell you what part of code caused the crash. It’s pretty simple once you know how to do it

RE: How to debug random crash? - Added by 3togo over 5 years ago

etl +connect etet.drtk.org:27964
or

etl +connect 157.7.137.226:27964

3togo wrote:

Sorry for typo mistakes
the server is :
etet.drtk.com

Radegast wrote:

That server is offline at the moment, so I cannot connect to it. You would have to compile ET:L with build type set to Debug, execute ET:L under a debugger and connect to that server. It would then tell you what part of code caused the crash. It’s pretty simple once you know how to do it

RE: How to debug random crash? - Added by Radegast over 5 years ago

I use gdb debugger, but the problem with this mod happens before the crash. It’s because the mod isn’t loaded at all:

Failed loading /home/jsimek/.etlwolf/etet/ui.mp.i386.so: undefined symbol: CG_RW_ParseError

RE: How to debug random crash? - Added by Spyhawk over 5 years ago

I can run the etet v0.2.0 mod locally without any visible problem.

RE: How to debug random crash? - Added by Radegast over 5 years ago

Spyhawk: where did you download it from?

RE: How to debug random crash? - Added by Spyhawk over 5 years ago

Radegast wrote:

Spyhawk: where did you download it from?

http://etet.drtk.org

Check the "Release" link on the bottom.

RE: How to debug random crash? - Added by 3togo over 5 years ago

But I could run the mod remotely without a problem even with this error.

—– CL_Shutdown —–
Sys_LoadDll(/home/eli/.etlwolf/etet/ui.mp.i386.so)... failed: "Failed loading /home/eli/.etlwolf/etet/ui.mp.i386.so: /home/eli/.etlwolf/etet/ui.mp.i386.so: undefined symbol: CG_RW_ParseError"
Sys_LoadDll(/home/eli/.apps/etlegacy/etet/ui.mp.i386.so)... succeeded
Sys_LoadDll(ui) found vmMain function at 0xd99d4f80
145 UI cvars in use.
Total favourite servers restored: 0
Closing SDL audio device...
SDL audio device shut down.
RE_Shutdown( 1 )
SDL input devices shut down.
———————–
eli@eli-iMac:~$

Radegast wrote:

I use gdb debugger, but the problem with this mod happens before the crash. It’s because the mod isn’t loaded at all:

[...]

RE: How to debug random crash? - Added by IR4T4 over 5 years ago

Sys_LoadDll(/home/eli/.etlwolf/etet/ui.mp.i386.so)... failed: "Failed loading /home/eli/.etlwolf/etet/ui.mp.i386.so: /home/eli/.etlwolf/etet/ui.mp.i386.so: undefined symbol: CG_RW_ParseError" 

.. lib from homepath isn’t used
Sys_LoadDll(/home/eli/.apps/etlegacy/etet/ui.mp.i386.so)... succeeded

RE: How to debug random crash? - Added by 3togo over 5 years ago

I got two different version of ui.mp.i386.so. The one working is the one that have "no symbols" in symbol table. pretty obscure

eli@eli-iMac:~/.apps/etlegacy/etet$ objdump -x ui.mp.i386.so

ui.mp.i386.so: file format elf32-i386
ui.mp.i386.so
architecture: i386, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x00004430

Program Header:
LOAD off 0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**12
filesz 0x00045d94 memsz 0x00045d94 flags r-x
LOAD off 0x00046000 vaddr 0x00046000 paddr 0x00046000 align 2**12
filesz 0x00005bc0 memsz 0x00323700 flags rw-
DYNAMIC off 0x000462a8 vaddr 0x000462a8 paddr 0x000462a8 align 2**2
filesz 0x000000d0 memsz 0x000000d0 flags rw-
NOTE off 0x000000d4 vaddr 0x000000d4 paddr 0x000000d4 align 2**2
filesz 0x00000024 memsz 0x00000024 flags r–
STACK off 0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**2
filesz 0x00000000 memsz 0x00000000 flags rw-

Dynamic Section:
NEEDED libc.so.6
SONAME ui.mp.i386.so
INIT 0x00004294
FINI 0x0003ee68
HASH 0x000000f8
GNU_HASH 0x000001bc
STRTAB 0x000003dc
SYMTAB 0x000001fc
STRSZ 0x00000143
SYMENT 0x00000010
PLTGOT 0x00046440
PLTRELSZ 0x000000a8
PLTREL 0x00000011
JMPREL 0x000041ec
REL 0x0000059c
RELSZ 0x00003c50
RELENT 0x00000008
VERNEED 0x0000055c
VERNEEDNUM 0x00000001
VERSYM 0x00000520
RELCOUNT 0x00000787

Version References:
required from libc.so.6:
0x09691f73 0x00 04 GLIBC_2.1.3
0x0d696913 0x00 03 GLIBC_2.3
0x0d696910 0x00 02 GLIBC_2.0

Sections:
Idx Name Size VMA LMA File off Algn
0 .note.gnu.build-id 00000024 000000d4 000000d4 000000d4 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
1 .hash 000000c4 000000f8 000000f8 000000f8 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .gnu.hash 00000040 000001bc 000001bc 000001bc 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
3 .dynsym 000001e0 000001fc 000001fc 000001fc 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
4 .dynstr 00000143 000003dc 000003dc 000003dc 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
5 .gnu.version 0000003c 00000520 00000520 00000520 2**1
CONTENTS, ALLOC, LOAD, READONLY, DATA
6 .gnu.version_r 00000040 0000055c 0000055c 0000055c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
7 .rel.dyn 00003c50 0000059c 0000059c 0000059c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
8 .rel.plt 000000a8 000041ec 000041ec 000041ec 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
9 .init 00000030 00004294 00004294 00004294 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
10 .plt 00000160 000042c4 000042c4 000042c4 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
11 .text 0003aa38 00004430 00004430 00004430 2**4
CONTENTS, ALLOC, LOAD, READONLY, CODE
12 .fini 0000001c 0003ee68 0003ee68 0003ee68 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
13 .rodata 00006ef0 0003eea0 0003eea0 0003eea0 2**5
CONTENTS, ALLOC, LOAD, READONLY, DATA
14 .eh_frame 00000004 00045d90 00045d90 00045d90 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
15 .ctors 0000000c 00046000 00046000 00046000 2**2
CONTENTS, ALLOC, LOAD, DATA
16 .dtors 00000008 0004600c 0004600c 0004600c 2**2
CONTENTS, ALLOC, LOAD, DATA
17 .jcr 00000004 00046014 00046014 00046014 2**2
CONTENTS, ALLOC, LOAD, DATA
18 .data.rel.ro 00000288 00046020 00046020 00046020 2**5
CONTENTS, ALLOC, LOAD, DATA
19 .dynamic 000000d0 000462a8 000462a8 000462a8 2**2
CONTENTS, ALLOC, LOAD, DATA
20 .got 000000c8 00046378 00046378 00046378 2**2
CONTENTS, ALLOC, LOAD, DATA
21 .got.plt 00000060 00046440 00046440 00046440 2**2
CONTENTS, ALLOC, LOAD, DATA
22 .data 00005720 000464a0 000464a0 000464a0 2**5
CONTENTS, ALLOC, LOAD, DATA
23 .bss 0031db40 0004bbc0 0004bbc0 0004bbc0 2**5
ALLOC
24 .comment 0000001c 00000000 00000000 0004bbc0 2**0
CONTENTS, READONLY
SYMBOL TABLE:
no symbols

ui.mp.i386.so (304 KB)

ui.mp.i386.so.bad (430 KB)

RE: How to debug random crash? - Added by 3togo over 5 years ago

The crash seems to be related to the stability of the network. As I failed to cross compile a debug version of et:L, I can’t do anything to trace the root cause

RE: How to debug random crash? - Added by 3togo over 5 years ago

I have eventually compiled with debug information and then I run

gdb –args ./etl +connect etet.drtk.org:27964

gdb>run

The program crashed without any keyboard responses. Is there any hotkey so that I could trigger gdb?

Joe

RE: How to debug random crash? - Added by Radegast over 5 years ago

Probably there is a way to do it and if you find out, please share the procedure with me.

My workaround is to run ET:L in a windowed mode (+set r_fullscreen 0) and when the crash occurs just Alt+TAB to the terminal and do backtrace.

RE: How to debug random crash? - Added by xafxs over 5 years ago

I’m etet developer. Having confirmed bg_ code had a invalid symbol reference in the ui module, I fixed it in latest revision and etet 0.2.1.
I appreciate your reporting this issue.

RE: How to debug random crash? - Added by 3togo over 5 years ago

xafxs wrote:

I’m etet developer. Having confirmed bg_ code had a invalid symbol reference in the ui module, I fixed it in latest revision and etet 0.2.1.
I appreciate your reporting this issue.

Thank you to Xafxs. I did compile your revised version. However, it crashed differently this time. The crash occurred after the completion of each round of the game. Below is the screen dump:

    Please forward to etet mod team.   
---------------------------------------
Version: 0.3.0 Linux
Map: etdo5
Signal: Segmentation fault (11)
Siginfo: 0xffdea78c
Code: 1
Faulting Memory Ref/Instruction: 0x10
DSO Information:
0xf7721000    (unknown)
0xf76da000    /lib/i386-linux-gnu/libpthread.so.0
0xf7675000    /usr/lib/i386-linux-gnu/libcurl.so.4
0xf75dc000    /usr/lib/i386-linux-gnu/libSDL-1.2.so.0
0xf7541000    /usr/lib32/fglrx/libGL.so.1
0xf74e6000    /usr/lib/i386-linux-gnu/libjpeg.so.8
0xf73fd000    /usr/lib32/libstdc++.so.6
0xf73ba000    /lib/i386-linux-gnu/libm.so.6
0xf739c000    /usr/lib32/libgcc_s.so.1
0xf71e8000    /lib/i386-linux-gnu/libc.so.6
0xf7720000    /lib/ld-linux.so.2
0xf71b5000    /usr/lib/i386-linux-gnu/libidn.so.11
0xf719a000    /usr/lib/i386-linux-gnu/librtmp.so.0
0xf7143000    /lib/i386-linux-gnu/libssl.so.1.0.0
0xf6f95000    /lib/i386-linux-gnu/libcrypto.so.1.0.0
0xf6f58000    /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2
0xf6f49000    /usr/lib/i386-linux-gnu/liblber-2.4.so.2
0xf6ef7000    /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2
0xf6edd000    /usr/lib32/libz.so.1
0xf6de6000    /usr/lib/i386-linux-gnu/libasound.so.2
0xf6de1000    /lib/i386-linux-gnu/libdl.so.2
0xf6ddc000    /usr/lib/i386-linux-gnu/libpulse-simple.so.0
0xf6d8d000    /usr/lib/i386-linux-gnu/libpulse.so.0
0xf6c58000    /usr/lib/i386-linux-gnu/libX11.so.6
0xf6c44000    /usr/lib/i386-linux-gnu/libXext.so.6
0xf6b78000    /usr/lib/i386-linux-gnu/libcaca.so.0
0xf6b5d000    /usr/lib32/fglrx/libatiuki.so.1
0xf6a97000    /usr/lib/i386-linux-gnu/libgnutls.so.26
0xf6a13000    /lib/i386-linux-gnu/libgcrypt.so.11
0xf6943000    /usr/lib/i386-linux-gnu/libkrb5.so.3
0xf691b000    /usr/lib/i386-linux-gnu/libk5crypto.so.3
0xf6916000    /lib/i386-linux-gnu/libcom_err.so.2
0xf690d000    /usr/lib/i386-linux-gnu/libkrb5support.so.0
0xf68f6000    /lib/i386-linux-gnu/libresolv.so.2
0xf68da000    /usr/lib/i386-linux-gnu/libsasl2.so.2
0xf689e000    /usr/lib/i386-linux-gnu/libgssapi.so.3
0xf6895000    /lib/i386-linux-gnu/librt.so.1
0xf6826000    /usr/lib/i386-linux-gnu/pulseaudio/libpulsecommon-4.0.so
0xf681b000    /lib/i386-linux-gnu/libjson-c.so.2
0xf67cf000    /lib/i386-linux-gnu/libdbus-1.so.3
0xf67ae000    /usr/lib/i386-linux-gnu/libxcb.so.1
0xf667f000    /lib/i386-linux-gnu/libslang.so.2
0xf6648000    /lib/i386-linux-gnu/libncursesw.so.5
0xf6626000    /lib/i386-linux-gnu/libtinfo.so.5
0xf6613000    /usr/lib/i386-linux-gnu/libtasn1.so.3
0xf65f4000    /usr/lib/i386-linux-gnu/libp11-kit.so.0
0xf65ef000    /lib/i386-linux-gnu/libgpg-error.so.0
0xf65eb000    /lib/i386-linux-gnu/libkeyutils.so.1
0xf65e3000    /usr/lib/i386-linux-gnu/libheimntlm.so.0
0xf6561000    /usr/lib/i386-linux-gnu/libkrb5.so.26
0xf64c1000    /usr/lib/i386-linux-gnu/libasn1.so.8
0xf648b000    /usr/lib/i386-linux-gnu/libhcrypto.so.4
0xf6475000    /usr/lib/i386-linux-gnu/libroken.so.18
0xf646b000    /lib/i386-linux-gnu/libwrap.so.0
0xf63f8000    /usr/lib/i386-linux-gnu/libsndfile.so.1
0xf63f1000    /usr/lib/i386-linux-gnu/libasyncns.so.0
0xf63ed000    /usr/lib/i386-linux-gnu/libXau.so.6
0xf63e6000    /usr/lib/i386-linux-gnu/libXdmcp.so.6
0xf63bc000    /usr/lib/i386-linux-gnu/libwind.so.0
0xf63ac000    /usr/lib/i386-linux-gnu/libheimbase.so.1
0xf6365000    /usr/lib/i386-linux-gnu/libhx509.so.5
0xf62b0000    /usr/lib/i386-linux-gnu/libsqlite3.so.0
0xf627e000    /lib/i386-linux-gnu/libcrypt.so.1
0xf6265000    /lib/i386-linux-gnu/libnsl.so.1
0xf6230000    /usr/lib/i386-linux-gnu/libFLAC.so.8
0xf60b8000    /usr/lib/i386-linux-gnu/libvorbisenc.so.2
0xf608c000    /usr/lib/i386-linux-gnu/libvorbis.so.0
0xf6083000    /usr/lib/i386-linux-gnu/libogg.so.0
0xf770c000    /lib/i386-linux-gnu/libnss_compat.so.2
0xf7700000    /lib/i386-linux-gnu/libnss_nis.so.2
0xec7c5000    /lib/i386-linux-gnu/libnss_files.so.2
0xeac49000    /usr/lib/i386-linux-gnu/libXcursor.so.1
0xeac3e000    /usr/lib/i386-linux-gnu/libXrender.so.1
0xf76f8000    /usr/lib/i386-linux-gnu/libXfixes.so.3
0xe890e000    /usr/lib32/fglrx/dri/fglrx_dri.so
0xe883b000    /usr/lib32/fglrx/libatiadlxx.so
0xec7ef000    /usr/lib/i386-linux-gnu/libXinerama.so.1
0xec7d6000    /lib/i386-linux-gnu/libnss_dns.so.2
0xd60aa000    /home/eli/.etlegacy/etet/ui.mp.i386.so
0xd23db000    /home/eli/.etlegacy/etet/cgame.mp.i386.so
Stack frames: 2 entries
Backtrace:
(1) .() [0x813e671]
(2) [0xf771f410]
-8<--------------------------------->8-

Attempting to clean up.
tDOUBLE SIGNAL FAULT: Received signal 11, exiting...
----- CL_Shutdown -----
Sys_LoadDll(/home/eli/.etlegacy/etet/ui.mp.i386.so)... succeeded
Sys_LoadDll(ui) found vmMain function at 0xd60d51e0
Segmentation fault (core dumped)
eli@eli-iMac:~/git/etlegacy/build$ 

RE: How to debug random crash? - Added by 3togo over 5 years ago

Another core dump

*** Error in `.': double free or corruption (!prev): 0x0b96e378 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x767c2)[0xf73417c2]
/lib/i386-linux-gnu/libc.so.6(+0x77510)[0xf7342510]
/usr/lib/i386-linux-gnu/libSDL2-2.0.so.0(SDL_free+0x1b)[0xf7680e6b]
/usr/lib/i386-linux-gnu/libSDL2-2.0.so.0(SDL_FreeSurface+0xd0)[0xf76bb470]
/usr/lib/i386-linux-gnu/libSDL2-2.0.so.0(SDL_DestroyWindow+0xf4)[0xf76c02a4]
/usr/lib/i386-linux-gnu/libSDL2-2.0.so.0(SDL_VideoQuit+0x58)[0xf76c04d8]
/usr/lib/i386-linux-gnu/libSDL2-2.0.so.0(SDL_QuitSubSystem+0x205)[0xf762a0b5]
.[0x80f505c]
.[0x80df88a]
*** Error in `.': corrupted double-linked list.[0x80ecea0]
.[0x80ef0ed]
.[0x80b79c6]
.[0x8088212]
/home/eli/.etlwolf/etet/cgame.mp.i386.so(trap_R_RegisterShader+0x37)[0xd2c8af97]
DOUBLE SIGNAL FAULT: Received signal 6, exiting...
/home/eli/.etlwolf/etet/cgame.mp.i386.so(+0x7d99a)[0xd2c5899a]
Segmentation fault (core dumped)
eli@eli-iMac:~/etlegacy/build_release$ 

xafxs wrote:

I’m etet developer. Having confirmed bg_ code had a invalid symbol reference in the ui module, I fixed it in latest revision and etet 0.2.1.
I appreciate your reporting this issue.

RE: How to debug random crash? - Added by xafxs over 5 years ago

I found cgame.mp.i386.so(+0x7d99a) is below code in CG_RegisterGraphics().
cgs.media.onFireShader2 = trap_R_RegisterShader( "entityOnFire1" );
In etet r100, I removed unused cgMedia_t members and did clean CG_RegisterGraphics(), therefore above code may be affected by the changes, so I rollbacked ones.

I think this problem does not relate to the engine, so please report to http://etet.drtk.org/forum/ if you get crash even in latest revision.

RE: How to debug random crash? - Added by Dragonji over 5 years ago

xafxs wrote:

I think this problem does not relate to the engine, so please report to http://etet.drtk.org/forum/ if you get crash even in latest revision.

Any chance for English website?

RE: How to debug random crash? - Added by xafxs over 5 years ago

Dragonji wrote:

xafxs wrote:

I think this problem does not relate to the engine, so please report to http://etet.drtk.org/forum/ if you get crash even in latest revision.

Any chance for English website?

Forum’s default language is turned to English now. but it’s difficult to rewrite wiki for English because I cannot write English early.

(1-22/22)