Tuesday, January 25, 2011

view call stack of crashed application on android

On android, when a process crashes in native code, the call stack of the process will be saved to a log file in /data/tombstomes/, and written to logcat as well. The information is helpful for debugging.
Unfortunately, the call stack doesn't show in human readable format, file name, function name. Instead, it's shown as module name (e.g., libc.so) and memory address of the instruction. We can use addr2line to translate the address to corresponding file name and function name if we have the binary of the module that contains symbol information.
To make it easier to use, this function is included in agdb tool (see here for more). We can use "agdb.py -r -e module_name address" to find out the function name of specified address within the module.

When we have a long call stack, instead of running the command above for each line in the call stack manually, we can feed the whole call stack to agdb through pipe and get the full resolved call stack. For example, use  "adb logcat | agdb.py -r" command for adb logcat output with below contents:

22 F/ASessionDescription(   33): frameworks/base/media/libstagefright/rtsp/ASessionDescription.cpp:264 CHECK_GT( end,s) failed:  vs.
23 I/DEBUG   (   30): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
24 I/DEBUG   (   30): Build fingerprint: 'generic/generic/generic:2.3.1/GINGERBREAD/eng.raymond.20101222.130550:eng/test-keys'
25 I/DEBUG   (   30): pid: 33, tid: 450  >>> /system/bin/mediaserver <<<
26 I/DEBUG   (   30): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad
27 I/DEBUG   (   30):  r0 deadbaad  r1 0000000c  r2 00000027  r3 00000000
28 I/DEBUG   (   30):  r4 00000080  r5 afd46668  r6 40806c10  r7 00000000
29 I/DEBUG   (   30):  r8 8031db1d  r9 0000fae0  10 00100000  fp 00000001
30 I/DEBUG   (   30):  ip ffffffff  sp 40806778  lr afd19375  pc afd15ef0  cpsr 00000030
31 I/DEBUG   (   30):          #00  pc 00015ef0  /system/lib/libc.so
32 I/DEBUG   (   30):          #01  pc 00001440  /system/lib/liblog.so
33 I/DEBUG   (   30):
34 I/DEBUG   (   30): code around pc:
35 I/DEBUG   (   30): afd15ed0 68241c23 d1fb2c00 68dae027 d0042a00
36 I/DEBUG   (   30): afd15ee0 20014d18 6028447d 48174790 24802227
37 I/DEBUG   (   30): afd15ef0 f7f57002 2106eb56 ec92f7f6 0563aa01
38 I/DEBUG   (   30): afd15f00 60932100 91016051 1c112006 e818f7f6
39 I/DEBUG   (   30): afd15f10 2200a905 f7f62002 f7f5e824 2106eb42
40 I/DEBUG   (   30):
41 I/DEBUG   (   30): code around lr:
42 I/DEBUG   (   30): afd19354 b0834a0d 589c447b 26009001 686768a5
43 I/DEBUG   (   30): afd19364 220ce008 2b005eab 1c28d003 47889901
44 I/DEBUG   (   30): afd19374 35544306 d5f43f01 2c006824 b003d1ee
45 I/DEBUG   (   30): afd19384 bdf01c30 000281a8 ffffff88 1c0fb5f0
46 I/DEBUG   (   30): afd19394 43551c3d a904b087 1c16ac01 604d9004
47 I/DEBUG   (   30):
48 I/DEBUG   (   30): stack:
49 ........................
92 I/DEBUG   (   30):     408067e4  6f697470
93 I/BootReceiver(   75): Copying /data/tombstones/tombstone_09 to DropBox (SYSTEM_TOMBSTONE)

we get:


22 F/ASessionDescription(   33): frameworks/base/media/libstagefright/rtsp/ASessionDescription.cpp:264 CHECK_GT( end,s) failed:  vs.
23 I/DEBUG   (   30): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
24 I/DEBUG   (   30): Build fingerprint: 'generic/generic/generic:2.3.1/GINGERBREAD/eng.raymond.20101222.130550:eng/test-keys'
25 I/DEBUG   (   30): pid: 33, tid: 450  >>> /system/bin/mediaserver <<<
26 I/DEBUG   (   30): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad
27 I/DEBUG   (   30):  r0 deadbaad  r1 0000000c  r2 00000027  r3 00000000
28 I/DEBUG   (   30):  r4 00000080  r5 afd46668  r6 40806c10  r7 00000000
29 I/DEBUG   (   30):  r8 8031db1d  r9 0000fae0  10 00100000  fp 00000001
30 I/DEBUG   (   30):  ip ffffffff  sp 40806778  lr afd19375  pc afd15ef0  cpsr 00000030
31 I/DEBUG   (   30):          #00  pc 00015ef0  /system/lib/libc.so
32 I/DEBUG   (   30):          #00  __libc_android_abort: abort.c:82
33 I/DEBUG   (   30):          #01  pc 00001440  /system/lib/liblog.so
34 I/DEBUG   (   30):          #01  __android_log_assert: logd_write.c:235
35 I/DEBUG   (   30):
36 I/DEBUG   (   30): code around pc:
37 I/DEBUG   (   30): afd15ed0 68241c23 d1fb2c00 68dae027 d0042a00
38 I/DEBUG   (   30): afd15ee0 20014d18 6028447d 48174790 24802227
39 I/DEBUG   (   30): afd15ef0 f7f57002 2106eb56 ec92f7f6 0563aa01

Similarly, we copy a tombstone file to our development pc, and use "cat tombstone_01.txt | agdb.py -r" command to resolve call stack addresses in the tombstone log file.

11 comments:

Anonymous said...

Good stuff. thanks.

rxwen said...

If you use vim, do check out this post:
ease android stacktrace examination in vim with agdb

Anonymous said...

I think we need to have root permission to execute "echo tombstone_01.txt | agdb.py -r" (or "cat tombstone_01.txt | agdb.py -r")

Is there any equivalent way to achieve this with user permission?

rxwen said...

May be I'm not clear enough in the post. The "cat tombstone_01.txt | agdb.py -r" command is run on your development pc, not on the android device.
So, it's not required to have root permission. A non-root user should suffice to copy the tombstone file from device.

ajay kasam said...

Can we use this script for Android running Intel based processor??

Regards
Ajay Kasam

ajay kasam said...

Can we use this script for debugging native Android libraries running on Intel based chipsets??

Regards
Ajay Kasam

rxwen said...

Theoretically speaking, yes.
I haven't worked in intel based android before. But I believe it can work well as long as it can find the gdb and addr2line utilities in intel toolchain.
You can update find_addr2line and find_debugger functions in agdb.py to suit your own needs.

Android app development said...

This is one of the thoughtful and advance post.I like your blog quality.This is one of the great post.
Android app developers

Android developers said...

I acquire credible progression in every post. Your newer posts are artlessly admirable compared to your posts in the past. Keep up the adequate work.

Zenoreo said...

hi,
Is this possible to handle native crash log from a MIPS android device?

rx wen said...

Yes, theoritically you can use it for any platform, as long as you have the addr2line utility for the target platform.
The name of the addr2line is defined as "DEFAULT_ADDR2LINE_NAME = 'arm-eabi-addr2line'" in the script, change it to desired value.