Debugging Mismatched Core Files

Related Topics:

Core File Debugging with dbx

core_lo_pathmap Environment Variable

dbx has limited support for the debugging of "mismatched" core files (for example, core files produced on a system running a different version or patch level of the Solaris^TM operating environment).

Sometimes a core file is created on one system (the core-host) and you want to load the core file on another machine (the dbx-host) to debug it. However, two problems with libraries may arise when you do so:

• The shared libraries used by the program on the core-host won't be the same libraries as those on the dbx-host. To get proper stack traces involving the libraries, you'll want to make these original libraries available on the dbx-host.
• dbx uses system libraries in /usr/lib to help understand the implementation details of the run time linker and threads library on the system. It may also be necessary to provide these system libraries from the core-host so that dbx can understand the runtime linker data structures and the threads data structures.

The user libraries and system libraries can change in patches as well as major Solaris upgrades, so this problem can even occur on the same host, if, for example, a patch was installed after the core file was collected, but before running dbx on the core file.

dbx may display one or more of the following error messages when you load a "mismatched" core file:

dbx: core file read error: address 0xff3dd1bc not available

dbx: warning: could not initialize librtld_db.so.1 -- trying 
libDP_rtld_db.so

dbx: cannot get thread info for 1 -- generic libthread_db.so error

dbx: attempt to fetch registers failed - stack corrupted

dbx: read of registers from (0xff363430) failed -- debugger service 
failed

To eliminate the library problems and debug a "mismatched" core file with dbx, you can now do the following:

1. Set the dbx environment variable core_lo_pathmap to on.

2. Use the pathmap command to tell dbx where the correct libraries for the core file are located.

3. Use the debug command to load the program and the core file.

For example, assuming that the root partition of the core-host has been exported over NFS and can be accessed via /net/core-host/ on the dbx-host machine, you would use the following commands to load the program prog and the core file prog.core for debugging:

(dbx) dbxenv core_lo_pathmap on

(dbx) pathmap /usr /net/core-host/usr

(dbx) pathmap /appstuff /net/core-host/appstuff

(dbx) debug prog prog.core

If you are not exporting the root partition of the core-host, you must copy the libraries by hand. You need not re-create the symbolic links. (For example, you need not make a link from libc.so to libc.so.1; just make sure libc.so.1 is available.)

Things to remember:

• The pathmap command does not recognize a pathmap for '/' so you cannot use the following command:

pathmap / /net/core-host

• The single-argument mode for the pathmap command does not work with loadobject pathnames, so use the two argument from-path to-path mode.
• Debugging the core file is likely to work better if the dbx-host has either the same or a more recent version of Solaris than the core-host, though this is not always necessary.
• The system libraries that you might need are:

• For the runtime linker:

  /usr/lib/librtld_db.so.1
  /usr/lib/sparcv9/librtld_db.so.1

• For the threads library, depending on which implementation of libthread you are using:

  /usr/lib/libthread_db.so.1
  /usr/lib/sparcv9/libthread_db.so.1
  /usr/lib/lwp/libthread_db.so.1
  /usr/lib/lwp/sparcv9/libthread_db.so.1

  You will need the SPARC V9 versions if dbx is running on Solaris 7 or 8 (if the command isalist displays sparcv9) since these system libraries are loaded and used as part of dbx, not as part of the target program.

• If you are looking at a core file from a threaded program, andthe where commanddoes not display a stack, try using lwp commands. For example:.

(dbx) where

current thread: t@0

[1] 0x0(), at 0xffffffff

(dbx) lwps

o>l@1 signal SIGSEGV in _sigfillset()

(dbx) lwp l@1

(dbx) where

=>[1] _sigfillset(), line 2 in "lo.c"

[2] _liblwp_init(0xff36291c, 0xff2f9740, ...

[3] _init(0x0, 0xff3e2658, 0x1, ...

...

The lack of a thread stack can indicate a problem with thread_db.so.1 Therefore, you might also want to try copying the proper libthread_db.so.1 library from the core-host