mdb and adb on a Live System

Solaris's debugger environment allows us access to extremely detailed views of exactly what is happening inside the operating system. This level of detail is well beyond what most system administrators want or need. But every now and again we trip over a problem that requires a microscopic view of the system state. Some links to debugging sessions are provided at the bottom of the page.

mdb was introduced in Solaris 8 as a replacement for the adb and code>crash commands. It extends the adb command set, so it makes sense to look at the basic functionality of the two utilities side by side. For practical purposes, only mdb should be used for systems running Solaris 8+. In Solaris 10, adb is implemented as a link to mdb in adb compatibility mode.

This page has been updated to deal specifically with mdb, with some usage notes for adb. There may be some additional differences between the two that have not been noted. In general, anything that can be done on adb can be done in the same way on mdb, but the converse is not true.

mdb can look at the live kernel with the command:
mdb -k

adb can be run to analyze a live system with the command:
adb -k -P adb: /dev/ksyms /dev/mem
To tweak the kernel, the command is:
adb -kw -P adb: /dev/ksyms /dev/mem
(Here the -P option sets a prompt. Since adb does not have one by default, this can help reduce confusion.)

To look at a kernel crash dump, specify the name list and image by running:
mdb -k unix.# vmcore.#

Kernel Debuggers

The associated kernel debuggers, kadb and kmdb also use much of the same syntax. The old kadb must be loaded at boot time. To do this, from the ok> prompt:
ok> boot kadb -d
kadb: (hit the "return" key)
kadb[0]: kmem_flags/W 01
kadb[0]: :c
(Loading kadb this way means that kadb will only be effective for this current boot session.)

Fortunately, kmdb no longer requires this level of fiddling. In Solaris 10, it can be loaded by running
mdb -K
from the console. (Invoking kmdb this way activates the session at the same time. Note that when the kmdb session is active, the system is stopped.) In order to unload kmdb, it is necessary to run
mdb -U
or quit out of the debugger session with
::quit -u

kmdb can be entered at boottime like kadb. To boot into kmdb, perform the following:
ok> boot kmdb -d
0> kmem_flags/W 01
0> :c

Loading kmdb or kadb at boot time means that the system must be rebooted to unload it. Note that a system that crashes with the kernel debugger loaded returns to the kernel debugger prompt rather than the ok> prompt (unless nopanicdebug has been set to 1, either via kmdb or the /etc/system).

We can also set and manipulate breakpoints within our environment to drop us to the kernel debugger prompt when specified actions take place.

If we have the debugger loaded and have not set a breakpoint, we can enter the kernel debugger environment by sending a break (on a serial console), Stop-A or L1-A (on Sparc systems with a Sun keyboard) or F1-A (for x86/x64 systems with a local keyboard).

Not all mdb macros are accessible to kmdb; use $M to see a list of available macros.

Modular Debugger

Besides being able to analyze a core dump in the same way as adb, mdb is a modular debugger which allows the end user to create custom tools and commands to do almost anything.

The modularity of mdb is its main strength. Modules can allow us to look at programs in a number of contexts, both live and post-mortem. These tools are located in loadable modules which can be accessed via dlopen(). These modules are called dmods and include both dcmds (commands) and walkers. ("Walker" commands allow mdb to change the target to a different part of the program structure.)

mdb also has a number of good interactive features, including command line history, editing and logging. In addition, there is a syntax-checking facility and a built-in output pager.

Target Properties

The target is the object under inspection by mdb. It may be a core file, a live kernel, a crash dump, a user process, a data file, or an ELF object file.

The following types of target properties are available to be read and/or written by mdb:

• address spaces: Allows reading and writing data from the target's virtual address space.
• symbol table: Allows access to the symbol tables (both static and dynamic) of the target's primary executable file.
• external data: Read target's external data buffers.
• load objects: Objects can be loaded within mdb
• threads: Execution of threads can be controlled.

Dot

In mdb jargon, the current object is known as "dot" ("."). This represents an address in the memory space. Walkers are commands that allow us to shift our focus to another area of memory.

Process Analysis

We can look at a particular process by either running the command as an argument to mdb or by specifying the mdb -p PID command for the given process ID.

Command Syntax

A simple command has the following syntax:
[address] [,count] command
where a command is a verb followed by modifiers. Verbs include:

• ?: Examine code or variables in executable object file
• /: Examine value.
• =: Value printed in different formats.
• $<: Invoke miscellaneous commands, including macros.
• >: Assign value to variable or register.
• <: Read value from variable or register.

The address is usually provided by an expression. In mdb, an expression is one of the following:

• An integer: May be specified in binary (0i), hexidecimal (0x), or decimal ( 0t).
• 0tnatural-number.natural-number: Specifies a decimal floating-point number.
• 'string-of-characters': Generates an integer by converting the characters to ASCII equivalents.
• <identifier: The value of the indicated variable.
• identifier: The value of the indicated symbol.
• (expression): The value of the expression.
• .: The value of the current location.
• &: The value of the location most recently used to execute a dcmd.
• +: The incremented value of the current location.
• ^: The decremented value of the current location.

Registers

• %g0-%g7: General registers.
  g0=zero
  g7=address of current thread.
• %i0-%i7: Input registers.
  i6=Frame pointer (for tracing previous function through stack.)
• %o0=%o7: Output registers.
  o6=Stack pointer (sp)
  o7=program counter (pc).
• %l0-%l7: Local registers.

Variables

Variables are assigned using the or ::typeset dcmds. Variables may use non-reserved names consisting of sequences of letters, digits, underscores or periods. The value of a variable is a 64-bit unsigned integer.

The following variables are persistent:

• 0: Most recent value printed by \ / ? or =
• 9: Most recent count from $<
• b: Virtual address of the base of the data section.
• d: Size of the data section (bytes).
• e: Virtual address of entry point.
• hits: Number of times the event specifier has been matched.
• m: Magic number of target's primary object file.
• t: Size of text section (bytes).
• thread: Current representative thread's identifier.

Symbols

In an expression context, a symbol is evaluated, usually to the virtual address associated with the symbol. The symbol table can be examined by the ::nm dcmd and the ::nm -P dcmd (for the private symbol table). Symbols can be scoped via the backtick character: object`file`name evaluates to the address of name from file in the object kernel module.

Headers

The header files give information about the structures we will examine with adb. These files are located in /usr/include/* and /usr/platform/arch_name/include/*

General Commands

The following are the most commonly-used general commands. A full list can be obtained from the mdb prompt using the ::dcmds dcmd.

In the listings below, commands of the following forms are part of adb compatibility mode:

• :character
• $character

The commands have been grouped together based on the ways in which they are most commonly used:

• Control Commands
• Input & Output Commands
• Format Specifications
• System Examination
• Target Examination
• Tracing, Watchpoints & Breakpoints
• General Debugging

Control Commands
$< or $<<	Replace input with a macro or source a macro.
$>filename or ::log filename	Log session to a file. If no filename, use default.
|	Pipe. Allows simple commands to be joined.
!	Shell escape. Acts as a pipe to a shell command. (Not available in kmdb.)
//	Comment. Following words on the same line are ignored.
$M	Show built-in macros (Kernel debugger only).
$Pstring	Set prompt to string.
$Q ::quit	Quit. (From kmdb, use -u option to avoid exiting to ok> prompt.)
$W	Re-open target in writable mode.
$p	Change target context.
$w	Control output page width.
:A	Attach to a process or core file.
:R	Release attachment.
:k	Kill and release targets.
$v	Print non-zero variables.
> ::typeset	Assign a variable.
::dcmds	Print available commands.
::nm	Print symbol table. (-P specifies a private symbol table. Manipulated with ::nmadd and ::nmdelete
::help dcmd	Provide usage notes on a dcmd.
::typeset	Manipulate variable.
::walk	Walk data structure.
::walkers	List available walkers.

Input & Output Commands
$< $<<	Replace input with a macro or source a macro.
$>filename	Log session to a file. If no filename, use default.
address/format-spec /format-spec	Read the value in a memory address formatted as format-spec. If no address is provided, use dot.
address/W value	Write the value in the four bytes starting with address. If no address is provided, use dot. v, w or Z may also be used instead of W to write 1, 2 or 8 bytes, respectively.
address=format-spec =format-spec	Format immediate value of address or dot.
?	Read/write primary object file.
@format-spec	Read/write physical address as format-spec.
\format-spec	Read/write physical address as format-spec.

The difference between / and = is subtle. For example, to find the address holding the value of the maxphys symbol in decimal, we would run:
maxphys=D
To find the value inside the above address, we would use / like:
maxphys/D

Format Specification
Note that the ::formats dcmd prints out a full list of supported formats.
D	Display in signed decimal.
i	Display as a disassembled instruction.
U	Display in unsigned decimal.
X	Display in signed hexidecimal.
0txyz	Specifies xyz as a decimal value.

System Examination
cpu$<cpus	Display cpu0.
cpun$<cpu	Display cpu #n.
$<msgbuf	Display message buffer, including all console messages up to panic.
<sp$<stacktrace	Use the stack pointer address (sp) to display a detailed stack trace.
$r ::regs	Display general registers, including program counter and stack pointer.
::callout	Print callout table.
::cpuinfo -v	Information about activities of CPUs, including runqueue inhabitants.
::cpuregs ::cpuregs -c cpuid	Print CPU registers. kmdb only. Can specify a cpu.
::cpustack ::cpustack -c cpuid	Print CPU stack. kmdb only. Can specify a cpu.
::dnlc	Print DNLC contents.
::ipcs	Print SVR4 IPC information.
::kmalog	Display kernel memory log and stack traces.
::kmastat	Print current kernel memory allocations
::memstat	Print current memory usage.
::nm	Print symbol table. (-P specifies a private symbol table. Manipulated with ::nmadd and ::nmdelete
::ps	List processes with associated threads and lwps
::ptree	Print process tree.

Target Examination
$?	Print status and registers.
$C	Show call trace and arguments, saved frame pointer and saved program counter for each stack frame.
$X, $Y, $x, $y and ::fpregs	Display floating point registers.
$c	Display stack backtrace.
$e	Print list of global symbols.
$f	Print list of source files.
$l	Print representative thread's lwp ID.
$m	Print address space mappings.
$r ::regs	Display general registers, including program counter and stack pointer.
as::as2proc	Convert as pointer to a proc_t pointer.
::devbindings	devinfo nodes bound to device-name or major-num.
::devinfo	Detailed devinfo of node.
::devinfo2driver	Driver name for this node.
::devnames	Print devnames array.
::devt	Display dev_t's major & minor numbers.
::did2thread	Kernel thread for this ID.
::dumpaddress	Dump memory from address.
::findfalse	Find potentially falsely shared structures.
::findleaks	Search for potential kernel memory leaks.
::findlocks	Find locks held by specified thread.
threadp::findstack	Find kernel thread stack for associated thread.
::inode	Display summary of inode_t.
::kmsqid	Display message queue structure (kmsqid).
::ksemid	Display a semaphore structure (ksemid).
::kshmid	Display a shared memory structure (kshmid).
::pgrep pattern	Find proc_t pointers matching the pattern.
0tPID::pid2proc	Convert decimal PID to a proc_t pointer.
procp::ps	Process information matching the associated proc_t.
::status	Print summary of target status.
sobj::walk blocked	Walk threads blocked on a particular synchronization object (sobj).
procp::walk thread	Walk threads of associated process.
sobj::wchaninfo -v	Blocked on condition variables for a particular synchronization object (sobj).
address::whatis	Attempts to identify address contents.
vnode::whereopen	Processes with vnode open.

Tracing, Watchpoints and Breakpoints
(Breakpoints for kernel debugger only.)
$b	Show all breakpoints.
$i	Print list of ignored signals.
:a	Set a watchpoint.
:b	Set a breakpoint.
:c or ::cont	Continue target execution.
:d	Delete a breakpoint.
:e	Step over next instruction.
:i	Step over next instruction.
:k	Kill and release targets.
:p	Set execute access watchpoints.
:r	Run new target process.
:s	Step target to next instruction.
:t	Stop on delivery of specified signals.
:u	Step out of current function.
:w	Set write access watchpoint.
:z	Delete all breakpoints.

General Debugging
$G	Toggle C++ demangling.
$V	Toggle disassembly mode.
$g	Toggle C++ demangling.
address::dis	Disassemble text starting at address.

Comparison Operators
==	Logical equality.
!=	Logical inequality.
&	Bitwise AND.
|	Bitwise OR.
^	Bitwise XOR.

Usage Examples

There are several usage examples available on the web. Here are a few:

• Eric Schrock debugs an x86 race condition
• Stacey Marshall debugs a segmentation fault (with very helpful comments)
• C Omand debugs a Subversion problem.