Monday, April 22, 2013

mdb and adb on a Live System

Solaris's debugger environment allows us access to extremely detailed views of exactly what is happening inside the operating system. This level of detail is well beyond what most system administrators want or need. But every now and again we trip over a problem that requires a microscopic view of the system state. Some links to debugging sessions are provided at the bottom of the page.

mdb was introduced in Solaris 8 as a replacement for the adb and code>crash commands. It extends the adb command set, so it makes sense to look at the basic functionality of the two utilities side by side. For practical purposes, only mdb should be used for systems running Solaris 8+. In Solaris 10, adb is implemented as a link to mdb in adb compatibility mode.

This page has been updated to deal specifically with mdb, with some usage notes for adb. There may be some additional differences between the two that have not been noted. In general, anything that can be done on adb can be done in the same way on mdb, but the converse is not true.

mdb can look at the live kernel with the command:
mdb -k

adb can be run to analyze a live system with the command:
adb -k -P adb: /dev/ksyms /dev/mem
To tweak the kernel, the command is:
adb -kw -P adb: /dev/ksyms /dev/mem
(Here the -P option sets a prompt. Since adb does not have one by default, this can help reduce confusion.)

To look at a kernel crash dump, specify the name list and image by running:
mdb -k unix.# vmcore.#

Kernel Debuggers

The associated kernel debuggers, kadb and kmdb also use much of the same syntax. The old kadb must be loaded at boot time. To do this, from the ok> prompt:
ok> boot kadb -d
kadb: (hit the "return" key)
kadb[0]: kmem_flags/W 01
kadb[0]: :c

(Loading kadb this way means that kadb will only be effective for this current boot session.)

Fortunately, kmdb no longer requires this level of fiddling. In Solaris 10, it can be loaded by running
mdb -K
from the console. (Invoking kmdb this way activates the session at the same time. Note that when the kmdb session is active, the system is stopped.) In order to unload kmdb, it is necessary to run
mdb -U
or quit out of the debugger session with
::quit -u

kmdb can be entered at boottime like kadb. To boot into kmdb, perform the following:
ok> boot kmdb -d
0> kmem_flags/W 01
0> :c

Loading kmdb or kadb at boot time means that the system must be rebooted to unload it. Note that a system that crashes with the kernel debugger loaded returns to the kernel debugger prompt rather than the ok> prompt (unless nopanicdebug has been set to 1, either via kmdb or the /etc/system).

We can also set and manipulate breakpoints within our environment to drop us to the kernel debugger prompt when specified actions take place.

If we have the debugger loaded and have not set a breakpoint, we can enter the kernel debugger environment by sending a break (on a serial console), Stop-A or L1-A (on Sparc systems with a Sun keyboard) or F1-A (for x86/x64 systems with a local keyboard).

Not all mdb macros are accessible to kmdb; use $M to see a list of available macros.

Modular Debugger

Besides being able to analyze a core dump in the same way as adb, mdb is a modular debugger which allows the end user to create custom tools and commands to do almost anything.

The modularity of mdb is its main strength. Modules can allow us to look at programs in a number of contexts, both live and post-mortem. These tools are located in loadable modules which can be accessed via dlopen(). These modules are called dmods and include both dcmds (commands) and walkers. ("Walker" commands allow mdb to change the target to a different part of the program structure.)

mdb also has a number of good interactive features, including command line history, editing and logging. In addition, there is a syntax-checking facility and a built-in output pager.

Target Properties

The target is the object under inspection by mdb. It may be a core file, a live kernel, a crash dump, a user process, a data file, or an ELF object file.

The following types of target properties are available to be read and/or written by mdb:

  • address spaces: Allows reading and writing data from the target's virtual address space.
  • symbol table: Allows access to the symbol tables (both static and dynamic) of the target's primary executable file.
  • external data: Read target's external data buffers.
  • load objects: Objects can be loaded within mdb
  • threads: Execution of threads can be controlled.


In mdb jargon, the current object is known as "dot" ("."). This represents an address in the memory space. Walkers are commands that allow us to shift our focus to another area of memory.

Process Analysis

We can look at a particular process by either running the command as an argument to mdb or by specifying the mdb -p PID command for the given process ID.

Command Syntax

A simple command has the following syntax:
[address] [,count] command
where a command is a verb followed by modifiers. Verbs include:

  • ?: Examine code or variables in executable object file
  • /: Examine value.
  • =: Value printed in different formats.
  • $<: Invoke miscellaneous commands, including macros.
  • >: Assign value to variable or register.
  • <: Read value from variable or register.

The address is usually provided by an expression. In mdb, an expression is one of the following:

  • An integer: May be specified in binary (0i), hexidecimal (0x), or decimal ( 0t).
  • 0tnatural-number.natural-number: Specifies a decimal floating-point number.
  • 'string-of-characters': Generates an integer by converting the characters to ASCII equivalents.
  • <identifier: The value of the indicated variable.
  • identifier: The value of the indicated symbol.
  • (expression): The value of the expression.
  • .: The value of the current location.
  • &: The value of the location most recently used to execute a dcmd.
  • +: The incremented value of the current location.
  • ^: The decremented value of the current location.


  • %g0-%g7: General registers.
    g7=address of current thread.
  • %i0-%i7: Input registers.
    i6=Frame pointer (for tracing previous function through stack.)
  • %o0=%o7: Output registers.
    o6=Stack pointer (sp)
    o7=program counter (pc).
  • %l0-%l7: Local registers.


Variables are assigned using the or ::typeset dcmds. Variables may use non-reserved names consisting of sequences of letters, digits, underscores or periods. The value of a variable is a 64-bit unsigned integer.

The following variables are persistent:

  • 0: Most recent value printed by \ / ? or =
  • 9: Most recent count from $<
  • b: Virtual address of the base of the data section.
  • d: Size of the data section (bytes).
  • e: Virtual address of entry point.
  • hits: Number of times the event specifier has been matched.
  • m: Magic number of target's primary object file.
  • t: Size of text section (bytes).
  • thread: Current representative thread's identifier.


In an expression context, a symbol is evaluated, usually to the virtual address associated with the symbol. The symbol table can be examined by the ::nm dcmd and the ::nm -P dcmd (for the private symbol table). Symbols can be scoped via the backtick character: object`file`name evaluates to the address of name from file in the object kernel module.


The header files give information about the structures we will examine with adb. These files are located in /usr/include/* and /usr/platform/arch_name/include/*

General Commands

The following are the most commonly-used general commands. A full list can be obtained from the mdb prompt using the ::dcmds dcmd.

In the listings below, commands of the following forms are part of adb compatibility mode:

  • :character
  • $character

The commands have been grouped together based on the ways in which they are most commonly used:

Control Commands
$< or $<< Replace input with a macro or source a macro.
$>filename or
::log filename
Log session to a file. If no filename, use default.
| Pipe. Allows simple commands to be joined.
! Shell escape. Acts as a pipe to a shell command. (Not available in kmdb.)
// Comment. Following words on the same line are ignored.
$M Show built-in macros (Kernel debugger only).
$Pstring Set prompt to string.
Quit. (From kmdb, use -u option to avoid exiting to ok> prompt.)
$W Re-open target in writable mode.
$p Change target context.
$w Control output page width.
:A Attach to a process or core file.
:R Release attachment.
:k Kill and release targets.
$v Print non-zero variables.
Assign a variable.
::dcmds Print available commands.
::nm Print symbol table. (-P specifies a private symbol table. Manipulated with ::nmadd and ::nmdelete
::help dcmd Provide usage notes on a dcmd.
::typeset Manipulate variable.
::walk Walk data structure.
::walkers List available walkers.

Input & Output Commands
Replace input with a macro or source a macro.
$>filename Log session to a file. If no filename, use default.
Read the value in a memory address formatted as format-spec. If no address is provided, use dot.
address/W value Write the value in the four bytes starting with address. If no address is provided, use dot. v, w or Z may also be used instead of W to write 1, 2 or 8 bytes, respectively.
Format immediate value of address or dot.
? Read/write primary object file.
@format-spec Read/write physical address as format-spec.
\format-spec Read/write physical address as format-spec.
The difference between / and = is subtle. For example, to find the address holding the value of the maxphys symbol in decimal, we would run:
To find the value inside the above address, we would use / like:

Format Specification
Note that the ::formats dcmd prints out a full list of supported formats.
D Display in signed decimal.
i Display as a disassembled instruction.
U Display in unsigned decimal.
X Display in signed hexidecimal.
0txyz Specifies xyz as a decimal value.

System Examination
cpu$<cpus Display cpu0.
cpun$<cpu Display cpu #n.
$<msgbuf Display message buffer, including all console messages up to panic.
<sp$<stacktrace Use the stack pointer address (sp) to display a detailed stack trace.
Display general registers, including program counter and stack pointer.
::callout Print callout table.
::cpuinfo -v Information about activities of CPUs, including runqueue inhabitants.
::cpuregs -c cpuid
Print CPU registers. kmdb only. Can specify a cpu.
::cpustack -c cpuid
Print CPU stack. kmdb only. Can specify a cpu.
::dnlc Print DNLC contents.
::ipcs Print SVR4 IPC information.
::kmalog Display kernel memory log and stack traces.
::kmastat Print current kernel memory allocations
::memstat Print current memory usage.
::nm Print symbol table. (-P specifies a private symbol table. Manipulated with ::nmadd and ::nmdelete
::ps List processes with associated threads and lwps
::ptree Print process tree.

Target Examination
$? Print status and registers.
$C Show call trace and arguments, saved frame pointer and saved program counter for each stack frame.
$X, $Y, $x, $y and ::fpregs Display floating point registers.
$c Display stack backtrace.
$e Print list of global symbols.
$f Print list of source files.
$l Print representative thread's lwp ID.
$m Print address space mappings.
Display general registers, including program counter and stack pointer.
as::as2proc Convert as pointer to a proc_t pointer.
::devbindings devinfo nodes bound to device-name or major-num.
::devinfo Detailed devinfo of node.
::devinfo2driver Driver name for this node.
::devnames Print devnames array.
::devt Display dev_t's major & minor numbers.
::did2thread Kernel thread for this ID.
::dumpaddress Dump memory from address.
::findfalse Find potentially falsely shared structures.
::findleaks Search for potential kernel memory leaks.
::findlocks Find locks held by specified thread.
threadp::findstack Find kernel thread stack for associated thread.
::inode Display summary of inode_t.
::kmsqid Display message queue structure (kmsqid).
::ksemid Display a semaphore structure (ksemid).
::kshmid Display a shared memory structure (kshmid).
::pgrep pattern Find proc_t pointers matching the pattern.
0tPID::pid2proc Convert decimal PID to a proc_t pointer.
procp::ps Process information matching the associated proc_t.
::status Print summary of target status.
sobj::walk blocked Walk threads blocked on a particular synchronization object (sobj).
procp::walk thread Walk threads of associated process.
sobj::wchaninfo -v Blocked on condition variables for a particular synchronization object (sobj).
address::whatis Attempts to identify address contents.
vnode::whereopen Processes with vnode open.

Tracing, Watchpoints and Breakpoints
(Breakpoints for kernel debugger only.)
$b Show all breakpoints.
$i Print list of ignored signals.
:a Set a watchpoint.
:b Set a breakpoint.
:c or ::cont Continue target execution.
:d Delete a breakpoint.
:e Step over next instruction.
:i Step over next instruction.
:k Kill and release targets.
:p Set execute access watchpoints.
:r Run new target process.
:s Step target to next instruction.
:t Stop on delivery of specified signals.
:u Step out of current function.
:w Set write access watchpoint.
:z Delete all breakpoints.

General Debugging
$G Toggle C++ demangling.
$V Toggle disassembly mode.
$g Toggle C++ demangling.
address::dis Disassemble text starting at address.

Comparison Operators
== Logical equality.
!= Logical inequality.
& Bitwise AND.
| Bitwise OR.
^ Bitwise XOR.

Usage Examples

There are several usage examples available on the web. Here are a few:

No comments: