Troubleshooting Intermittent Problems

Intermittent problems are extremely difficult to troubleshoot. Any reproducible problem can be troubleshot, if for no other reason than that each individual component can be proven to not be the problem through experimentation. Problems that are not reproducible cannot be approached in the same way.

Problems present as intermittent for one of two reasons:

1. We have not identified the real cause of the problem.
2. The problem is being caused by failing or flaky hardware.

The first possibility should be addressed by going back to brainstorming hypotheses.

It may be helpful to bring a fresh perspective into the brainstorming session, either by bringing in different people, or by sleeping on the problem.

The second problem is tougher. There are hardware diagnostics tests that can be run to try to identify the failing piece of hardware.

The first thing to do is to perform general maintenance on the system. Re-seat memory chips, processors, expansion boards and hard drives.

Once general maintenance has been performed, test suites like SunVTS can perform stress-testing on a system to try to trigger the failure and identify the failing part.

It may be the case, however, that the costs associated with this level of troubleshooting are prohibitive. In this case, we may want to attempt to shotgun the problem.

Shotgunning is the practice of replacing potentially failing parts without having identified them as actually being flaky. In general, parts are replaced by price point, with the cheapest parts being replaced first.

Though we are likely to inadvertently replace working parts, the cost of the replacement may be cheaper than the costs of the alternatives (like the downtime cost associated with stress testing).

When parts are removed during shotgunning, it is important to discard them rather than keep them as spares. Any part you remove as part of a troubleshooting exercise is questionable. (After all, what if a power surge caused multiple parts to fail? Or what if there was a cascading failure?) It does not make sense to have questionable parts in inventory; such parts would be useless for troubleshooting, and putting questionable parts into service just generates additional downtime down the road.

This practice may violate your service contract if performed without the knowledge and consent of your service provider.

Regardless of the method used to deal with intermittent problems, it is essential to keep good records. Relationships between our problem and other events may only become clear when we look at patterns over time. We may only be confident that we have really resolved the problem if we can demonstrate that we've gone well beyond the usual re-occurrence frequency without the problem re-emerging.

Troubleshooting Hard Drive Connectivity

Disk drive connectivity problems on a Solaris 2.x systems can be caused by software, hardware or PROM configuration problems.

Software Problems

New devices may require that the appropriate /dev and /devices files be created. This can be done through use of the drvconfig and disks commands, but it is usually done by performing a boot -r from the ok> prompt.

Once the system is back, the root user should be able to run format and see the disk listed as available. The disk can then be partitioned and labelled with format and have filesystems created with newfs or mkfs, as appropriate.

The presence of the appropriate /dev and /devices files can be verified by running the commands ls -lL /dev/dsk and ls -lL /dev/rdsk and making sure that they are block and character special files respectively, with major numbers depending on the driver used. (See the ls man page if you are not sure what this means.)

Files that can cause problems with hard drive connectivity include:

/dev/dsk/c#t#d#s# or /dev/rdsk/c#t#d#s# and related /devices files
/etc/name_to_major
/etc/minor_perm

Problems with the /dev and /devices files can be corrected directly by removing the offending files and recreating them, either directly with mknod and ln -s or indirectly with drvconfig , disks or boot -r (as appropriate).

Hardware Problems

The most common sources of hard drive connectivity problems (once the device files are built) are loose cables and terminators. Check these first before proceeding.

The system SCSI buses can be probed at the ok> prompt. To set this up, perform the following:

ok> setenv auto-boot? false
ok> reset
ok> probe-scsi-all
(after output)
ok> setenv auto-boot? true
(if appropriate)

This will give a hardware mapping of all SCSI devices on the system. If the hard drive in question does not appear, you have either a hardware problem or a PROM search path problem. To check for the PROM search path problem, run the following:

ok> printenv

Look for the pcia-probe-list or sbus-probe-default parameters and make sure that they are set to the default for your system.

Some additional hardware diagnostics are available at the PROM monitor (ok>) prompt. Additional information may come from navigating the PROM device tree at the ok> prompt.

Sun CD ROM Troubleshooting

Many CD ROM problems are software rather than hardware problems. If your problem is hardware-related, you can look at our Hardware Diagnostics page. Our page on Hard Drive Connectivity may also be useful by analogy.

Some Sun patches resolve security holes in vold by commenting devices out of the /etc/rmmount.conf file. While this is effective, it also disables the devices that are commented out.

For vold to work with your CD ROM and floppy drives within CDE in the designed fashion, you should uncomment the following lines in your /etc/rmmount.conf:

ident hsfs ident_hsfs.so cdrom
ident ufs ident_ufs.so floppy cdrom pcmem
ident pcfs ident_pcfs.so floppy pcmem
action cdrom action_filemgr.so
action floppy action_filemgr.so

Note that this will allow mounted CDs and floppies to be mounted with SUID permissions. For systems where this is inappropriate, your /etc/rmmount.conf file should also contain the following:

mount * ufs -o nosuid
mount * hsfs -o nosuid

(Note that pcfs does not understand SUID, so pcfs does not pose a risk in the same way that hsfs and ufs do.)

The Solaris Volume Manager documentation contains substantial additional information about the operation of vold.

ZFS Management

ZFS was first publicly released in the 6/2006 distribution of Solaris 10. Previous versions of Solaris 10 did not include ZFS.

ZFS is flexible, scalable and reliable. It is a POSIX-compliant filesystem with several important features:

• integrated storage pool management
• data protection and consistency, including RAID
• integrated management for mounts and NFS sharing
• scrubbing and data integrity protection
• snapshots and clones
• advanced backup and restore features
• excellent scalability
• built-in compression
• maintenance and troubleshooting capabilities
• automatic sharing of disk space and I/O bandwidth across disk devices in a pool
• endian neutrality

No separate filesystem creation step is required. The mount of the filesystem is automatic and does not require vfstab maintenance. Mounts are controlled via the mountpoint attribute of each file system.

Pool Management

Members of a storage pool may either be hard drives or slices of at least 128MB in size.

To create a mirrored pool:
zpool create -f pool-name mirror c#t#d# c#t#d#
To check a pool's status, run:
zpool status -v pool-name
To list existing pools:
zpool list
To remove a pool and free its resources:
zpool destroy pool-name
A destroyed pool can sometimes be recovered as follows:
zpool import -D

Additional disks can be added to an existing pool. When this happens in a mirrored or RAID Z pool, the ZFS is resilvered to redistribute the data. To add storage to an existing mirrored pool:
zpool add -f pool-name mirror c#t#d# c#t#d#

Pools can be exported and imported to transfer them between hosts.
zpool export pool-name
zpool import pool-name
Without a specified pool, the import command lists available pools. zpool import

To clear a pool's error count, run:
zpool clear pool-name

Although virtual volumes (such as those from DiskSuite or VxVM) can be used as base devices, it is not recommended for performance reasons.

Filesystem Management

Similar filesystems should be grouped together in hierarchies to make management easier. Naming schemes should be thought out as well to make it easier to group administrative commands for similarly managed filesystems.

When a new pool is created, a new filesystem is mounted at /pool-name.

To create another filesystem:
zfs create pool-name/fs-name
To delete a filesystem:
zfs destroy filesystem-name

To rename a ZFS filesystem:
zfs rename old-name new-name

Properties are set via the zfs set command.
To turn on compression:
zfs set compression=on pool-name/filesystem-name
To share the filesystem via NFS:
zfs set sharenfs=on pool-name/fs-name
zfs set sharenfs="mount-options " pool-name/fs-name
Rather than editing the /etc/vfstab:
zfs set mountpoint= mountpoint-name pool-name/filesystem-name

Quotas are also set via the same command:
zfs set quota=#gigG pool-name/filesystem-name

RAID Levels

ZFS filesystems automatically stripe across all top-level disk devices. (Mirrors and RAID-Z devices are considered to be top-level devices.) It is not recommended that RAID types be mixed in a pool. (zpool tries to prevent this, but it can be forced with the -f flag.)

The following RAID levels are supported:

• RAID-0 (striping)
• RAID-1 (mirror)
• RAID-Z (similar to RAID 5, but with variable-width stripes to avoid the RAID 5 write hole)
• RAID-Z2

The zfs man page recommends 3-9 disks for RAID-Z pools.

Performance Monitoring

ZFS performance management is handled differently than with older generation file systems. In ZFS, I/Os are scheduled similarly to how jobs are scheduled on CPUs. The ZFS I/O scheduler tracks a priority and a deadline for each I/O. Within each deadline group, the I/Os are scheduled in order of logical block address.

Writes are assigned lower priorities than reads, which can help to avoid traffic jams where reads are unable to be serviced because they are queued behind writes. (If a read is issued for a write that is still underway, the read will be executed against the in-memory image and will not hit the hard drive.)

In addition to scheduling, ZFS attempts to intelligently prefetch information into memory. The algorithm tries to pick information that is likely to be needed. Any forward or backward linear access patterns are picked up and used to perform the prefetch.

The zpool iostat command can monitor performance on ZFS objects:

• USED CAPACITY: Data currently stored
• AVAILABLE CAPACITY: Space available
• READ OPERATIONS: Number of operations
• WRITE OPERATIONS: Number of operations
• READ BANDWIDTH: Bandwidth of all read operations
• WRITE BANDWIDTH: Bandwidth of all write operations

The health of an object can be monitored with
zpool status

Snapshots and Clones

To create a snapshot:
zfs snapshot pool-name/filesystem-name@ snapshot-name
To clone a snapshot:
zfs clone snapshot-name filesystem-name
To roll back to a snapshot:
zfs rollback pool-name/filesystem-name@snapshot-name

zfs send and zfs receive allow clones of filesystems to be sent to a development environment.

The difference between a snapshot and a clone is that a clone is a writable, mountable copy of the file system. This capability allows us to store multiple copies of mostly-shared data in a very space-efficient way.

Each snapshot is accessible through the .zfs/snapshot in the /pool-name directory. This can allow end users to recover their files without system administrator intervention.

Zones

If the filesystem is created in the global zone and added to the local zone via zonecfg, it may be assigned to more than one zone unless the mountpoint is set to legacy.
zfs set mountpoint=legacy pool-name/filesystem-name

To import a ZFS filesystem within a zone:
zonecfg -z zone-name

add fs
set dir=mount-point
set special=pool-name/filesystem-name
set type=zfs
end
verify
commit
exit

Administrative rights for a filesystem can be granted to a local zone:
zonecfg -z zone-name

add dataset
set name=pool-name/filesystem-name
end
commit exit

Data Protection

ZFS is a transactional file system. Data consistency is protected via Copy-On-Write (COW). For each write request, a copy is made of the specified block. All changes are made to the copy. When the write is complete, all pointers are changed to point to the new block.

Checksums are used to validate data during reads and writes. The checksum algorithm is user-selectable. Checksumming and data recovery is done at a filesystem level; it is not visible to applications. If a block becomes corrupted on a pool protected by mirroring or RAID, ZFS will identify the correct data value and fix the corrupted value.

Raid protections are also part of ZFS.

Scrubbing is an additional type of data protection available on ZFS. This is a mechanism that performs regular validation of all data. Manual scrubbing can be performed by:
zpool scrub pool-name
The results can be viewed via:
zpool status
Any issues should be cleared with:
zpool clear pool-name

The scrubbing operation walks through the pool metadata to read each copy of each block. Each copy is validated against its checksum and corrected if it has become corrupted.

Hardware Maintenance

To replace a hard drive with another device, run:
zpool replace pool-name old-disk new-disk

To offline a failing drive, run:
zpool offline pool-name disk-name
(A -t flag allows the disk to come back online after a reboot.)

Once the drive has been physically replaced, run the replace command against the device:
zpool replace pool-name device-name
After an offlined drive has been replaced, it can be brought back online:
zpool online pool-name disk-name

Firmware upgrades may cause the disk device ID to change. ZFS should be able to update the device ID automatically, assuming that the disk was not physically moved during the update. If necessary, the pool can be exported and re-imported to update the device IDs.

Troubleshooting ZFS

The three categories of errors experienced by ZFS are:

• missing devices: Missing devices placed in a "faulted" state.
• damaged devices: Caused by things like transient errors from the disk or controller, driver bugs or accidental overwrites (usually on misconfigured devices).
• data corruption: Data damage to top-level devices; usually requires a restore. Since ZFS is transactional, this only happens as a result of driver bugs, hardware failure or filesystem misconfiguration.

It is important to check for all three categories of errors. One type of problem is often connected to a problem from a different family. Fixing a single problem is usually not sufficient.

Data integrity can be checked by running a manual scrubbing:
zpool scrub pool-name
zpool status -v pool-name
checks the status after the scrubbing is complete.

The status command also reports on recovery suggestions for any errors it finds. These are reported in the action section. To diagnose a problem, use the output of the status command and the fmd messages in /var/adm/messages.

The config section of the status section reports the state of each device. The state can be:

• ONLINE: Normal
• FAULTED: Missing, damaged, or mis-seated device
• DEGRADED: Device being resilvered
• UNAVAILABLE: Device cannot be opened
• OFFLINE: Administrative action

The status command also reports READ, WRITE or CHKSUM errors.

To check if any problem pools exist, use
zpool status -x
This command only reports problem pools.

If a ZFS configuration becomes damaged, it can be fixed by running export and import.

Devices can fail for any of several reasons:

• "Bit rot:" Corruption caused by random environmental effects.
• Misdirected Reads/Writes: Firmware or hardware faults cause reads or writes to be addressed to the wrong part of the disk.
• Administrative Error
• Intermittent, Sporadic or Temporary Outages: Caused by flaky hardware or administrator error.
• Device Offline: Usually caused by administrative action.

Once the problems have been fixed, transient errors should be cleared:
zpool clear pool-name

In the event of a panic-reboot loop caused by a ZFS software bug, the system can be instructed to boot without the ZFS filesystems:
boot -m milestone=none
When the system is up, remount / as rw and remove the file /etc/zfs/zpool.cache. The remainder of the boot can proceed with the
svcadm milestone all command. At that point import the good pools. The damaged pools may need to be re-initialized.

Scalability

The filesystem is 128-bit. 256 quadrillion zetabytes of information is addressable. Directories can have up to 256 trillion entries. No limit exists on the number of filesystems or files within a filesystem.

ZFS Recommendations

Because ZFS uses kernel addressable memory, we need to make sure to allow enough system resources to take advantage of its capabilities. We should run on a system with a 64-bit kernel, at least 1GB of physical memory, and adequate swap space.

While slices are supported for creating storage pools, their performance will not be adequate for production uses.

Mirrored configurations should be set up across multiple controllers where possible to maximize performance and redundancy.

Scrubbing should be scheduled on a regular basis to identify problems before they become serious.

When latency or other requirements are important, it makes sense to separate them onto different pools with distinct hard drives. For example, database log files should be on separate pools from the data files.

Root pools are not yet supported in the Solaris 10 6/2006 release, though they are anticipated in a future release. When they are used, it is best to put them on separate pools from the other filesystems.

On filesystems with many file creations and deletions, utilization should be kept under 80% to protect performance.

The recordsize parameter can be tuned on ZFS filesystems. When it is changed, it only affects new files. zfs set recordsize=size tuning can help where large files (like database files) are accessed via small, random reads and writes. The default is 128KB; it can be set to any power of two between 512B and 128KB. Where the database uses a fixed block or record size, the recordsize should be set to match. This should only be done for the filesystems actually containing heavily-used database files.

In general, recordsize should be reduced when iostat regularly shows a throughput near the maximum for the I/O channel. As with any tuning, make a minimal change to a working system, monitor it for long enough to understand the impact of the change, and repeat the process if the improvement was not good enough or reverse it if the effects were bad.

The ZFS Evil Tuning Guide contains a number of tuning methods that may or may not be appropriate to a particular installation. As the document suggests, these tuning mechanisms will have to be used carefully, since they are not appropriate to all installations.

For example, the Evil Tuning Guide provides instructions for:

• Turning off file system checksums to reduce CPU usage. This is done on a per-file system basis:
  zfs set checksum=off filesystem

zfs set checksum='on | fletcher2 | fletcher4 | sha256' filesystem
• Limiting the ARC size by setting
  set zfs:zfs_arc_max
  in /etc/system on 8/07 and later.
• If the I/O includes multiple small reads, the file prefetch can be turned off by setting
  zfs:zfs_prefetch_disable
  on 8/07 and later.
• If the I/O channel becomes saturated, the device level prefetch can be turned off with
  set zfs:zfs_vdev_cache_bshift = 13
  in /etc/system for 8/07 and later
• I/O concurrency can be tuned by setting
  set zfs:zfs_vdev_max_pending = 10
  in /etc/system in 8/07 and later.
• If storage with an NVRAM cache is used, cache flushes may be disabled with
  set zfs:zfs_nocacheflush = 1
  in /etc/system for 11/06 and later.
• ZIL intent logging can be disabled. (WARNING: Don't do this.)
• Metadata compression can be disabled. (Read this section of the Evil Tuning Guide first-- you probably do not need to do this.)

Sun Cluster Integration

ZFS can be used as a failover-only file system with Sun Cluster installations.

If it is deployed on disks also used by Sun Cluster, do not deploy it on any Sun Cluster quorum disks. (A ZFS-owned disk may be promoted to be a quorum disk on current Sun Cluster versions, but adding a disk to a ZFS pool may result in quorum keys being overwritten.)

ZFS Internals

Max Bruning wrote an excellent paper on how to examine the internals of a ZFS data structure. (Look for the article on the ZFS On-Disk Data Walk.) The structure is defined in ZFS On-Disk Specification.

Some key structures:

• uberblock_t: The starting point when examining a ZFS file system. 128k array of 1k uberblock_t structures, starting at 0x20000 bytes within a vdev label. Defined in uts/common/fs/zfs/sys/uberblock_impl.h Only one uberblock is active at a time; the active uberblock can be found with
  zdb -uuu zpool-name
• blkptr_t: Locates, describes, and verifies blocks on a disk. Defined in uts/common/fs/zfs/sys/spa.h.
• dnode_phys_t: Describes an object. Defined by uts/common/fs/zfs/sys/dmu.h
• objset_phys_t: Describes a group of objects. Defined by uts/common/fs/zfs/sys/dmu_objset.h
• ZAP Objects: Blocks containing name/value pair attributes. ZAP stands for ZFS Attribute Processor. Defined by uts/common/fs/zfs/sys/zap_leaf.h
• Bonus Buffer Objects:
  • dsl_dir_phys_t: Contained in a DSL directory dnode_phys_t; contains object ID for a DSL dataset dnode_phys_t
  • dsl_dataset_phys_t: Contained in a DSL dataset dnode_phys_t; contains a blkprt_t pointing indirectly at a second array of dnode_phys_t for objects within a ZFS file system.
  • znode_phys_t: In the bonus buffer of dnode_phys_t structures for files and directories; contains attributes of the file or directory. Similar to a UFS inode in a ZFS context.

Introduction to Solaris 10 Zones

Zones are containers to segregate services so that they do not interfere with each other. One zone, the global zone, is the locus for system-wide administrative functions. Non-global zones are not able to interact with each other except through network interfaces. When using management commands that reference PIDs, only processes in the same zone will be visible from any non-global zone.

Zones requiring network connectivity have at least one dedicated IP address. Non-global zones cannot observe each other's network traffic. Users in the global zone, however, are able to observe the functioning of processes in non-global zones. It is usually good practice to limit user access to the global zone to system administrators. Other processes and users should be assigned to a non-global zone.

Each zone is assigned a zone name and a unique numeric zone ID. The global zone always has the name "global" and ID "0." A node name is also assigned to each zone, including global. The node names are independent of the zone names.

Each zone has a path to its root directory relative to the global zone's root directory.

A non-global zone's scheduling class is set to be the same as the system's scheduling class. If a zone is assigned to a resource pool, its scheduling class can be controlled by controlling the pool's scheduling class.

Non-global zones can have their own zone administrators. Their authority is limited to their home zone.

The separation of the environments allows for better security, since the security for each zone is independent. Separation also allows for the installation of environments with distinct profiles on the same hardware.

The virtualization of the environment makes it easier to duplicate an environment on different physical servers.

ZFS is supported in Solaris 10 zones from the 6/2006 release and after.

Zone Installation

The system administrator configures new non-global zones via the zonecfg command, administers them via zoneadm and logs into them via zlogin.

Zone States

Zone state information can be viewed with zoneadm list -iv from the global zone. Non-global zones have one of the following states:

• configured: Configuration complete and in stable storage.
• incomplete: Installation or uninstallation underway
• installed: Configuration instantiated on system. Zone has no associated virtual platform.
• ready: Virtual platform established, zsched started, IPs plumbed, filesystems mounted, zone ID assigned. No zone processes started yet.
• running: This state entered when zone init process starts.
• shutting down: Zone being halted.
• down: Transitional state during zone shutdown.

Zone Control Commands

The following control commands can be used to manage and monitor transitions between states:

• zlogin options zone-name
• zoneadm -z zone-name boot
• zoneadm -z zone-name halt
• zoneadm -z zone-name install
• zoneadm -z zone-name ready
• zoneadm -z zone-name reboot
• zoneadm -z zone-name uninstall
• zoneadm -z zone-name verify
• zonecfg -z zone-name: Interactive mode; can be used to remove properties of the following types: fs, device, rctl, net, attr
• zonecfg -z zone-name commit
• zonecfg -z zone-name create
• zonecfg -z zone-name delete
• zonecfg -z zone-name verify

Resource Management

Zones can be used to dynamically control resource allocations on a zone basis. This means that an application on a zone can be isolated and prevented from throttling other processes in other zones on the same server.

Zone Components

The following components may be included in a zone:

• Zone name
• zonepath: Path to the zone root in the global zone's file space.
• autoboot: Whether to start the zone automatically. (Note that the svc:/system/zones:default service needs to be running in SMF for this to work.)
• pool: Associate the zone with a resource pool; multiple zones may share a pool.
• net: Zone network interface
• fs: File systems from the zone's /etc/vfstab, automounted file systems configured within the zone, manually mounted file systems or ZFS mounts from within the zone.
• dataset: This allows a non-global zone to manage a ZFS file system.
• inherit-pkg-dir: In a sparse root zone, represents directories containing packaged software that a non-global zone shares with the global zone. (Should not be used in a whole root zone.)
• device: Devices that should be configured in a non-global zone.
• rctl: Zone-wide resource controls such as zone.cpu-shares and zone.max-lwps
• attr: Zone comments

The components can be set using the zonecfg command.

zonecfg Interactive Mode

In interactive mode, zonecfg can refer to either a global scope or a specific resource. If no zone is specified in the original zonecfg command, the scope is global by default.

If a select or add subcommand is used to specify a resource, the scope limited to that resource until an end or cancel command is issued.

The following subcommands are supported:

• add: Add the specified resource or property to the configuration in the scope.
• cancel: Ends the resource specification and returns to the global scope without retaining partially specified resources.
• commit: Dump current configuration to disk.
• create: In-memory configuration begun for a new zone. A -t template option specifies a template, -F overwrites an existing configuration, and -b creates a blank configuration.
• delete: Destroy the specified configuration.
• end: Ends the resource specification
• exit: Ends the zonecfg session.
• export: Export the configuration in a form that can be used in a command file.
• help: Context-sensitive help depending on the current scope
• info: Display information about the configuration of the current scope.
• remove: Remove the specified resource or property from the command scope.
• revert: Return to the last state written to disk.
• select: From the global scope, changes scope to the specified resource
• set: Set the specified property to the specified value
• verify: Verify the current configuration for correctness.

Adding Resources

dataset:
zonecfg:zone-name> add dataset
zonecfg:zone-name:dataset> set name=pool/filesys
zonecfg:zone-name:dataset> end

fs:
zonecfg:zone-name> add fs
zonecfg:zone-name:fs> set directory=/mountpoint
zonecfg:zone-name:fs> set special=/dev/dsk/c#t#d#s#
zonecfg:zone-name:fs> set raw=/dev/rdsk/c#t#d#s#
zonecfg:zone-name:fs> set type=ufs
zonecfg:zone-name:fs> add options logging
zonecfg:zone-name:fs> end

inherit-pkg-dir:
zonecfg:zone-name> add inherit-pkg-dir
zonecfg:zone-name:inherit-pkg-dir> set dir=/package-home
zonecfg:zone-name:inherit-pkg-dir> end

net:
zonecfg:zone-name> add net
zonecfg:zone-name:net> set physical=interface-name
zonecfg:zone-name:net> set address=xxx.xxx.xxx.xxx
zonecfg:zone-name:net> end
(Examples of interface names include hme0 and bge0.)

rctl:
zonecfg:zone-name> add rctl
zonecfg:zone-name:rctl> set name=resource-name
zonecfg:zone-name:rctl> add value (priv=priv-level,limit=#,action=action-type)
zonecfg:zone-name:rctl> end

(See Resource Management for information about what sorts of values to use.)

Zone Models

There are two different zone models, sparse and whole root.

Sparse zones only contain a subset of the packages installed into the root zone. Additional packages can be brought in using the inherit-pkg-dir resources. Each sparse zone requires about 100MB of free space in the global file system. 40MB of free RAM are also recommended.

Whole root zones contain all required packages and also any optional Solaris packages that have been selected. The disk space required for whole root zones is as much as is required for a full installation. Whole root zones allow maximum configuration within the zone context.

Zone Creation Example

From within the global zone:
# zonecfg -z zone-name
zonecfg:zone-name> create
zonecfg:zone-name> set zonepath=/zone-root-path
(Note that the zone's root path cannot be on ZFS, though that capability is coming.)
zonecfg:zone-name> set autoboot=true
zonecfg:zone-name> add fs
zonecfg:zone-name:fs> set dir=/mount-point
zonecfg:zone-name:fs> set special=/global-source-dir
zonecfg:zone-name:fs> set type=lofs
zonecfg:zone-name:fs> end
(Inside the non-global zone, the mounted loopback file system will be readable and writable.)
zonecfg:zone-name> add dataset zonecfg:zone-name:dataset> set name=zone-pool/ZFS-filesys
zonecfg:zone-name:dataset> end
zonecfg:zone-name> verify
zonecfg:zone-name> commit
zonecfg:zone-name> end

Additional Reading

System Administration Guide: Solaris Containers-Resource Management and Solaris Zones

Solaris Process Scheduling

In Solaris, highest priorities are scheduled first. Kernel thread scheduling information can be revealed with ps -elcL.

A process can exist in one of the following states: running, sleeping or ready.

Kernel Threads Model

The Solaris 10 kernel threads model consists of the following major objects:

• kernel threads: This is what is scheduled/executed on a processor
• user threads: The user-level thread state within a process.
• process: The object that tracks the execution environment of a program.
• lightweight process (lwp): Execution context for a user thread. Associates a user thread with a kernel thread.

In the Solaris 10 kernel, kernel services and tasks are executed as kernel threads. When a user thread is created, the associated lwp and kernel threads are also created and linked to the user thread.

(This single-level model was first introduced in Solaris 8's alternative threads library, which was made the default in Solaris 9. Prior to that, user threads had to bind to an available lwp before becoming eligible to run on the processor.)

Priority Model

The Solaris kernel is fully preemptible. This means that all threads, including the threads that support the kernel's own activities, can be deferred to allow a higher- priority thread to run.

Solaris recognizes 170 different priorities, 0-169. Within these priorities fall a number of different scheduling classes:

• TS (timeshare): This is the default class for processes and their associated kernel threads. Priorities within this class range 0-59, and are dynamically adjusted in an attempt to allocate processor resources evenly.
• IA (interactive): This is an enhanced version of the TS class that applies to the in-focus window in the GUI. Its intent is to give extra resources to processes associated with that specific window. Like TS, IA's range is 0-59.
• FSS (fair-share scheduler): This class is share-based rather than priority- based. Threads managed by FSS are scheduled based on their associated shares and the processor's utilization. FSS also has a range 0-59.
• FX (fixed-priority): The priorities for threads associated with this class are fixed. (In other words, they do not vary dynamically over the lifetime of the thread.) FX also has a range 0-59.
• SYS (system): The SYS class is used to schedule kernel threads. Threads in this class are "bound" threads, which means that they run until they block or complete. Priorities for SYS threads are in the 60-99 range.
• RT (real-time): Threads in the RT class are fixed-priority, with a fixed time quantum. Their priorities range 100-159, so an RT thread will preempt a system thread.

Of these, FSS and FX were implemented in Solaris 9. (An extra-cost option for Solaris 8 included the SHR (share-based) class, but this has been subsumed into FSS.)

Fair Share Scheduler

The default Timesharing (TS) scheduling class in Solaris attempts to allow each process on the system to have relatively equal CPU access. The nice command allows some management of process priority, but the new Fair Share Scheduler (FSS) allows more flexible process priority management that integrates with the project framework.

Each project is allocated a certain number of CPU shares via the project.cpu-shares resource control. Each project is allocated CPU time based on its cpu-shares value divided by the sum of the cpu-shares values for all active projects.

Anything with a zero cpu-shares value will not be granted CPU time until all projects with non-zero cpu-shares are done with the CPU.

The maximum number of shares that can be assigned to any one project is 65535.

FSS can be assigned to processor sets, resulting in more sensitive control of priorities on a server than raw processor sets. The dispadmin command command controls the assignment of schedulers to processor sets, using a form like:
dispadmin -d FSS
To enable this change now, rather than after the next reboot, run a command like the following:
priocntl -s -C FSS
priocntl can control cpu-shares for a project:
priocntl -r -n project.cpu-shares -v number-shares -i project project-name

The Fair Share Scheduler should not be combined with the TS, FX (fixed-priority) or IA (interactive) scheduling classes on the same CPU or processor set. All of these scheduling classes use priorities in the same range, so unexpected behavior can result from combining FSS with any of these. (There is no problem, however, with running TS and IA on the same processor set.)

To move a specific project's processes into FSS, run something like:
priocntl -s -c FSS -i projid project-ID

All processes can be moved into FSS by first converting init, then the rest of the processes:
priocntl -s -c FSS -i pid 1
priocntl -s -c FSS -i all

Implementation Details

Time Slicing for TS and IA

TS and IA scheduling classes implement an adaptive time slicing scheme that increases the priority of I/O-bound processes at the expense of compute-bound processes. The exact values that are used to implement this can be found in the dispatch table. To examine the TS dispatch table, run the command dispadmin -c TS -g. (If units are not specified, dispadmin reports time values in ms.)

The following values are reported in the dispatch table:

• ts_quantum: This is the default length of time assigned to a process with the specified priority.
• ts_tqexp: This is the new priority that is assigned to a process that uses its entire time quantum.
• ts_slpret: The new priority assigned to a process that blocks before using its entire time quantum.
• ts_maxwait: If a thread does not receive CPU time during a time interval of ts_maxwait, its priority is raised to ts_lwait.
• ts_lwait:

The man page for ts_dptbl contains additional information about these parameters.

dispadmin can be used to edit the dispatch table to affect the decay of priority for compute-bound processes or the growth in priority for I/O-bound processes. Obviously, the importance of the different types of processing on different systems will make a difference in how these parameters are tweaked. In particular, ts_maxwait and ts_lwait can prevent CPU starvation, and raising ts_tqexp slightly can slow the decline in priority of CPU-bound processes.

In any case, the dispatch tables should only be altered slightly at each step in the tuning process, and should only be altered at all if you have a specific goal in mind.

The following are some of the sorts of changes that can be made:

• Decreasing ts_quantum favors IA class objects.
• Increasing ts_quantum favors compute-bound objects.
• ts_maxwait and ts_lwait control CPU starvation.
• ts_tqexp can cause compute-bound objects' priorities to decay more or less rapidly.
• ts_slpret can cause I/O-bound objects' priorities to rise more or less rapidly.

RT objects time slice differently in that ts_tqexp and ts_slpret do not increase or decrease the priority of the

IA objects add 10 to the regular TS priority of the process in the active window. This priority shifts with the focus on the active window. object. Each RT thread will execute until its time slice is up or it is blocked while waiting for a resource.

Time Slicing for FSS

In FSS, the time quantum is the length of time that a thread is allowed to run before it has to release the processor. This can be checked using
dispadmin -c FSS -g

The QUANTUM is reported in ms. (The output of the above command displays the resolution in the RES parameter. The default is 1000 slices per second.) It can be adjusted using dispadmin as well. First, run the above command and capture the output to a text file (filename.txt). Then run the command:
dispadmin -c FSS -s filename.txt

Callouts

Solaris handles callouts with a callout thread that runs at maximum system priority, which is still lower than any RT thread. RT callouts are handled separately and are invoked at the lowest interrupt level, which ensures prompt processing.

Priority Inheritance

Each thread has two priorities: global priority and inherited priority. The inherited priority is normally zero unless the thread is sitting on a resource that is required by a higher priority thread.

When a thread blocks on a resource, it attempts to "will" or pass on its priority to all threads that are directly or indirectly blocking it. The pi_willto() function checks each thread that is blocking the resource or that is blocking a thread in the syncronization chain. When it sees threads that are a lower priority, those threads inherit the priority of the blocked thread. It stops traversing the syncronization chain when it hits an object that is not blocked or is higher priority than the willing thread.

This mechanism is of limited use when considering condition variable, semaphore or read/write locks. In the latter case, an owner-of-record is defined, and the inheritance works as above. If there are several threads sharing a read lock, however, the inheritance only works on one thread at a time.

Thundering Herd

When a resource is freed, all threads awaiting that resource are woken. This results in a footrace to obtain access to that object; one succeeds and the others return to sleep. This can lead to wasted overhead for context switches, as well as a problem with lower priority threads obtaining access to an object before a higher-priority thread. This is called a "thundering herd" problem.

Priority inheritance is an attempt to deal with this problem, but some types of syncronization do not use inheritance.

Turnstiles

Each syncronization object (lock) contains a pointer to a structure known as a turnstile. These contain the data needed to manipulate the syncronization object, such as a queue of blocked threads and a pointer to the thread that is currently using the resource. Turnstiles are dynamically allocated based on the number of allocated threads on the system. A turnstile is allocated by the first thread that blocks on a resource and is freed when no more threads are blocked on the resource.

Turnstiles queue the blocked threads according to their priority. Turnstiles may issue a signal to wake up the highest-priority thread, or they may issue a broadcast to wake up all sleeping threads.

Adjusting Priorities

The priority of a process can be adjusted with priocntl or nice, and the priority of an LWP can be controlled with priocntl().

Real Time Issues

STREAMS processing is moved into its own kernel threads, which run at a lower priority than RT threads. If an RT thread places a STREAMS request, it may be serviced at a lower priority level than is merited.

Real time processes also lock all their pages in memory. This can cause problems on a system that is underconfigured for the amount of memory that is required.

Since real time processes run at such a high priority, system daemons may suffer if the real time process does not permit them to run.

When a real time process forks, the new process also inherits real time privileges. The programmer must take care to prevent unintended consequences. Loops can also be hard to stop, so the programmer also needs to make sure that the program does not get caught in an infinite loop.

Interrupts

Interrupt levels run between 0 and 15. Some typical interrupts include:

• soft interrupts
• SCSI/FC disks (3)
• Tape, Ethernet
• Video/graphics
• clock() (10)
• serial communications
• real-time CPU clock
• Nonmaskable interrupts (15)

Solaris 10 Resource Management

Solaris 10 resource mangement is a major step forward over what was available in Solaris 8 and 9. In Solaris 10, we can manage resources at a zone, project or task level. This page focuses mainly on project-level resource management. Additional information is available in Sun's System Administration Guide: Solaris Containers-Resource Management and Solaris Zones on the Sun Documentation Web Site.

Projects

Projects are collections of tasks, which are collections of processes. A new task is started in a project when a new session is opened by a login, cron, newtask, setproject or su command. Each process belongs to only one task, and each task belongs to only one project.

The default project for a user is determined as per the getdefaultproj() man page.

When there is more than one policy in place for a particular object, the smallest container's control is enforced first.

Projects are maintained via the /etc/project file. Changes to /etc/project become available for new tasks in a project. (prctl and rctladm are used to perform runtime changes.)

The fields in an /etc/project entry are:

• projname: Name of the project.
• projid: Unique numerical project identifier less than UID_MAX (2147483647).
• comment: Project description.
• user-list: Comma-separated list of users.
• group-list: Comma-separated list of groups.
• attributes: Semicolon-separated list of name-value pairs, such as resource controls, in a name[=value] format.

After a default Solaris 10 installation, /etc/project contains the following:

system:0::::(default project for system processes and daemons)
user.root:1::::(processes owned by the root user)
noproject:2::::(IP Quality of Service)
default:3::::(default assigned to every otherwise unassigned user)
group.staff:10::::(default used for unassigned users in the "staff" group)

Parameters are set by adding them to the last field of the project entry:
projectname:101::::project.max-lwps=(privileged,200,deny)

Management Commands

Commands for managing project attributes include the following:

• projects: Displays project memberships for users, lists projects from the project database, prings information on given projects.
• newtask: Executes the shell or command in a new task in the current project.
• projadd: Adds a new entry to the /etc/project entry.
• projmod: Modifies information for a project in /etc/project.
• projdel: Deletes a project from /etc/project.
• rctladm: Displays/modifies global state of active resource controls, sets logging or actions.
• prctl: Displays/modifies local resource controls.
• ipcs: Identifies which IPC objects are being used in a project.
• rcapadm: Manages rcapd memory-capping daemon.
• prstat -J: Displays resource consumption on a per-project basis
• priocntl -i project-name: Sets/displays scheduling parameters of the project.
• poolbind -i project-name: Assigns a project to a resource pool.

Usage examples are provided at the end of this page.

Privilege Levels

Each resource control threshhold needs to be associated with one of the following privilege levels:

• basic: Can be modified by owner of calling process.
• privileged: Only modifiable by superuser
• system: Fixed for the duration of the operating system instance

IPC Resource Controls

The Solaris 10 IPC resource management framework fixes some serious problems in the older SVR4-based system. Some parameters were converted to be set dynamically, some defaults were increased, some parameters were retired, and the names of the surviving parameters were changed to be more human-readable.

In older Solaris versions, the resource limits were system-wide (causing potential conflicts) and reboots were required for even minor changes.

The Solaris 10 system permits project-based resource controls and allows controls to be monitored and changed via prctl.

Additional information about IPC resource management can be found on the IPC Issues page.

For the purposes of IPC resource management, the following are the important parameters:

• project.max-shm-ids: Maximum shared memory IDs for a project. Replaces shmmni
• project.max-sem-ids: Maximum semaphore IDs for a project. Replaces semmni
• project.max-msg-ids: Maximum message queue IDs for a project. Replaces msgmni
• project.max-shm-memory: Total amount of shared memory allowed for a project. Replaces shmmax
• process.max-sem-nsems: Maximum number of semaphores allowed per semaphore set. Replaces semmsl
• process.max-sem-ops: Maximum number of semaphore operations allowed per semop. Replaces semopm
• process.max-msg-messages: Maximum number of messages on a message queue. Replaces msgtql
• process.max-msg-qbytes: Maximum number of bytes of messages on a message queue. Replaces msgmnb

An Oracle-specific example is provided below.

Other Resource Controls

The new Solaris 10 resource controls include compatibility interfaces to the old rlimit-style resource controls. Existing applications using the old interfaces can continue to run unchanged.

Additional Resource Controls:

• [zone|project].cpu-shares: Maximum CPU shares allowed (under Fair Share Scheduler)
• [task|process].max-cpu-time: Maximum CPU time available to processes in this task.
• project.max-contracts: Maximum number of contracts allowed
• project.max-crypto-memory: Total kernel memory usable by libpkcsll for hardware crypto accelleration.
• project.max-device-locked-memory: Total locked memory allowed.
• process.max-address-space: Maximum address space.
• process.max-core-size: Maximum core dump size.
• process.max-data-size: Maximum heap size.
• process.max-file-descriptor: Maximum file descriptor index.
• process.max-file-size: Maximum file offset allowed for writes.
• process.max-stack-size: Maximum stack memory segment available.
• [zone|project|task].max-lwps: Maximum lwps available to this project.
• process.max-port-events: Maximum events per port.
• project.max-port-ids: Maximum allowable event ports.
• project.max-tasks: Maximum allowable tasks.
• rcap.max-rss: Maximum physical memory consumption by processes in project.

A full list of resources is available on the resource_controls man page.

Resources beginning with the rcap string are associated with the rcapd resource-capping daemon.

rcapd

rcapd caps memory useage within a project. In each zone, rcapd can be enabled via
rcapadm -E
This command will start rcapd and set it up in SMF so that it will be restarted automatically.

We can use projmod to set the memory cap for a project:
projmod -s -K rcap.max-rss=sizeMB project-name
Alternatively, we can set the rcap.max-rss control directly in /etc/project.

rcapd does not account for shared memory in an intuitive way. To be safe, we need to allow enough room for shared memory to be included under the cap. We should not depend solely on rcapd to manage process memory.

Logging

Global logging can be enabled by setting syslog=level with rctladm, where level is one of the usual syslog levels: debug, info, notice, warning, err, crit, alert or emerge.

Actions

It is possible to use rctladm to specify one of the following actions on a process that violates the control:

• none: No action taken. (Useful for monitoring.)
• deny: Denies request.
• signal=: Enable a signal. See the rctladm man page for a list of allowed signals.

Command Examples

The projadd man page provides an example of how to add a project:

The following command creates the project salesaudit and sets the resource controls specified as arguments to the -K option.

projadd -p 111 -G sales,finance -c "Auditing Project" -K "rcap.max-rss=10GB" -K "process.max-file-size=(priv,50MB,deny)" -K "task.max-lwps=(priv,100,deny)" salesaudit

This command would produce the following entry in /etc/project:
salesaudit:111:Auditing Project::sales,finance:process.max-file-size=(priv,52428800,deny); rcap.max-rss=10737418240;task.max-lwps=(priv,100,deny)

To start up a task under this project, run the following:
newtask -p salesaudit command

A running process can be associated with a new task:
newtask -v -p project-name -c PID

To verify the project governing the current shell, we would run:
id -p

To view resource constraints for a process, we would run something like the following:
prctl -n resource-name -i process PID

To view resource constraints for the current shell, we could run:
prctl $$

To temporarily set resource constraints on a particular project, we could run something like:
prctl -n resource-name -t privilege-level -v value -e action -i project project-name

To activate logging on a global resource control facility, run something like:
rctladm -e syslog=level resource-name

To list all existing projects, run:
projects -l

To see how a project's IPC objects are allocated against existing limits, run something like:
ipcs -J

To display a process's project id, use a command of the form:
ps -o projid -p PID

To match project or task ids for pgrep, pkill or prstat commands, use the -T or -J options:
pgrep -J project-IDs
pkill -T task-IDs
prstat -J

Oracle Setup Example

Oracle 9i recommends several minimum semaphore and shared memory settings. Since Solaris 10 has increased the defaults on several settings above previous levels, and since several other ones have become obsolete, only the shmmax parameter should need to be set.

In particular, the new defaults for some key parameters are:

• semmni: 128 (100 recommended)
• semmsl: 512 (256 recommended)
• shmmni: 128 (100 recommended)

The following are obsolete:

• semmns
• shmmin
• shmseg

The projmod command can be used to set the shmmax to the desired level (default is 1/4 physical memory):
projmod -sK "project.max-shm-memory=(privileged,gigabytes-sharedGB,deny)" project-name

It makes sense to set up projects (and project limits) for each environment on the server. To ensure that each instance actually starts up in the proper project, the startup scripts will need to include a
newtask -p project-name
line.

A full example of this type is found in Chapter 4 of The Sun BluePrints Guide to Solaris Containers.

Default Project

The default project for a user is determined as per the getdefaultproj() man page:

The getdefaultproj() function first looks up the project key word in the user_attr database used to define user attributes in restricted Solaris environments. If the database is available and the keyword is present, the function looks up the named project, returning NULL if it cannot be found or if the user is not a member of the named project. If absent, the function looks for a match in the project database for the special project user.username. If no match is found, or if the user is excluded from project user.username, the function looks at the default group entry of the passwd database for the user, and looks for a match in the project database for the special name group.groupname, where groupname is the default group associated with the password entry corresponding to the given username. If no match is found, or if the user is excluded from project group.groupname, the function returns NULL. A special project entry called 'default' can be looked up and used as a last resort, unless the user is excluded from project 'default'. On successful lookup, this function returns a pointer to the valid project structure. By convention, the user must have a default project defined on a system to be able to log on to that system.

Additional Reading

Additional information is available in Sun's System Administration Guide: Solaris Containers-Resource Management and Solaris Zones and The Sun BluePrints Guide to Solaris Containers on the Sun Documentation Web Site.

Resource Pool Management

The purpose of resource pools is to segregate different workloads to eliminate their contention for certain types of resources. Resource pools allow persistent configuration for workload processor set and scheduler assignments.

Pools can operate on the following types of elements:

• system: The system as a whole
• pool: Define resource pools
• pset: Processor sets.
• cpu: Single processor.

Each project can have a project.pool attribute set in /etc/project in order to assign a resource pool. This would be set by a command of the following form:
projmod -a -K project.pool=poolname project-name

Dynamic resource pools are implemented via the poold daemon. poold can only execute in the global zone.

The default static pools configuration file is /etc/pooladm.conf. If this file exists, its configuration is applied at boot time.

The poolcfg command operates on a configuration file, though poolcfg -d may be used to modify the current kernel state. pooladm is used to activate the pool configuration contained in a file or to display information about the configuration currently in the kernel.

To enable a configuration, run:
pooladm -e To instantiate the configuration in a file, run:
pooladm -c filename
To dump the active configuration to a file, run:
pooladm -s filename
To disable pools, run
pooladm -d
To display the pool configuration in a "nice" format, run:
poolcfg -c info

poolstat can be used to display information on resource pools. If executed from a non-global zone, however, it will only display information for the pool assigned to that zone.

Multiple non-global zones can be associated with a particular resource pool. When this is done, processes from within one of these zones cannot be assigned to a different resource pool via poolbind, even when the attempt is made from the global zone.

poold Properties

The following are the property names for poold:

• system.poold.log-level: Logging level
• system.poold.log-location: Log location
• system.poold.monitor-interval: Sampling interval
• system.poold.history-file: Decision history
• pset.max: Maximum CPUs for a pset.
• pset.min: Minimum CPUs for a pset.
• cpu.pinned: (true/false) Whether the CPU is pinned.
• system.poold.objectives: System objectives.
• pset.poold.objectives: Pset objectives.
• pool.importance: User-assigned importance of a pool.
• pool.scheduler: Defines scheduler to be used by a pool.

These are assigned with poolcfg. For example, to assign a pool to a scheduler, run:
poolcfg -c 'modify pool pool-name (string pool.scheduler="scheduler-initials ")'

poold Configuration: Constraints and Objectives

Minimum and maximum numbers of processors available to a processor set can be set with the pset.min and pset.max properties. The cpu.pinned property indicates that a particular CPU should not be moved out of its processor set.

The libpool man page contains detailed information on property constraints.

Configuration objectives define the behavior of poold. The available objectives are:

• wt-load: The weighted-load objective provides pooled resources based on the weight of the active objects. It is a system-wide objective.
• locality: locality is defined on a pset. It defines how much weight is given to the closeness of available resources. It can be assigned values of tight (maximize locality), loose (minimize locality) or none (not taken into account-default).
• utilization: This is defined to be greater than (>), less than (<) or about (~) some percentage of utilization for a pset. The > and < operators can be used to set a range, but ~ must be used alone if at all.

The objectives are also set with poolcfg:
poolcfg -c 'modify system system-name (string system.poold.objectives="objective "),

If wt-load is set, note that the psets need to have "locality none" set as an objective.

Processor Sets

Each processor can belong to only one processor set at a time. The default processor set is created when poold is enabled. It cannot be destroyed.

To create a processor set with a range of possible numbers of CPUs, run:
poolcfg -c 'create pset pset-name (uint pset.min = min-cpus; unit pset.max = max-cpus)'
To create a pool, run:
poolcfg -c 'create pool pool-name'
To associate a pool and a pset, run:
poolcfg -c 'associate pool pool-name (pset pset-name)'
To apply the changes, run:
pooladm -c

To transfer processors between processor sets, run:
poolcfg -dc 'transfer number-procs from pset pset-source to pset-target'

Additional Reading

Additional information is available in Sun's System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

Book Review: Root Cause Analysis by Duke Okes

Okes has done a remarkable job creating a book that is readable, concise and useful. There are several well-known books which will smother you with statistics and studies. Okes' book is a common sense introduction to root cause analysis that is general enough to be widely applicable, but specific enough to help most front-line people sharpen their troubleshooting process.

The examples are well-chosen and pointed, and the graphs and charts are explicit and well-explained.

If you need to troubleshoot or perform root cause analyses, this is an excellent book to keep on your bookshelf.

Book Review: Oracle Solaris 11 System Administration: The Complete Reference

Authors of books in the "Complete Reference" series have always had a tough row to hoe. They attempt to condense an entire set of product documentation to a single, referenceable volume. And all that has to be done in the time frame allowed by software release schedules.

Kudos to the authors of this particular volume for having pulled together a lucid, reasonably detailed summary of the most important parts of Solaris. Despite the name, no one book can actually be a complete reference for something as complex as Solaris, but this book is a pretty good addition to any Solaris admin's bookshelf.

One of the difficulties of writing a book by committee is maintaining the same audience and consistent focus across chapters and topics. In this case, there are some chapters where the level of detail does not match what is present in other chapters. And in some cases, there is a distinct audience mis-match.

I came across one howler in Chapter 10: "differences between shells are subtle, indistinguishable to most regular Solaris 11 users." The author of that chapter goes on to explain that for that reason, the chapter will focus on bash. Fair enough on the selection of the most common shell for detailed discussion. But you have to wonder how useful a chapter is going to be if the assumed audience is too clueless to see the difference in how sh and csh (for example) execute even simple actions like setting an environment variable.

It would be nearly impossible for a book of this type to earn 5 stars. But this volume gets a solid 4 in my book.

Process Accounting

Process accounting can be turned on by referencing the /etc/init.d/acct file, either in a command line with start/stop, or by linking to the appropriate rc file:

ln -s /etc/init.d/acct /etc/rc2.d/S22acct

ln -s /etc/init.d/acct /etc/rc0.d/K22acct

The /etc/acct/holidays file may also need to be edited, if different monitoring is desired during holidays.

The root and adm crontabs can also be edited to run dodisk, ckpacct, monacct, and runacct.

Process accounting logs can be examined directly using acctcom.

Kernel Memory

Kernel memory size can be tracked using the sar -k command. The total of the "alloc" fields is the kernel memory size. If it appears to be growing without bound, you may have a memory leak. It should be noted that not all buckets are tracked by sar -k, so the reported memory size is not as accurate as that reported by crash.

On occasion there are problems related to memory leaks in the kernel or one of the associated modules. In these cases, kmastat can provide useful information pinpointing the source of the leak.

To check on kernel memory allocations on a running system, run a crash session as follows:

# crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> kmastat

The first number on the "Total" line represents the total amount of memory allocated by the kernel. If this is a significant fraction of available system memory and growing, there is a problem.

The output from the kmastat command also contains information on a number of "buckets" or categories for memory allocation.

Additional information can be obtained via the kmausers command, but this requires that we load kadb prior to booting. To do this, reach the ok> prompt, then:

ok> boot kadb -d
kadb: (hit the "return" key)
kadb[0]: kmem_flags/W 01
kadb[0]: :c

Loading kadb this way means that kadb will only be effective for this current boot session.

Once the system is up, we can either force a core dump via STOP-A/ ok> sync, or we can examine the live system. In either case, inside the crash session we would type:

>kmausers bucket_name

The result will show memory allocations inside that bucket. The names of functions inside each allocation will be a tip-off to what is actually grabbing the memory allocation.

A script can be run from cron to capture this information. The format of this script would be something like:

#!/bin/sh
date >> log_file
echo "kmastat" | /usr/sbin/crash -w log_file
sleep 20
echo "kmausers kmem_alloc_2048" | /usr/sbin/crash -w log_file


Slab Allocator

Solaris 2.4+ uses a kernel memory allocator known as a slab allocator.

A kernel memory allocator performs the following functions:

• Allocate memory
• Initialize objects/structures
• Use objects/structures
• Deconstruct objects/structures
• Free memory

The structures in the memory objects include sub-objects such as linked list headers, mutexes, reference counts and condition variables. In the case of Solaris, the deconstruction step includes setting objects to their initial settings, which can save time when the memory objects have to be re-initialized.

A translation lookaside buffer (TLB) is an associative cache of recent address translations. When the MMU (memory management unit) cannot find a translation in the TLB, it lookus it up in the address maps and loads the address into the TLB. Entries in the TLB are replaced on a least recently used basis.

The slab allocator is organized as a collection of object caches. Each of these caches contains only one type of object (proc structures, vnodes, etc). The kernel is responsible for restoring each object to its initial state when it is released. When a cache requires additional space, the allocator gives a slab of memory from the page-level allocator and creates objects from it. The slab contains enough memory for several object instances. A small part of the slab is used by the cache to manage memory in the slab; the rest is divided into buffers that are the size of the object. The allocator then initializes these buffers with the appropriate constructor.

When the page-level allocator needs to recover memory, unused slabs are reaped by deconstructing the objects on slabs whose objects are all free, then removing the slab from the cache in question.

The structure for each slab includes unused space at the beginning of the slab (coloring area), the set of objects, more unused space (the amount left over after the maximum number of objects has been created), and a slab data area. Each object also includes a four byte area for a free list pointer. The slab data area includes a count of in-use objects, pointers for a doubly-linked list of slabs in the same cache, and a pointer to the first free area in the slab. The coloring areas are different sizes for each slab in a cache (where possible). This allows a balanced distribution of traffic on the hardware caches and memory busses by varying the offsets for the different slabs.

Large object slabs are slightly different in that management data is stored in a separate pool of memory, since large slabs are usually multiples of a page in size. A hash table is also maintained to provide lookups between the management area and the slabs.

Lock Contention

Four types of locking are available on Solaris:

• Mutexes
• Semaphores (counters) (not the same as IPC semaphores)
• Condition variables (generalized semaphores)
• Multiple-reader, single-writer locks

The following types of locking problems can occur:

• Lock contention (due to excessively coarse granularity or inappropriate lock type)
• Deadlock (each process is waiting for a lock held by another process)
• Lost locks
• Race conditions
• Incomplete or buggy lock implementation

Mutex Locks

A "mutex lock" is a "mutual exclusion lock." It is created by the LDSTUB (load-store-unsigned-byte) instruction, which is an atomic (indivisible) operation that reads a byte from memory and writes 0xFF into that location. (When the lock is cleared, 0x00 is written back to the memory location.)

If the value that was read from memory is already 0xFF, another processor has already set the lock. At that point, the processor can "spin" by sitting in a loop and testing to see if the lock has cleared (i.e., been written back to 0x00). This sort of "spin lock" is usually used when the wait time for the lock is expected to be short. (If the wait is expected to be longer, the process should sleep so that the CPU can be used by another process. This is known as a "block.")

Adaptive Locks

Solaris 2.x provides a type of locking known as adaptive locks. When one thread attempts to acquire one of these that is held by another thread, it checks to see if the second thread is active on a processor. If it is, the first thread spins. If the second thread is blocked, the first thread blocks as well.

Read/Write Locks

This type of lock allows multiple concurrent reads, but prevents other accesses of the resource when writes are taking place.

Lock Contention Indicators

One indicator of a possible lock contention problem is when vmstat reports that the system is not idle, but that cpu/sy dominates cpu/us. (Note: this observation is only true if the system is not running an NFS server or other major service that runs from inside the kernel.)

One way to pin down a lock contention problem is by tracing the problem process with truss.

Another way to attempt to track down the problem is with mpstat. The smtx measurement shows the number of times a CPU failed to obtain a mutex immediately. The master CPU (the one taking the clock interrupt--usually CPU 0) will tend to have a high reading. Depending upon CPU speed, a reading of more than 500 may be an indication of a system in trouble. If the smtx is greater than 500 on a single CPU and sys dominates usr (ie, system time is larger than user time, and system time is greater than 20%), it is likely that mutex contention is occurring.

Similarly, mpstat/srw value reports on the number of times that a CPU failed to obtain a read/write lock immediately.

For Solaris 2.6 and above, the lockstat command can help to pin down the culprit. The kernel takes a performance hit while lockstat is running, so you probably only want to use this command while you are actually looking at the output.

With lockstat, look for large counts (indv), especially with long locking times (nsec).

In any case, extreme mutex contention problems should be reported to Sun. Changes have been implemented in current versions of the SunOS 5.x kernel that dramatically increase the scalability of the operating system over multiple processors. Unless additional issues are brought to the vendor's attention, they cannot be expected to correct them in future releases.