Solaris Troubleshooting

Internet Explorer 8 and 9 Rendering Problems

2011-06-29T21:52:00.002-04:00

Thanks, James, for pointing out the rendering problems with Internet Explorer 8 and 9. I found the HTML setting that fixes the problem. Evidently IE 8 and 9 want the width set on the nested table containing the body of the page, even though the table is itself nested in a cell whose width is already set.

Thanks, Microsoft, for demonstrating just how highly you prize conformity to standards and backwards compatibility with your own products.

The re-formatted pages have been sent to Princeton and will hopefully be live in the next few days.

2011 Re-print of Solaris Troubleshooting Handbook

2011-06-07T01:59:00.003-04:00

The 2011 re-print of the Solaris Troubleshooting Handbook is available from the printer at: https://www.createspace.com/3617377

Amazon will also be distributing the book, and it should be listed within 3-5 business days.

Please provide comments here as to any errata that may be discovered, or on sections that should be targeted for updates in future editions. People who provide substantive comments may be invited to assist in reviewing the next edition.

As noted in an earlier post, I have been forced to moderate comments due to a large amount of sexually explicit spam posted as comments to this blog. If you do not see your comment right away, it is probably just waiting for me to review and publish it.

Thanks,
--Scott

Moderating Comments

2009-09-13T09:22:00.002-04:00

I'm afraid that I've been forced to moderate comments on this blog due to someone who appears to be posting sexually explicit comments in Japanese. I tried just deleting them as they appear, but I'm getting tired of doing that.

If you're the person posting the comments, go open your own blog.

Sun Cluster Page

2008-08-25T20:41:00.000-04:00

I've added a new Sun Cluster page to the site. I'll be expanding it over time.

--Scott

ZFS Page Update

2008-08-25T17:30:00.000-04:00

I've submitted additional updates to the ZFS page.

In particular, the revised version of the page contains some basic reference information on ZFS structure internals, as well as links to papers that can be used to walk the internal data structures of a ZFS file system:

ZFS
On-Disk Data Walk in the OpenSolaris Developer Conference Proceedings.

ZFS On-Disk Specification

ZFS page updated

2008-07-20T21:01:00.002-04:00

The ZFS page has been updated with references to the Evil Tuning Guide and some of the tuning parameters that are discussed there.

Sun Cluster, HAStoragePlus and VxFS fsck

2008-04-27T10:36:00.001-04:00

It turns out that when you have a decent number of VxFS file systems, there is a particularly nasty bug that can jump out and bite you in the *.

It turns out that Sun Cluster wants the vfstab entries for these file systems to be listed as being in the first fsck pass. This particular bug is Bug ID 6572900. Note that the bug description says that this fix does not work for all sites. Fortunately, the workaround seems to be working here.

This bug report is a good example of what I hate to see in bug reports from vendors (not just Sun). The status on the bug is reported as "fix delivered" (presumably the above workaround, since I don't see any patches), but the bug report notes that the workaround doesn't work for all sites. If customers are still having the problem, I don't see how we can claim that a fix has been delivered.

--Scott

Enclosure Based Naming and Sun Cluster

2008-04-27T09:58:00.000-04:00

As I posted previously, we are working on getting our Sun Cluster installation running. We had originally hoped to use ZFS for this project, which would have made deployment much easier. Unfortunately, the risks associated with the bug discussed in my ZFS Difficulties post are just too high to allow us to deploy it for production use at this time.

Instead, we are moving to Veritas Volume Manager and File System (VxVM and VxFS) for control of the data directories and shared storage.

We hit a hitch associated with Enclosure Based Naming. It turns out that if your storage drivers return devices in a format including the WWN, Enclosure-Based Naming is mandatory. Since the names are assigned based on some obscure, secret algorithm, you are virtually guaranteed that the different systems in your cluster will have different Enclosure Based Names associated with them.

Sun Cluster, on the other hand, insists that all of your devices be named the same way on all systems in the cluster.

Fortunately, Sun Infodoc 215296 describes how to resolve this impasse for VxVM 4.1+.

First, boot the node that you will be fixing in non-cluster mode. You can do this with a
boot -x
command. Make sure that no disk groups are imported from the shared storage before proceeding.

The /etc/vx/disk.info file contains a mapping from the WWN-based device names (that you see in OS-based commands like format) to the Enclosure Based Names. You can edit this file directly, changing only the numerical part of the Enclosure Based Name that does not match between the different servers.

After making this change, and before you reboot, run the following command to re-configure the storage:
vxconfigd -k
(Note that volumes and disk groups will freak out at this point if you have them imported on the system, since they see the disks being migrated to a "different" location. You really want to have deported the disk groups before you start, or else you may find yourself doing plex recovery.)

You can verify that the changes took effect properly by running
vxdisk -e list
and matching up the disk names to your reference configuration.

Good luck!

--Scott

ZFS Difficulties

2008-04-18T20:06:00.002-04:00

We've been evaluating ZFS as a replacement for VxVM and VxFS in some of our production clusters. We encountered some difficulties.

ZFS has supported our development environment for about a year now, and we have enjoyed the flexibility and feature set of ZFS--especially the snapshot management and the ease of volume management. The performance, however, has left something to be desired. We had hoped to demonstrate that we could get adequate performance on ZFS by being more aggressive with tuning.

During the initial testing, we did not use ZFS mirroring or RAIDZ, though we did separate the Oracle log and temp files into separate pools as suggested in the ZFS Best Practices Guide, and we did tweak several tuning parameters as suggested in the ZFS Evil Tuning Guide.

We were able to get performance that met our requirements, but we experienced a much more serious problem when our test system blew a CPU. One of our pools became corrupted, which resulted in a panicked server. This was unexpected; we had isolated our test environment on a non-global zone and used dataset to delegate the zpool administration to the zone. We had expected a corrupted zpool to cause problems that were isolated to the zone, not to cause problems to the entire server.

This server was part of a nascent Sun Cluster configuration that we were also testing; the failover server attempted to import the zpool and promptly panicked. Investigation with zdb -lv revealed that the metadata had become corrupted as a result of the processor failure and resulting panic. We opened a call with Sun, and I posted message with the details to the ZFS Discuss list.

The upshot is that Solaris 10u5 and earlier do not have a way to prevent a panic on zpool corruption. Nevada allows you to specify how the OS will react to zpool corruption, but this will not be brought forward to Solaris 10 until update 6.

We could have reduced the likelihood of zpool metadata corruption by having more than one zdev in each pool (allowing the metadata replicas to have copies on different zdevs). Sun tells us that this would have narrowed the window in which we could have corrupted the metadata, but that the problem itself will not be fixed until update 6 later this year.

Until we get this problem resolved, I don't see being able to use ZFS in a critical production environment. If you have any light to shine on any of these issues, please post a comment here or on the ZFS Discuss list.

--Scott

Update on Book Project

2008-04-18T20:05:00.004-04:00

I'm in the process of re-writing a couple of the chapters, which is going slower than I would like. It is getting closer to finished. If you send me your email address, I will let you know when the book is released.

(NOTE: The 2011 re-print of this book is available from the printer at: https://www.createspace.com/3617377 )

Book Project

2007-08-19T22:42:00.001-04:00

I am compiling the information from the web site into a book. If you would like to be a technical reviewer for the book, please let me know, and I will send you a copy of the current draft. People who provide me with comments will be thanked on the acknowledgment page; I'm afraid I'm not in a position to offer anything more than that.

You can contact me through the email address listed for this blog, or by adding a comment to this blog posting. The folks at Princeton are not involved in this project, so please don't direct any inquiries there.

--Scott

inittab update

2007-08-07T21:58:00.000-04:00

I updated the inittab page to correct the error that Todd pointed out. Thanks for the help! Please let me know if you find other errors or pages that need better explanation.

The LOPSA conference in Cherry Hill was great! If you didn't make it this year, make sure to put it on your calendars for next year.

--Scott

Solaris Multipathing, System Bus pages updated

2007-07-25T00:30:00.000-04:00

I updated the Solaris Multipathing and System Bus pages. We tripped over an issue with editing the scsi_vhci.conf for a third party storage device. The syntax is incredibly picky and wants new entries to be padded out with spaces, as I describe in the updated page.

--Scott

p-Commands page

2007-07-09T18:36:00.000-04:00

The long-promised P-Commands page has been posted.

--Scott

Updated Presentation and Lecture Notes

2007-07-09T18:34:00.001-04:00

I updated the Impress presentation and lecture notes documents and moved them to the Troubleshooting Essentials share.

--Scott

August SysAdmin

2007-07-09T18:24:00.001-04:00

The August SysAdmin has an article based on the Troubleshooting Methodology page.

--Scott

Upcoming Class

2007-05-16T19:01:00.000-04:00

I'll be teaching a Solaris Troubleshooting class at the upcoming Sysadmin Days conference in Cherry Hill, NJ in August.

Class Description:

Troubleshooting is where the rubber meets the road for system administrators. Our employers need us to identify and fix the root cause of our problems with a minimum of disruption and cost. Our technical chops are important, but we also need to work collaboratively with our coworkers and customers to resolve the problem for once and for all. This course covers several techniques and tools for organizing a successful, collaborative troubleshooting process.

But this is not just a touchy-feely class! Solaris 10 includes a raft of tools for getting to the bottom of performance, hardware and software environments. We'll review several key tools for looking at different aspects of system and application functioning. Some of the tools and techniques we'll look at include:

* Error messages--how to collect and interpret them
* Fault profiles--handy rules of thumb
* Getting at the system's guts with DTrace
* Examining a crash dump
* Using the p-tools and the /proc pseudo-filesystem
* Cool free tools from the Web

Scott Cromar

Scott Cromar has been a fan of Solaris since using a Sun workstation to investigate the sex life of sea grasses while an undergraduate. He is currently a Unix Technical Lead with more than a dozen years of experience supporting Solaris and Linux systems in academic and financial services production environments. Along the way, he created Princeton University's Solaris Troubleshooting web site and the companion Solaris Troubleshooting Blog. Besides technical articles exposing the seamy side of life on the seabed, he has published articles for SANS and SysAdmin Magazine.

LOPSA-NJ Presentation

2007-04-29T21:44:00.000-04:00

I'll be presenting some troubleshooting tips to the LOPSA NJ meeting on May 3 at the Lawrence Public Library at 7pm. My Impress presentation is included below. (Impress is part of the OpenOffice suite.):

(I moved the presentation's share to the top of the page.)

Date: Thursday, May 3, 2007, 7:00pm (social), 7:30pm (discussion)

Topic: An overview of some of the different ways to troubleshoot
problems in a Solaris 10 environment, with emphasis on the tools that
are specific to Solaris

Speaker: Scott Cromar

Description: The talk will be a discussion of a general
troubleshooting approach including the use of Ishikawa and
Interrelationship diagrams in troubleshooting. With a brief overview of
some of the tools available in Solaris 10 and a brief introduction to
DTrace.

Location: Lawrence Headquarters Branch of the Mercer County Library,
Community Room 2

Service Management Facility Page Update

2007-03-21T19:08:00.000-04:00

The SMF Page has been updated with information about how to find the scripts that run stops and starts.

Resource Management article

2007-03-11T21:07:00.000-04:00

The April SysAdmin Magazine carries an article based on the Resource Management page.

ZFS Management Page Updates

2007-03-11T21:05:00.000-04:00

I updated the ZFS Management and Troubleshooting page with information from the OpenSolaris documentation pages for ZFS.

Zones Page

2007-03-06T21:18:00.000-05:00

I just submitted some updates for the Zones page.

Service Management Facility and Fault Management Updates

2007-03-03T20:55:00.000-05:00

I submitted updates for the Service Management Facility and Fault Management Architecture pages. They will probably be available early next week.

Resource Management and Scheduling Corrections

2007-02-08T18:15:00.000-05:00

I've corrected some minor errors on the Resource Management and Scheduling pages.

SysAdmin Magazine confirms that they will be publishing my Solaris Resource Management article in their April issue.

--Scott

Updates to Error, RAM and DiskIO Pages

2007-01-30T06:57:00.000-05:00

I've just submitted an expanded Error Messages page as well as corrections to the Memory and Disk I/O page.

Thanks in particular to Brian for pointing out the error on the Memory page!

If you see things that need correcting or clarifying, please post them as comments; I have the blog set up to forward them to me as email. (If you know me and my email address, feel free to send them to me directly, but I've had to pull my email off of the web to try to get my spam levels under control.)

--Scott

Error Messages

2007-01-09T18:37:00.000-05:00

The Error Messages page has gotten a major facelift. In particular, it now sports a list of common error messages as well as a corrected link to the Solaris 8 Common Messages manual.

Thanks to Bill and Laurie at Princeton for resolving a problem with the index page over the weekend.

--Scott

Updates for mdb, dtrace and diskio pages

2006-12-31T23:40:00.000-05:00

I just sent a pile of updates to Princeton. They will probably be added to the site on the 2nd.

The index page has been re-written to include pointers to some of the major sources used for the material on the site.

An Intermittent Problems page has been added to describe how to deal with intermittent problems.

The mdb, dtrace and Disk I/O pages have been rewritten and expanded. The adb and mdb pages were merged to reflect the fact that Solaris 7 is really a dead OS at this point in time. The Disk I/O page was updated to reflect current Solaris 10 information from Solaris Internals and Solaris Performance and Tools by McDougall, Mauro and Gregg, and especially to include pointers to the really cool tools on the DTrace Toolkit page.

The kstat page has been updated to provide some additional information, and the netstat page has been changed to reflect the death of netstat -k.

The next major effort for the site will be an expansion of the zones page. (The current page is really not much more than a placeholder to avoid dead links on the other pages that refer to zones.)

I am also working on a root cause analysis page. I am finding that this page is involving a lot of reading of business publications; the business community seems to be way ahead of us on thinking about this issue.

--Scott

mdb and kmdb pages

2006-12-26T19:31:00.000-05:00

I've submitted the following new pages to Princeton for inclusion: Intermittent Problems, mdb and kmdb. I also made significant improvements to the dtrace page.

dtrace, methodology, SMF

2006-12-21T20:44:00.000-05:00

I've submitted pages on
general methodology, dtrace and SMF. Depending on Princeton's work schedule, they may not be up until after the holidays.

Updated Load Average and ZFS Tuning

2006-12-20T20:56:00.000-05:00

I've posted updates to the discussion of Load Averages and ZFS Tuning.

SysAdmin article

2006-12-18T21:22:00.000-05:00

An expanded and rewritten version of the Resource Management page has been tentatively accepted by SysAdmin as for its April 2007 issue.

Updated pages

2006-12-18T21:19:00.000-05:00

I've submitted some updated pages for Resource Management, ZFS and Scheduling.

I also added a beginning of a page on Zones.

--Scott

ZFS and Resource Pools

2006-12-07T00:41:00.000-05:00

Additional pages for ZFS and Resource Pools have been submitted. The scheduler page has been expanded.

Sources for the ZFS page include the following:

Solaris ZFS Administration Guide

Brune, Corey, ZFS Administration, SysAdmin Magazine Jan 2007

Scheduling page

2006-12-05T20:56:00.000-05:00

I've added a new scheduling page. It is still a work in progress.

--Scott

Ishikawa and Interrelationship Diagrams

2006-12-03T22:10:00.000-05:00

I've been working on a page including information on some formal troubleshooting methods. En route, I have been looking at Cause-and-Effect (Ishikawa fishbone) diagrams and Interrelationship Diagrams.

Here are some of the noteworthy web pages I've been looking at:

Concordia: Cause and Effect Diagram and
Concordia Interrelationship Diagram provide a nice introduction to the two types of diagrams.

HCI Cause and Effect Diagram provides a slightly longer article, including some historical informaton about Ishikawa diagrams.

balancedscorecard.org Cause and Effect Diagram provides a much more in-depth view of Ishikawa diagrams.

questlearningskills.org Interrelationship Diagrams provides a howto level article about Interrelationship diagrams.

ASQ Interrelationship Diagrams provides a slightly longer article about Interrelationship Diagrams.

Root Cause Analysis: A Framework for Tool Selection provides a nice comparison of Ishikawa and Interrelationship diagrams, as well as Current Reality diagrams.

Resource Management resources

2006-11-30T08:05:00.000-05:00

I've been looking at documentation on Resource Management over the last few days. Here are some of the articles that I have found. Unfortunately, much of the information I found is based on the Solaris 9 and even Solaris 8 implementations of Resource Manager, which is only somewhat useful when looking at Solaris 10.

If you are aware of additional resources, please feel free to add them to the comments on this post.

Here are the best items I've found:

System Administration Guide: Solaris Containers-Resource Management and Solaris Zones from the Solaris 10 documentation. This is quite well-written, though organized differently than I would have done it.

The Sun BluePrints Guide to Solaris Containers by Foxwell, Lageman, Hoogeveen, Rozenfeld, Setty and Victor. The Resource Management section is also quite well-written, and I found the organization to be more helpful than the manual in the Solaris 10 docs.

Solaris Resource Management by Galvin in SysAdmin. This is a high-level introduction. Though it is specific to Solaris 9, it is still the best quick introduction to the subject I've come across.

Capping a Solaris processes memory by matty is a blog page describing the ability of Solaris 10 to use rcapd to manage memory. This is a brief but thorough discussion of this topic.

Introduction

2006-11-29T12:54:00.000-05:00

This blog is designed to be a companion to my Solaris Troubleshooting web site, hosted by Princeton University.

I used to have an email link to solicit feedback on the web site. I received some outstanding feedback, but I also received an outstanding amount of spam.

I am in the process of updating the site to include more Solaris 10 specific information, especially with regards to Resource Management and dtrace. I've posted a first cut at a Resource Management page.

Thanks to everyone who contributed to the old Solaris 8 site, and a special thanks to Princeton University for continuing to host the site long after I no longer worked on their Unix team.

--Scott Cromar