Sun System Bus Sizing

This is an older posting I made to the original Solaris Troubleshooting site. The information is a little dated in terms of available hardware options, but the concepts and philosophy are the same.

The system bus has a fixed bandwidth. Too many devices on the bus can create more traffic than the bus can handle, which results in contention and packet loss.

prtdiag -v reports on many system bus configuration issues. It is possible to see if the bus is overloaded by adding up the reported capacities of the devices on a bus and seeing if they exceed the capacity of the bus.

Where possible, similar cards can be placed on the same board so that the interrupts are directed to the same CPU (and associated caches).

This is a table of typical system bus capacities:

Bus	Speed	Width	Burst Bandwidth	Sustained Bandwidth
MBus	33MHz	64 bit	264 MB/s	86 MB/s
MBus	36MHz	64 bit	288 MB/s	94 MB/s
MBus	40MHz	64 bit	320 MB/s	105 MB/s
MBus	50MHz	64 bit	400 MB/s	130 MB/s
XDBus	40MHz	64 bit	320 MB/s	250 MB/s
XDBus	50MHz	64 bit	400 MB/s	312 MB/s
UPA	72MHz	128 bit	1.15 GB/s	1 GB/s
UPA	83.5MHz	128 bit	1.3 GB/s	1.2 GB/s
UPA	100MHz	128 bit	1.5 GB/s	1.44 GB/s
Gigaplane	83.5MHz	256 bit	2.6 GB/s	2.5 GB/s
GigaplaneXB	100MHz	1024 bit	12.8 GB/s	12.8 GB/s

Peripheral Buses

The two peripheral buses on Sun systems are Sbus and PCI bus. Sbus runs at 20-25MHz and comes in 32 or 64 bit sizes. Peak Sbus bandwidth is 200 MB/s. The venerable Sbus has been retired in favor of the newer PCI bus as PCI bus performance has improved.

PCI buses runs at 33 or 66MHz and may be 32 or 64 bit. The peak PCI bus bandwidth is 528 MB/s for 64-bit buses at 66MHz.

(For desktop PC hardware, 33MHz PCI buses are still common. 33 MHz buses have peak bandwidths of 264 MB/s for 64-bit and 132 MB/s for 32-bit.)

Newer PCI-x buses run at 133MHz and allow up to 1066 MB/s. PCI-x 2.0 defines clock rates of 266MHz and 533MHz, with peak bandwidths of 2.1 GB/s and 4.2 GB/s, respectively.

Low profile PCI buses are becoming more common, since their smaller form factor fits well with the increasing miniaturization of the system. Low profile PCI comes in MD1 and MD2 flavors, with the primary difference being the shorter length of the MD1 cards. Currently, they do not support 64-bit PCI extensions.

Mini PCI cards are also produced for use in portable and sealed case computers. They are small in size, do not support 64-bit extensions and have a different connector layout, but still otherwise follow the PCI standard.

SCSI Bus

SCSI buses can operate at one of these speeds:

• 4 MB/s (asynchronous)
• 5 MB/s (synchronous)
• 10 MB/s (fast)
• 20 MB/s (ultra, fast/wide or fast-20)
• 40 MB/s (ultra/wide or narrow ultra-2)
• 80MB/s (wide ultra-2)
• 160 MB/s (ultra-3 or ultra-160)
• 320 MB/s (ultra-320)

SCSI buses and devices negotiate speed between the controller and the devices on the chain. prtconf can report information that can be used to determine the speed of a particular device.

The scsi_options parameter can be set in the /etc/system file to limit bus speed or set other characteristics. Check device documentation to determine if these settings need to be specified.

SCSI chains may be made of single-ended (SE) or differential connections. Differential connections come in low voltage (LVD) and high voltage (HVD) variants. SE, LVD, and HVD should not be mixed, as this may damage the equipment.

Differential connections permit longer chains, but the hardware is usually more expensive. Single-ended chains must be less than 6m in length; LVD chains must be less than 12m in length; HVD chains must be less than 20m for synchronous connections or 25m for asynchronous connections. (Remember that chain length includes the length of the connectors and cabling in the devices, not just the external cable.)

Starting with Ultra 2, only differential connections are available. Only LVD is available for Ultra160 or Ultra320.

The SCSI target numbers represent attachment points on the SCSI chain. Each target number may include as many as 8 devices (luns or logical unit numbers). Embedded SCSI devices only include one lun.

Higher target numbers receive better service. On a narrow bus, the target priorities run 7 -> 0. On a wide bus, they run 7 -> 0, then 15 -> 8. The host adapter is usually 7. This can cause problems where busy disks and tape devices share a SCSI bus, since tape devices are usually assigned target 6.