Sunday, April 07, 2013

Sun System Bus Sizing

This is an older posting I made to the original Solaris Troubleshooting site. The information is a little dated in terms of available hardware options, but the concepts and philosophy are the same.

The system bus has a fixed bandwidth. Too many devices on the bus can create more traffic than the bus can handle, which results in contention and packet loss.

prtdiag -v reports on many system bus configuration issues. It is possible to see if the bus is overloaded by adding up the reported capacities of the devices on a bus and seeing if they exceed the capacity of the bus.

Where possible, similar cards can be placed on the same board so that the interrupts are directed to the same CPU (and associated caches).

This is a table of typical system bus capacities:

Bus Speed Width Burst
Bandwidth
Sustained
Bandwidth
MBus 33MHz 64 bit 264 MB/s 86 MB/s
MBus 36MHz 64 bit 288 MB/s 94 MB/s
MBus 40MHz 64 bit 320 MB/s 105 MB/s
MBus 50MHz 64 bit 400 MB/s 130 MB/s
XDBus 40MHz 64 bit 320 MB/s 250 MB/s
XDBus 50MHz 64 bit 400 MB/s 312 MB/s
UPA 72MHz 128 bit 1.15 GB/s 1 GB/s
UPA 83.5MHz 128 bit 1.3 GB/s 1.2 GB/s
UPA 100MHz 128 bit 1.5 GB/s 1.44 GB/s
Gigaplane 83.5MHz 256 bit 2.6 GB/s 2.5 GB/s
GigaplaneXB 100MHz 1024 bit 12.8 GB/s 12.8 GB/s

Peripheral Buses

The two peripheral buses on Sun systems are Sbus and PCI bus. Sbus runs at 20-25MHz and comes in 32 or 64 bit sizes. Peak Sbus bandwidth is 200 MB/s. The venerable Sbus has been retired in favor of the newer PCI bus as PCI bus performance has improved.

PCI buses runs at 33 or 66MHz and may be 32 or 64 bit. The peak PCI bus bandwidth is 528 MB/s for 64-bit buses at 66MHz.

(For desktop PC hardware, 33MHz PCI buses are still common. 33 MHz buses have peak bandwidths of 264 MB/s for 64-bit and 132 MB/s for 32-bit.)

Newer PCI-x buses run at 133MHz and allow up to 1066 MB/s. PCI-x 2.0 defines clock rates of 266MHz and 533MHz, with peak bandwidths of 2.1 GB/s and 4.2 GB/s, respectively.

Low profile PCI buses are becoming more common, since their smaller form factor fits well with the increasing miniaturization of the system. Low profile PCI comes in MD1 and MD2 flavors, with the primary difference being the shorter length of the MD1 cards. Currently, they do not support 64-bit PCI extensions.

Mini PCI cards are also produced for use in portable and sealed case computers. They are small in size, do not support 64-bit extensions and have a different connector layout, but still otherwise follow the PCI standard.

SCSI Bus

SCSI buses can operate at one of these speeds:
  • 4 MB/s (asynchronous)
  • 5 MB/s (synchronous)
  • 10 MB/s (fast)
  • 20 MB/s (ultra, fast/wide or fast-20)
  • 40 MB/s (ultra/wide or narrow ultra-2)
  • 80MB/s (wide ultra-2)
  • 160 MB/s (ultra-3 or ultra-160)
  • 320 MB/s (ultra-320)
SCSI buses and devices negotiate speed between the controller and the devices on the chain. prtconf can report information that can be used to determine the speed of a particular device.

The scsi_options parameter can be set in the /etc/system file to limit bus speed or set other characteristics. Check device documentation to determine if these settings need to be specified.

SCSI chains may be made of single-ended (SE) or differential connections. Differential connections come in low voltage (LVD) and high voltage (HVD) variants. SE, LVD, and HVD should not be mixed, as this may damage the equipment.

Differential connections permit longer chains, but the hardware is usually more expensive. Single-ended chains must be less than 6m in length; LVD chains must be less than 12m in length; HVD chains must be less than 20m for synchronous connections or 25m for asynchronous connections. (Remember that chain length includes the length of the connectors and cabling in the devices, not just the external cable.)

Starting with Ultra 2, only differential connections are available. Only LVD is available for Ultra160 or Ultra320.

The SCSI target numbers represent attachment points on the SCSI chain. Each target number may include as many as 8 devices (luns or logical unit numbers). Embedded SCSI devices only include one lun.

Higher target numbers receive better service. On a narrow bus, the target priorities run 7 -> 0. On a wide bus, they run 7 -> 0, then 15 -> 8. The host adapter is usually 7. This can cause problems where busy disks and tape devices share a SCSI bus, since tape devices are usually assigned target 6.

No comments: