Solaris Process Scheduling

In Solaris, highest priorities are scheduled first. Kernel thread scheduling information can be revealed with ps -elcL.

A process can exist in one of the following states: running, sleeping or ready.

Kernel Threads Model

The Solaris 10 kernel threads model consists of the following major objects:

• kernel threads: This is what is scheduled/executed on a processor
• user threads: The user-level thread state within a process.
• process: The object that tracks the execution environment of a program.
• lightweight process (lwp): Execution context for a user thread. Associates a user thread with a kernel thread.

In the Solaris 10 kernel, kernel services and tasks are executed as kernel threads. When a user thread is created, the associated lwp and kernel threads are also created and linked to the user thread.

(This single-level model was first introduced in Solaris 8's alternative threads library, which was made the default in Solaris 9. Prior to that, user threads had to bind to an available lwp before becoming eligible to run on the processor.)

Priority Model

The Solaris kernel is fully preemptible. This means that all threads, including the threads that support the kernel's own activities, can be deferred to allow a higher- priority thread to run.

Solaris recognizes 170 different priorities, 0-169. Within these priorities fall a number of different scheduling classes:

• TS (timeshare): This is the default class for processes and their associated kernel threads. Priorities within this class range 0-59, and are dynamically adjusted in an attempt to allocate processor resources evenly.
• IA (interactive): This is an enhanced version of the TS class that applies to the in-focus window in the GUI. Its intent is to give extra resources to processes associated with that specific window. Like TS, IA's range is 0-59.
• FSS (fair-share scheduler): This class is share-based rather than priority- based. Threads managed by FSS are scheduled based on their associated shares and the processor's utilization. FSS also has a range 0-59.
• FX (fixed-priority): The priorities for threads associated with this class are fixed. (In other words, they do not vary dynamically over the lifetime of the thread.) FX also has a range 0-59.
• SYS (system): The SYS class is used to schedule kernel threads. Threads in this class are "bound" threads, which means that they run until they block or complete. Priorities for SYS threads are in the 60-99 range.
• RT (real-time): Threads in the RT class are fixed-priority, with a fixed time quantum. Their priorities range 100-159, so an RT thread will preempt a system thread.

Of these, FSS and FX were implemented in Solaris 9. (An extra-cost option for Solaris 8 included the SHR (share-based) class, but this has been subsumed into FSS.)

Fair Share Scheduler

The default Timesharing (TS) scheduling class in Solaris attempts to allow each process on the system to have relatively equal CPU access. The nice command allows some management of process priority, but the new Fair Share Scheduler (FSS) allows more flexible process priority management that integrates with the project framework.

Each project is allocated a certain number of CPU shares via the project.cpu-shares resource control. Each project is allocated CPU time based on its cpu-shares value divided by the sum of the cpu-shares values for all active projects.

Anything with a zero cpu-shares value will not be granted CPU time until all projects with non-zero cpu-shares are done with the CPU.

The maximum number of shares that can be assigned to any one project is 65535.

FSS can be assigned to processor sets, resulting in more sensitive control of priorities on a server than raw processor sets. The dispadmin command command controls the assignment of schedulers to processor sets, using a form like:
dispadmin -d FSS
To enable this change now, rather than after the next reboot, run a command like the following:
priocntl -s -C FSS
priocntl can control cpu-shares for a project:
priocntl -r -n project.cpu-shares -v number-shares -i project project-name

The Fair Share Scheduler should not be combined with the TS, FX (fixed-priority) or IA (interactive) scheduling classes on the same CPU or processor set. All of these scheduling classes use priorities in the same range, so unexpected behavior can result from combining FSS with any of these. (There is no problem, however, with running TS and IA on the same processor set.)

To move a specific project's processes into FSS, run something like:
priocntl -s -c FSS -i projid project-ID

All processes can be moved into FSS by first converting init, then the rest of the processes:
priocntl -s -c FSS -i pid 1
priocntl -s -c FSS -i all

Implementation Details

Time Slicing for TS and IA

TS and IA scheduling classes implement an adaptive time slicing scheme that increases the priority of I/O-bound processes at the expense of compute-bound processes. The exact values that are used to implement this can be found in the dispatch table. To examine the TS dispatch table, run the command dispadmin -c TS -g. (If units are not specified, dispadmin reports time values in ms.)

The following values are reported in the dispatch table:

• ts_quantum: This is the default length of time assigned to a process with the specified priority.
• ts_tqexp: This is the new priority that is assigned to a process that uses its entire time quantum.
• ts_slpret: The new priority assigned to a process that blocks before using its entire time quantum.
• ts_maxwait: If a thread does not receive CPU time during a time interval of ts_maxwait, its priority is raised to ts_lwait.
• ts_lwait:

The man page for ts_dptbl contains additional information about these parameters.

dispadmin can be used to edit the dispatch table to affect the decay of priority for compute-bound processes or the growth in priority for I/O-bound processes. Obviously, the importance of the different types of processing on different systems will make a difference in how these parameters are tweaked. In particular, ts_maxwait and ts_lwait can prevent CPU starvation, and raising ts_tqexp slightly can slow the decline in priority of CPU-bound processes.

In any case, the dispatch tables should only be altered slightly at each step in the tuning process, and should only be altered at all if you have a specific goal in mind.

The following are some of the sorts of changes that can be made:

• Decreasing ts_quantum favors IA class objects.
• Increasing ts_quantum favors compute-bound objects.
• ts_maxwait and ts_lwait control CPU starvation.
• ts_tqexp can cause compute-bound objects' priorities to decay more or less rapidly.
• ts_slpret can cause I/O-bound objects' priorities to rise more or less rapidly.

RT objects time slice differently in that ts_tqexp and ts_slpret do not increase or decrease the priority of the

IA objects add 10 to the regular TS priority of the process in the active window. This priority shifts with the focus on the active window. object. Each RT thread will execute until its time slice is up or it is blocked while waiting for a resource.

Time Slicing for FSS

In FSS, the time quantum is the length of time that a thread is allowed to run before it has to release the processor. This can be checked using
dispadmin -c FSS -g

The QUANTUM is reported in ms. (The output of the above command displays the resolution in the RES parameter. The default is 1000 slices per second.) It can be adjusted using dispadmin as well. First, run the above command and capture the output to a text file (filename.txt). Then run the command:
dispadmin -c FSS -s filename.txt

Callouts

Solaris handles callouts with a callout thread that runs at maximum system priority, which is still lower than any RT thread. RT callouts are handled separately and are invoked at the lowest interrupt level, which ensures prompt processing.

Priority Inheritance

Each thread has two priorities: global priority and inherited priority. The inherited priority is normally zero unless the thread is sitting on a resource that is required by a higher priority thread.

When a thread blocks on a resource, it attempts to "will" or pass on its priority to all threads that are directly or indirectly blocking it. The pi_willto() function checks each thread that is blocking the resource or that is blocking a thread in the syncronization chain. When it sees threads that are a lower priority, those threads inherit the priority of the blocked thread. It stops traversing the syncronization chain when it hits an object that is not blocked or is higher priority than the willing thread.

This mechanism is of limited use when considering condition variable, semaphore or read/write locks. In the latter case, an owner-of-record is defined, and the inheritance works as above. If there are several threads sharing a read lock, however, the inheritance only works on one thread at a time.

Thundering Herd

When a resource is freed, all threads awaiting that resource are woken. This results in a footrace to obtain access to that object; one succeeds and the others return to sleep. This can lead to wasted overhead for context switches, as well as a problem with lower priority threads obtaining access to an object before a higher-priority thread. This is called a "thundering herd" problem.

Priority inheritance is an attempt to deal with this problem, but some types of syncronization do not use inheritance.

Turnstiles

Each syncronization object (lock) contains a pointer to a structure known as a turnstile. These contain the data needed to manipulate the syncronization object, such as a queue of blocked threads and a pointer to the thread that is currently using the resource. Turnstiles are dynamically allocated based on the number of allocated threads on the system. A turnstile is allocated by the first thread that blocks on a resource and is freed when no more threads are blocked on the resource.

Turnstiles queue the blocked threads according to their priority. Turnstiles may issue a signal to wake up the highest-priority thread, or they may issue a broadcast to wake up all sleeping threads.

Adjusting Priorities

The priority of a process can be adjusted with priocntl or nice, and the priority of an LWP can be controlled with priocntl().

Real Time Issues

STREAMS processing is moved into its own kernel threads, which run at a lower priority than RT threads. If an RT thread places a STREAMS request, it may be serviced at a lower priority level than is merited.

Real time processes also lock all their pages in memory. This can cause problems on a system that is underconfigured for the amount of memory that is required.

Since real time processes run at such a high priority, system daemons may suffer if the real time process does not permit them to run.

When a real time process forks, the new process also inherits real time privileges. The programmer must take care to prevent unintended consequences. Loops can also be hard to stop, so the programmer also needs to make sure that the program does not get caught in an infinite loop.

Interrupts

Interrupt levels run between 0 and 15. Some typical interrupts include:

• soft interrupts
• SCSI/FC disks (3)
• Tape, Ethernet
• Video/graphics
• clock() (10)
• serial communications
• real-time CPU clock
• Nonmaskable interrupts (15)