Notes for Week 4

The system log file is /var/log/messages, and security logging is done to /var/log/secure. Recent entries can be examined using "tail --lines 100 /var/log/messages". grep can be used to search the log file for recurring events of interest, such as reboots:
grep /var/log/messages -ie 'syslogd.*restart'
users logging in as root, or su-ing to become root:
grep /var/log/secure -ie 'root'
or hardware errors:
grep /var/log/messages -ie '(device file name)'
where "(device file name)" is, for example, hdd.

The kernel is essentially a large C program. The main program is responsible for interrogating the CPU, RAM, the I/O devices, the PCI bus, the Plug and Play BIOS, etc. The root partition is then mounted read-only (so that it can be checked without corrupting it) and the init program is started. init stays running as long as the system is up, and is the ultimate "ancestor" (parent, grandparent, etc.), of every process that runs.

The following output of the "pstree -npl" command illustrates this:

init(1)-+-migration/0(2)
        |-ksoftirqd/0(3)
        |-migration/1(4)
        |-ksoftirqd/1(5)
        |-events/0(6)
        |-events/1(7)
        |-khelper(8)
        |-kthread(9)-+-kblockd/0(12)
        |            |-kblockd/1(13)
        |            |-kacpid(14)
        |            |-khubd(90)
        |            |-pdflush(149)
        |            |-pdflush(150)
        |            |-aio/0(152)
        |            |-aio/1(153)
        |            |-kseriod(748)
        |            |-ata/0(817)
        |            |-ata/1(818)
        |            |-scsi_eh_0(820)
        |            |-scsi_eh_1(821)
        |            `-kpsmoused(894)
        |-kswapd0(151)
        |-kedac(895)
        |-kirqd(906)
        |-udevd(1010)
        |-syslogd(2102)
        |-klogd(2110)
        |-ntpd(2208)
        |-gpm(2226)
        |-lpd(2241)
        |-bash(2243)---startx(2272)---xinit(2288)-+-X(2289)
        |                                         `-sh(2317)-+-xscreensaver(2318)
        |                                                    `-wmaker(2319)---wmaker(2320)-+-endeavour2(2324)---emacs(10618)
        |                                                                                  |-xclock(2325)
        |                                                                                  |-xterm(2332)---luit(2333)---bash(2334)---pstree(10620)
        |                                                                                  `-seamonkey(10540)---run-mozilla.sh(10544)---seamonkey-bin(10549)-+-{seamonkey-bin}(10550)
        |                                                                                                                                                    |-{seamonkey-bin}(10551)
        |                                                                                                                                                    |-{seamonkey-bin}(10553)
        |                                                                                                                                                    |-{seamonkey-bin}(10556)
        |                                                                                                                                                    |-{seamonkey-bin}(10580)
        |                                                                                                                                                    |-{seamonkey-bin}(10581)
        |                                                                                                                                                    |-{seamonkey-bin}(10590)
        |                                                                                                                                                    |-{seamonkey-bin}(10591)
        |                                                                                                                                                    |-{seamonkey-bin}(10592)
        |                                                                                                                                                    `-{seamonkey-bin}(10593)
        |-agetty(2244)
        |-agetty(2245)
        |-agetty(2246)
        |-agetty(2247)
        `-agetty(2248)

The numbers in parentheses are "pids" (process identifiers); every process that is running has a unique pid whose maximum value is a function of RAM size (after the pids increment to the maximum, they start over again at the lowest unused number).

Some of the processes started by init are "daemons": programs that provide services to the operating system, it's users, or network clients. Their names typically end with a "d" (although not always), and are pronounced, for instance, "k"-"event"-"d". Some of them are self-explanatory, while others are not so:

pdflush helps handle writing of modified disk blocks back to the filesystem from the cache;
kswapd does essentially the same job for virtual memory pages and the swap partition;
khubd handles the USB hub (built in on most motherboards);

The daemons up to but not including syslogd are actually not separate programs, but are part of the kernel itself;

syslogd and klogd handle system and kernel logging (of error and status messages);
lpd is the print spooler daemon;
gpm is the general purpose mouse handler;
ntpd is the network time protocol daemon, which keeps the system clock synchronized to standard clocks on the Internet.

The monitor and keyboard directly attached to the PC which runs Linux is the "console". By entering Alt-F1 through Alt-F6, the console "operator" (as distinguished from any user which might telnet to Linux to use the system remotely) can switch between the "virtual consoles". The agetty processes listed above issue a login prompt to each of the virtual consoles; when a user enters a login name, the agetty program exits and passes that name to the login program, which asks for and validates the password, and then starts a shell (in this case bash), which supplies the command prompt. When the user executes the logout command, the login process exits and init starts another agetty.

The startx command is really a script which in turn runs the xinit program, which in turn runs X. X is the display "server", or driver, which switches the console to graphical mode and controls the display until X-Windows is exited. While X is running, the consoles are unavailable through the Alt-F# keys. X starts the window manager (in this case, wmaker, or WindowMaker), which controls drawing on the graphical display (windows, title bars, buttons, scroll bars, icons, etc.). There are a number of window managers available. Client programs running under the window manager include endeavour2 (a file manager), xclock, seamonkey and xterm. And of course, xterm emulates a terminal (telnet) session, and so starts a bash shell under which the pstree command whose output we have been analyzing was run.

The corresponding output of the "ps aux" command is

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1508   520 ?        S    Sep27   0:02 init [3]  
root         2  0.0  0.0      0     0 ?        S    Sep27   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Sep27   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Sep27   0:00 [migration/1]
root         5  0.0  0.0      0     0 ?        SN   Sep27   0:00 [ksoftirqd/1]
root         6  0.0  0.0      0     0 ?        S<   Sep27   0:00 [events/0]
root         7  0.0  0.0      0     0 ?        S<   Sep27   0:00 [events/1]
root         8  0.0  0.0      0     0 ?        S<   Sep27   0:00 [khelper]
root         9  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kthread]
root        12  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kblockd/0]
root        13  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kblockd/1]
root        14  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kacpid]
root        90  0.0  0.0      0     0 ?        S<   Sep27   0:00 [khubd]
root       149  0.0  0.0      0     0 ?        S    Sep27   0:00 [pdflush]
root       150  0.0  0.0      0     0 ?        S    Sep27   0:00 [pdflush]
root       151  0.0  0.0      0     0 ?        S    Sep27   0:00 [kswapd0]
root       152  0.0  0.0      0     0 ?        S<   Sep27   0:00 [aio/0]
root       153  0.0  0.0      0     0 ?        S<   Sep27   0:00 [aio/1]
root       748  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kseriod]
root       817  0.0  0.0      0     0 ?        S<   Sep27   0:00 [ata/0]
root       818  0.0  0.0      0     0 ?        S<   Sep27   0:00 [ata/1]
root       820  0.0  0.0      0     0 ?        S<   Sep27   0:00 [scsi_eh_0]
root       821  0.0  0.0      0     0 ?        S<   Sep27   0:00 [scsi_eh_1]
root       894  0.0  0.0      0     0 ?        S<   Sep27   0:00 [kpsmoused]
root       895  0.0  0.0      0     0 ?        S    Sep27   0:00 [kedac]
root       906  0.0  0.0      0     0 ?        S    Sep27   0:00 [kirqd]
root      1010  0.0  0.0   1580   500 ?        S< s Sep27   0:00 /sbin/udevd --daemon
root      2102  0.0  0.0   1560   568 ?        SNs  Sep27   0:00 syslogd -m 0
root      2110  0.0  0.0   2364  1324 ?        SNs  Sep27   0:00 klogd
root      2208  0.0  0.0   3976  1280 ?        SNs  Sep27   0:00 /usr/sbin/ntpd
root      2226  0.0  0.0   1716   368 ?        SNs  Sep27   0:00 /usr/sbin/gpm -m /dev/psaux -t ps2 
root      2241  0.0  0.0   3656   736 ?        SNs  Sep27   0:00 lpd Waiting  
ken       2243  0.0  0.0   4904  1484 tty1     Ss+  Sep27   0:00 -bash
root      2244  0.0  0.0   1504   496 tty2     Ss+  Sep27   0:00 /sbin/agetty -I \033(K tty2 9600
root      2245  0.0  0.0   1504   496 tty3     Ss+  Sep27   0:00 /sbin/agetty -I \033(K tty3 9600
root      2246  0.0  0.0   1508   500 tty4     Ss+  Sep27   0:00 /sbin/agetty -I \033(K tty4 9600
root      2247  0.0  0.0   1508   500 tty5     Ss+  Sep27   0:00 /sbin/agetty -I \033(K tty5 9600
root      2248  0.0  0.0   1504   496 tty6     Ss+  Sep27   0:00 /sbin/agetty -I \033(K tty6 9600
ken       2272  0.0  0.0   4856  1368 tty1     S+   Sep27   0:00 /bin/sh /usr/X11R6/bin/startx
ken       2288  0.0  0.0   2500   708 tty1     S+   Sep27   0:00 xinit /home/ken/.xinitrc -- -auth /home/ken/.serverauth.2272
root      2289  0.0  1.0  29540 22204 tty7     Rs+  Sep27   6:51 X :0 -auth /home/ken/.serverauth.2272
ken       2317  0.0  0.0   4856  1312 tty1     S    Sep27   0:00 sh /home/ken/.xinitrc
ken       2318  0.0  0.0   3824  2032 tty1     S    Sep27   0:03 xscreensaver -nosplash
ken       2319  0.0  0.0   7208  1128 tty1     S    Sep27   0:00 wmaker
ken       2320  0.0  0.1   8212  3668 tty1     S    Sep27   0:12 wmaker --for-real=
ken       2324  0.0  0.3  13636  7608 ?        Ss   Sep27   0:02 endeavour2
ken       2325  0.0  0.1   7156  2596 tty1     S    Sep27   0:00 xclock -strftime %a, %b %e - %l:%M %P -g 148x32+0-0 -digital -update 1
ken       2332  0.0  0.5  16160 10824 ?        Rs   Sep27   0:00 xterm -sb -sl 5000
ken       2333  0.0  0.0   4668   676 pts/2    Ss+  Sep27   0:00 /usr/X11R6/bin/luit
ken       2334  0.0  0.0   5264  2032 pts/3    Ss   Sep27   0:00 bash
ken      10540  0.0  0.0   4860  1400 ?        Ss   12:34   0:00 /bin/sh /usr/bin/seamonkey
ken      10544  0.0  0.0   4992  1420 ?        S    12:34   0:00 /bin/sh /usr/lib/seamonkey-1.0.1/run-mozilla.sh /usr/lib/seamonkey-1.0.1/seamonkey-bin
ken      10549  2.2  3.1 175288 64628 ?        Sl   12:34   0:23 /usr/lib/seamonkey-1.0.1/seamonkey-bin
ken      10618  0.4  0.3  12656  7792 ?        S    12:47   0:01 /usr/bin/emacs /home/ken/html/admin/week4.html
ken      10621  0.0  0.0   4040   852 pts/3    R+   12:51   0:00 ps aux

except that the process 10620 executing the pstree command in the previous output has been replaced by the process 10621 executing the ps command whose output appears above.

The columns are interpreted as follows:

USER = the login account under which the process is running;
PID = the process ID;
%CPU = the CPU time used by the process divided by the time the process has been running;
%MEM = the RAM used by the process divided by the total RAM available;
VSZ = the total virtual memory size of the process in Kb;
RSS = the size in Kb of the resident set (portion currently residing in RAM);
TTY = where text output goes (ttyn = virtual console, pts/n = xterm window)
STAT = process state
R = runnable (on the run queue)
S = sleeping
T = traced or stopped
W = paging
Z = a zombie process, which has terminated but its parent has not cleaned it up properly; init will do this when the parent process exits
START = clock time when process started
TIME = CPU time used to date
COMMAND = process command line

Note that there is no agetty for tty1; this is the virtual console in which all of the processes in the current interactive login session are running. Also note that X runs as root, although it were started by ken; X must run as root in order to control the display.

Information about processes running on the system is kept in the /proc directory. This is really a "pseudo-filesystem": it only exists in memory. It provides a filesystem interface to kernel data. Each process has its own directory whose name is /proc/pid (ie., the wmaker process in the example above would be located in the directory /proc/2320). In that directory is (among other things):
- the command line used to start the process (cat /proc/pid/cmdline);
- the current working directory of the process (ls -l /proc/pid/cwd);
- the process' environment variables (cat /proc/pid/environ | tr "\000" "\n");
- the program which is running (ls -l /proc/pid/exe);
- the files which the process currently has open (ls -l /proc/pid/fd);
- the root of the file system accessible to the process (ls -l /proc/pid/root) (see the chroot command);
- the process status (cat /proc/pid/status).
In addition, information about the hardware and operating system can be obtained by:
- cat /proc/bus/usb/devices (pay attention to currents and speeds)
- cat /proc/cmdline
- gzip -cd /proc/config.gz | less (this lists the kernel parameters for the currently running kernel)
- cat /proc/cpuinfo
- cat /proc/diskstats (see /usr/doc/linux-2.6.16.27/iostats.txt)
- cat /proc/ide/hda/model
- cat /proc/ide/hda/settings
- cat /proc/interrupts
- cat /proc/locks
- cat /proc/meminfo (see /usr/doc/linux-2.6.16.27/filesystems/proc.txt)
- cat /proc/mounts
- cat /proc/scsi/scsi
- cat /proc/stat (see /usr/doc/linux-2.6.16.27/filesystems/proc.txt)
- cat /proc/uptime (see /usr/doc/linux-2.6.16.27/filesystems/proc.txt)
- cat /proc/version
You can also find information on your pci devices in /proc/bus/pci/devices, but the lspci command is much more helpful.

In general, stay out of /proc/sys - unless you really know what you're doing, it is extremely easy to shoot yourself in the foot by changing parameters on the running kernel. That said, there are two parameters that I like to change at startup:
echo 1 > /proc/sys/vm/dirty_background_ratio
echo 1 > /proc/sys/vm/dirty_ratio
These cause pdflush to run more frequently, which I find spreads disk i/o out, reducing the times when the system seems to pause while buffers are being flushed. Your mileage may vary...

kill can be used to send signals to processes (see the signal man page in section 7 for the meanings of the various signals). Many daemons will re-load their configuration files when they receive a SIGHUP.

Please send comments or suggestions to the author.