Notes for Week 9

We begin by summarizing how our linux distribution was built. This should give you a better feel for how the various parts of a distribution fit together.

  1. There are three general "models" for software distribution:

    • The "proprietary" model: all software is built by the vendor and is shipped as executable code; patches are self-extracting and self-installing. Patches which are damaged during download can necessitate software reinstallation and this model allows minimal tailoring of the software to individual needs (ie., inclusion or exclusion of software components based on local requirements, and modification of system operating parameters), but in general this model is easiest to use. No source code is available to software licensees.

    • The "system generation" (or "sysgen") model: software is compiled by the vendor and is shipped either as object modules or as executables; more tailoring is generally possible and all patches are applied during sysgen, which is required before the software can be used. Source code is generally not supplied by the vendor. This was the favored approach of many mainframe vendors.

    • The "open source" model: all software and patches are released in source, and extensive tailoring is generally possible, even encouraged. Users can learn everything there is to know about the system if they are good C and/or assembly language programmers and once they know how the system works, can even make modifications or improvements to the system. These user-originated changes are often made available in future releases. Popular examples are Linux and FreeBSD.

  2. To build a linux system, you need a pre-existing system with (minimally) the following software:

    • bash (or another shell)

    • binutils (the assembler, linker and related software)

    • bzip2 (compression / decompression software, since many packages are distributed as .bz2 files)

    • coreutils (many linux command executables, including the following that we have used: cat, chgrp, chmod, chown, cp, dd, df, du, echo, id, ln, ls, md5sum, mkdir, mknod, mv, od, printenv, rm, rmdir, sleep, su, sync, tail, touch)

    • diffutils (cmp and diff)

    • findutils (find, etc.)

    • gawk (GNU awk, a C-like scripting language named after its original authors: Drs. Alfred Aho, Peter Weinberger and Brian Kernighan of Bell Labs)

    • gcc (the C and C++ compilers; in addition to the latest version, an older gcc is often required to reliably build the linux kernel)

    • gettext (utilities for Native Language Support)

    • glibc (libraries of subroutines used by C programs, documented in man page sections 2 and 3)

    • grep (global regular expression print, your friend)

    • gzip (compression / decompression software, since many packages are distributed as .gz files; gzip is not quite as effective as bzip2, but is generally much faster)

    • the linux kernel (the header files describing system interfaces and capabilities are required by most software packages)

    • m4 (a macro processor)

    • make (used for practically all software building)

    • ncurses (used for command line I/O)

    • patch (used to install patches, which are essentially diff output from comparing old and new source files)

    • perl (Practical Extraction and Reporting Language, a ubiquitous scripting language)

    • sed (stream editor, essentially an editor which is a filter)

    • tar (since most software packages are distributed as zipped tarballs: compressed archives containing many files and directories)

    • texinfo (info and related utilities)

    • util-linux (additional UNIX utilities, including fdformat, fdisk, hwclock, logger, mkswap, more, mount, umount)

    And if you want to run the tests on binutils and gcc (which are highly recommended), you need

    • dejagnu (the testing framework of the GNU project)

    • expect (which allows scripting of interactive programs)

    • tcl (Tool Command Language, a popular scripting language)

    All versions of all installed software must be inter-operable.

    This is a highly nontrivial exercise. Creating a linux distribution involves a great deal of effort in verifying inter-operability. This is a major contribution of the Linux From Scratch project. In general, you must read the README files associated with each package to determine which versions of other packages are required for correct operation.

  3. Being able to build a linux system is one thing; being able to run one is another. To have an operating build environment you must build and install:

    • boot scripts (in /etc/rc.d/init.d)

    • grub (the GRand Unified Boot loader, or LILO, the LInux LOader)

    • sysklogd (syslogd and klogd)

    • sysvinit (init, halt, reboot, shutdown, telinit, etc.)

    • udev (dynamic device file allocation and deallocation)

    and you must create a number of directories and files:

    • /bin, /boot, /dev, /etc, /home, /lib, /media, /mnt, /opt, /proc, /root, /sbin, /srv, /sys, /tmp, /usr, /var

    • /etc/opt, /usr/bin, /usr/include, /usr/lib, /usr/sbin, /usr/share, /usr/src
      /usr/local/bin, /usr/local/include, /usr/local/lib, /usr/local/sbin, /usr/local/share, /usr/local/src

    • /usr/share/doc, /usr/share/info, /usr/share/locale, /usr/share/man, /usr/share/man1-8, , /usr/share/misc, /usr/share/terminfo, /usr/share/zoneinfo
      /usr/local/share/doc, /usr/local/share/info, /usr/local/share/locale, /usr/local/share/man, /usr/local/share/man1-8, , /usr/local/share/misc, /usr/local/share/terminfo, /usr/local/share/zoneinfo

    • /var/cache, /var/lib, /var/local, /var/lock, /var/log, /var/mail, /var/opt, /var/run, /var/spool /var/lib/misc, /var/lib/locate

    • /bin/sh

    • /dev/console and /dev/null (the rest of /dev will be populated when we mount the dev pseudo-filesystem)

    • /etc/fstab, /etc/group, /etc/inittab, /etc/inputrc, /etc/ld.so.conf (see below), /etc/mtab, /etc/nsswitch.conf (used by glibc), /etc/passwd, /etc/profile, /etc/resolv.conf and /etc/syslog.conf

    • /etc/sysconfig/clock, /etc/sysconfig/network and /etc/sysconfig/network-devices/ifconfig.eth0/ipv4

    • /var/run/utmp, /var/log/btmp, /var/log/lastlog and /var/log/wtmp (all used by login)

  4. Now that you have an operating build system, your first step is to re-build all of the above software. This verifies the integrity of your build environment. Then you need to add the software that makes it a usable UNIX environment.

    A usable linux system includes the following additional packages:

    • autoconf (used for auto-configuration scripts)

    • automake (used to create make files, which are input to make)

    • bison (a parser generator)

    • db (data base handler for system data bases)

    • ed and one or more additional editors (ie, emacs, joe, vim)

    • e2fsprogs (including e2fsck and mke2fs)

    • file (the command line program to determine file types)

    • flex (generates programs for text pattern matching)

    • groff (used to format man pages)

    • iana-etc (/etc/protocols and /etc/services)

    • inetutils (ftp, ping, telnet, etc.)

    • iproute2

    • kbd (keyboard utilities)

    • less (is more...)

    • libtool (library support utilities)

    • man-db and the man pages (which you all know and love)

    • mktemp (used to make temporary files for scripts)

    • module-init-tools (module utilities such as insmod, lsmod, modprobe, rmmod)

    • net-tools (arp, ifconfig, route, etc.)

    • procps (kill, ps, etc.)

    • psmisc (pstree, etc.)

    • readline

    • shadow (user and group utilities such as useradd, usermod, groupadd, passwd)

    • zlib (a library of compression subroutines)

  5. Finally, a modern linux system should include the following software (and of course, their dependencies):

    • ALSA (the Advanced Linux Sound Architecture software)

    • ghostscript (so you can use postscript for printing)

    • giflib (library routines for manipulating gif files)

    • iptables (for firewall maintenance)

    • libjpeg (library routines for manipulating jpeg files)

    • libusb

    • LPRng (or CUPS, for print spooling)

    • mplayer (or some other multimedia player)

    • seamonkey (or some other web browser)

    • ntp (network time protocol daemon)

    • openssh (ssh and sftp)

    • openssl (secure socket layer support)

    • WindowMaker (or some other window manager)

    • Xorg (or XFree86; commonly referred to as X; the build of X is intimately involved with the kernel, so don't try this one for your first software build experience!)

    And of course anything you need to enjoy your system. Since a linux system is expected to grow as your needs do, it is always a good idea to build as many libraries as you can, so that when you want to install a new piece of software, it is likely that you have the libraries it requires. A complete list of the commands used to install the software in our distribution is in /root/lfs-6.2.in and /root/blfs-6.2.in.

  6. One of the most important things to do before building new software is to have a good backup. While most software builds are bug-free, it is always possible, especially during the installation phase, that a faulty install script might modify or remove files used by other packages.

  7. Software builds usually include the following steps, although depending on the package, one or more may be omitted or replaced. Always read the INSTALL file to make sure you are installing the software correctly!

    1. Download and un-tar the software.

    2. cd to the software directory.

    3. Install any required patches.

    4. ./configure - this performs a thorough check of the current system capabilities, including what software is already installed, and configures the build process accordingly. If a package requires or optionally uses other software, that software must be installed before configure will find it. Sometimes it needs a little help... Use "./configure --help" to list all of the configuration options available for this package.

      Note that configure will determine your CPU type and build the software for that processor. Since not all processors have the same capabilities, you should build the software on the same type of CPU, or a lower class of compatible CPU, that you intend to run it on. For instance, software built on a Pentium will run on any class of Pentium, but software built on a Pentium II will only run on a Pentium II or better. It is possible to build for a different processor, but that is outside the scope of this class.

      Be sure to check the output of configure to make sure the package will be built to your needs; some of these builds are LONG!

    5. make (sometimes more than one, with various parameters)

    6. make install (which must be done as root, on a writable filesystem).

    7. Create post-installation configuration files.

    It is a good idea to keep a record of exactly which commands were used to build each package, and a log of the build output. This can be done as follows:
    (./configure) 2>&1 | tee package.log

    (make && make install) 2>&1 | tee -a package.log

    The "2>&1" will redirect stderr to stdout, and the tee will allow you to see the output as it is written to the log file (the "-a" tells tee to append to the log file). The "&&" ensures that the "make install" will not run if the "make" fails. Of course, you should not do either if there were problems with the configure.

  8. Most packages make use of the shared libraries (".so" files). You can build statically-linked software, in which the required library routines are copied into the executable program, if you intend for the software to be used on a system which may not have the required libraries available. This makes the executables very big.

    If you install software which includes new libraries, the install process will use ldconfig to rebuild the runtime linkages. /etc/ld.so.conf contains a list of the directories in which the dynamic loader is to look for libraries. Any library directories other than /lib and /usr/lib must be included in /etc/ld.so.conf. ldd can be used to determine which libraries are required by a given program.

  9. If you are running short on disk space, you can reduce the size of binary executables and libraries using
    strip --strip-debug (library file name)

    strip --strip-unneeded (binary executable file name)

    Be sure NOT to do the latter on libraries!

  10. During the build of our distribution, several additional parameters and steps were performed:

    • as noted in week 1, the filesystem was organized so that all writeable files reside under the /var directory;

    • md5sums were checked on all downloads where they were available, and all make output was scanned for errors that are easily missed as the output scrolls down the screen;

    • the build was performed in a chroot environment (an isolated filesystem environment);

    • the order of the BLFS software build was chosen in an attempt to satisfy broadly-required prerequisites first;

    • gcc-3 was built in /opt because a few non-BLFS packages won't compile under the more demanding gcc-4;

    • a number of non-BLFS packages were included;

    • /etc/rc.d/init.d/configure and /etc/rc.d/init.d/varupd were added for DVD-ROM operation; and

    • compressdoc was used to gzip the man and info pages.

    A number of small configuration changes and fixes were made; see the comments in /root/lfs-6.2.in and /root/blfs-6.2.in.

  11. EXERCISES for Week 9:

    1. Test mplayer by playing a CD (mplayer cdda://1) and listening to Internet Radio (mplayer -playlist http://www.wvxu.org/listen/WVXULive.asx).

    2. You can burn a data CD using the commands 

      mkisofs -l -R -allow-leading-dots -o output.iso (input directory)
cdrecord -v dev=ATAPI:0,0,0 -data output.iso

      The mkisofs creates an ISO-9660 filesystem in the file output.iso, which is then burned using cdrecord.

      To burn a music CD, simply cd to a directory with wav files to be burned (make sure they will fit!), and 

      cdrecord -v dev=ATAPI:0,0,0 -audio -pad *.wav

    3. To burn a DVD, put everything you want to burn in a directory called dvdir. Then
      growisofs -dvd-compat -Z /dev/hda=dvdir
      If you have blank DVDs which are rated at a speed slower than your burner, you will want to include the option "-speed=1" to force the speed to a lower value (it won't actually burn at 1X, but it should burn slowly enough to work).

    4. Use ldd to find out which libraries are required by login, bash and seamonkey (check all running seamonkey executables).

    5. Follow the BLFS instructions to install HTML Tidy. Be sure to test it!

©2006, Kenneth R. Koehler. All Rights Reserved. This document may be freely reproduced provided that this copyright notice is included.

Please send comments or suggestions to the author.