Notes for Week 9

We begin by summarizing how our linux distribution was built. This should give you a better feel for how the various parts of a distribution fit together.

  1. There are three general "models" for software distribution:

    • The "proprietary" model: all software is built by the vendor and is shipped as executable code; patches are self-extracting and self-installing. Patches which are damaged during download can necessitate software reinstallation and this model allows minimal tailoring of the software to individual needs (ie., inclusion or exclusion of software components based on local requirements, and modification of system operating parameters), but in general this model is easiest to use. No source code is available to software licensees.

    • The "system generation" (or "sysgen") model: software is compiled by the vendor and is shipped either as object modules or as executables; more tailoring is generally possible and all patches are applied during sysgen, which is required before the software can be used. Source code is generally not supplied by the vendor. This was the favored approach of many mainframe vendors.

    • The "open source" model: all software and patches are released in source, and extensive tailoring is generally possible, even encouraged. Users can learn everything there is to know about the system if they are good C and/or assembly language programmers and once they know how the system works, can even make modifications or improvements to the system. These user-originated changes are often made available in future releases. Popular examples are Linux and FreeBSD.

  2. To build a linux system, you need a pre-existing system with (minimally) the following software:

    • bash (or another shell)

    • binutils (the assembler, linker and related software)

    • bzip2 (compression / decompression software, since many packages are distributed as .bz2 files)

    • coreutils (many linux command executables, including the following that we have used: cat, chgrp, chmod, chown, cp, dd, df, du, echo, id, ln, ls, md5sum, mkdir, mknod, mv, od, printenv, rm, rmdir, sleep, su, sync, tail, touch)

    • diffutils (cmp and diff)

    • findutils (find, etc.)

    • gawk (GNU awk, a C-like scripting language named after its original authors: Drs. Alfred Aho, Peter Weinberger and Brian Kernighan of Bell Labs)

    • gcc (the C and C++ compilers; in addition to the latest version, an older gcc is often required to reliably build the linux kernel)

    • gettext (utilities for Native Language Support)

    • glibc (libraries of subroutines used by C programs, documented in man page sections 2 and 3)

    • grep (global regular expression print, your friend)

    • gzip (compression / decompression software, since many packages are distributed as .gz files; gzip is not quite as effective as bzip2, but is generally much faster)

    • the linux kernel (the header files describing system interfaces and capabilities are required by most software packages)

    • make (used for practically all software building)

    • ncurses (used for command line I/O)

    • patch (used to install patches, which are essentially diff output from comparing old and new source files)

    • perl (Practical Extraction and Reporting Language, a ubiquitous scripting language)

    • sed (stream editor, essentially an editor which is a filter)

    • tar (since most software packages are distributed as zipped tarballs: compressed archives containing many files and directories)

    • texinfo (info and related utilities)

    • util-linux (additional UNIX utilities, including fdformat, fdisk, hwclock, logger, mkswap, more, mount, umount)

    And if you want to run the tests on binutils and gcc (which are highly recommended), you need

    • dejagnu (the testing framework of the GNU project)

    • expect (which allows scripting of interactive programs)

    • tcl (Tool Command Language, a popular scripting language)

    All versions of all installed software must be inter-operable.

    This is a highly nontrivial exercise. Creating a linux distribution involves a great deal of effort in verifying inter-operability. This is a major contribution of the Linux From Scratch project. In general, you must read the README files associated with each package to determine which versions of other packages are required for correct operation.

  3. Being able to build a linux system is one thing; being able to run one is another. To have an operating build environment you must build and install:

    • boot scripts (in /etc/rc.d/init.d)

    • grub (the GRand Unified Boot loader, or LILO, the LInux LOader)

    • make_devices (a script to set up /dev)

    • sysklogd (syslogd and klogd)

    • sysvinit (init, halt, reboot, shutdown, telinit, etc.)

    and you must create a number of files:

    • /bin/sh

    • /etc/fstab, group, inittab, ld.so.conf (see below), nsswitch.conf (used by glibc), passwd, resolv.conf and syslog.conf

    • /etc/sysconfig/clock, network and network-devices/ifconfig.eth0

    • /var/run/utmp; /var/log/btmp, lastlog and wtmp (all used by login)

  4. Now that you have an operating build system, your first step is to re-build all of the above software. This verifies the integrity of your build environment. Then you need to add the software that makes it a usable UNIX environment.

    A usable linux system includes the following additional packages:

    • autoconf (used for auto-configuration scripts)

    • automake (used to create make files, which are input to make)

    • bison (a parser generator)

    • ed and one or more additional editors (ie, emacs, joe, vim)

    • e2fsprogs (including e2fsck and mke2fs)

    • file (the command line program to determine file types)

    • flex (generates programs for text pattern matching)

    • groff (used to format man pages)

    • iana-etc (/etc/protocols and /etc/services)

    • inetutils (ftp, ping, telnet, etc.)

    • kbd (keyboard utilities)

    • less (is more...)

    • libtool (library support utilities)

    • m4 (a macro processor)

    • man (which you all know and love)

    • mktemp (used to make temporary files for scripts)

    • modutils (module utilities such as insmod, lsmod, modprobe, rmmod)

    • net-tools (arp, ifconfig, route, etc.)

    • procinfo (some utilities that use /proc)

    • procps (kill, ps, etc.)

    • psmisc (pstree, etc.)

    • shadow (user and group utilities such as useradd, usermod, groupadd, passwd)

    • zlib (a library of compression subroutines)

  5. Finally, a modern linux system should include:

    • ghostscript (so you can use postscript for printing)

    • iptables (for firewall maintenance)

    • LessTif (an open source version of the Motif libraries for X)

    • libjpeg (library routines for manipulating jpeg files)

    • libungif (library routines for manipulating gif files)

    • LPRng (or CUPS, for print spooling)

    • man-pages (the actual content)

    • mozilla (or some other web browser; mozilla currently requires GTK+, a Graphical Tool Kit, libIDL, libmng, which and Zip)

    • ntp (network time protocol daemon)

    • openssh (ssh and sftp)

    • openssl (secure socket layer support)

    • WindowMaker (or some other window manager)

    • Xorg (or XFree86; commonly referred to as X; the build of X is intimately involved with the kernel, so don't try this one for your first software build experience!)

    And of course anything you need to enjoy your system. A complete list of the files used to install the software in our distribution is in /root/installed.

  6. One of the most important things to do before building new software is to have a good backup. While most software builds are bug-free, it is always possible, especially during the installation phase, that a faulty install script might modify or remove files used by other packages.

  7. Software builds usually include the following steps, although depending on the package, one or more may be omitted or replaced. Always read the INSTALL file to make sure you are installing the software correctly!

    1. Download and un-tar the software.

    2. cd to the software directory.

    3. Install any required patches.

    4. ./configure - this performs a thorough check of the current system capabilities, including what software is already installed, and configures the build process accordingly. If a package requires or optionally uses other software, that software must be installed before configure will find it. Sometimes it needs a little help... Use "./configure --help" to list all of the configuration options available for this package.

      Note that configure will determine your CPU type and build the software for that processor. Since not all processors have the same capabilities, you should build the software on the same type of CPU, or a lower class of compatible CPU, that you intend to run it on. For instance, software built on a Pentium will run on any class of Pentium, but software built on a Pentium II will only run on a Pentium II or better. It is possible to build for a different processor, but that is outside the scope of this class.

      Be sure to check the output of configure to make sure the package will be built to your needs; some of these builds are LONG!

    5. make (sometimes more than one, with various parameters)

    6. make install (which must be done as root, on a writable filesystem).

    7. Create post-installation configuration files.

    It is a good idea to keep a record of exactly which commands were used to build each package, and a log of the build output. This can be done as follows:
    (./configure) 2>&1 | tee package.log

    (make && make install) 2>&1 | tee -a package.log

    The "2>&1" will redirect stderr to stdout, and the tee will allow you to see the output as it is written to the log file (the "-a" tells tee to append to the log file). The "&&" ensures that the "make install" will not run if the "make" fails. Of course, you should not do either if there were problems with the configure.

  8. Most packages make use of the shared libraries (".so" files). You can build statically-linked software, in which the required library routines are copied into the executable program, if you intend for the software to be used on a system which may not have the required libraries available. This makes the executables very big.

    If you install software which includes new libraries, the install process will use ldconfig to rebuild the runtime linkages. /etc/ld.so.conf contains a list of the directories in which the dynamic loader is to look for libraries. Any library directories other than /lib and /usr/lib must be included in /etc/ld.so.conf. ldd can be used to determine which libraries are required by a given program.

  9. If you are running short on disk space, you can reduce the size of binary executables and libraries using
    strip --strip-debug (library file name)

    strip --strip-unneeded (binary executable file name)

    Be sure NOT to do the latter on libraries!

  10. During the build of our distribution, several additional parameters and steps were performed:

    • A man page for tar is not included in the man-pages package, so one was copied from the build system. tar DOES have info pages.

    • compressdoc was used to gzip the man and info pages.

    • emacs was built with the additional parameters "--without-pop --with-x".

    • fcron was built with the additional parameter "--with-editor=/bin/jpico".

    • The make of the Xorg DRI (Direct Rendering Infrastructure) included the additional parameter "LINUXDIR=/usr/src/linux-2.4.26". Also, at the end of the DRI build the following depmod command was executed: "depmod -a 2.4.26 -F /boot/System.map". The CD-ROM distribution has since moved to the 2.6.12.3 kernel, but it is backward compatible with the Xorg configuration.

  11. EXERCISES for Week 9:

    Now we get to fix a couple of additional problems in your distribution, which prevent mplayer from being as useful as it should be. In addition, it can improve the performance of multimedia software immensely if you rebuild it for your specific hardware, so that it takes advantage of all available CPU features.

    1. Download http://www.xiph.org/paranoia/download/cdparanoia-III-alpha9.8.src.tgz and un-tar it. cd to its directory and build it as follows: 

      ./configure --prefix=/usr &&
make &&
make install

      A small mistake caused several of the cdparanoia header files to be lost, which prevents mplayer from including cdparanoia (and therefore CD playing capability).

    2. Now we must rebuild mplayer. Download ftp://ftp1.mplayerhq.hu/MPlayer/releases/MPlayer-1.0pre4.tar.bz2 and un-tar it. cd to its directory and build it as follows: 

      ./configure --prefix=/usr --confdir=/etc/mplayer \
    --enable-largefiles --enable-shared-pp --with-x11libdir=/usr/X11R6/lib

      Make sure cdparanoia and X support were included (video output drivers should include xv and x11); this was inadvertently not done on your distribution. Then issue the commands 

      make &&
make install

      Test mplayer by playing a CD (mplayer cdda://1), and viewing the mpg file in last week's notes.

    3. Build cdrtools as directed in Beyond Linux From Scratch. If you can build this successfully, AND have a CD-burner, you should be able to burn CDs.

      You can burn a data CD using the commands 

      mkisofs -l -R -allow-leading-dots -o output.iso (input directory)
cdrecord -v dev=ATAPI:0,0,0 -data output.iso

      The mkisofs creates an ISO-9660 filesystem in the file output.iso, which is then burned using cdrecord.

      To burn a music CD, simply cd to a directory with wav files to be burned (make sure they will fit!), and 

      cdrecord -v dev=ATAPI:0,0,0 -audio -pad *.wav

    4. Use ldd to find out which libraries are required by login, bash and mozilla (try both /usr/bin/mozilla and /usr/lib/mozilla-1.6/mozilla-bin).

    5. Build dvd+rw-tools as follows: 
      tar -xzf dvd+rw-tools-5.21.4.10.8.tar.gz 
cd dvd+rw-tools-5.21.4.10.8
(make && make install) 2>&1 | tee -a ../dvd+rw-tools.out
cd ..
rm -rf dvd+rw-tools-5.21.4.10.8
      After rebuilding your kernel next week, you should also be able to burn DVD-ROMs.

  12. Here are some software links; some you already have, some you don't; but none of these are covered by Beyond Linux From Scratch:

©2006, Kenneth R. Koehler. All Rights Reserved. This document may be freely reproduced provided that this copyright notice is included.

Please send comments or suggestions to the author.