Note: some sections are literal transcripts from http://fai-project.org which have been modeled to suit my personal needs at the time.

Primary Decisions

This section is a brief brainstorming and just pictures my point of view respectively thoughts initial to a first hands-on with some software tool or something unknown so far in general (whatever that may be; its a common habit that I am used to).

Why want somebody use FAI or something similar

In short. It is about time. We want to sustain O(1) even with increasing number of computer systems to administer. Not using a tool like FAI (Fully Automatic Installation) means this process scales with O(n) or even worse.

In other words — with FAI, if it takes you 1 hour to setup a computer manually, it may take you a little longer to configure FAI but then you can also install 100 computers in 1 hour once FAI is configured. Installing 100 computers manually may take you 100 hours.

• With FAI O(1) plus time to set up FAI (this is what we want)
• Without FAI O(n) (we do not want that)

Note, that in all cases the Landau notation is a rough approximation to what might then become true but that is fine since it is exactly what it is used for.

Adding to the time considerations, the reason to use FAI might be to, ensure 100% repetitive tasks on any number of systems, ease administration on any number of systems and do disaster recovery on failed systems.

Evaluating the Situation

This step assumes that the decision to use some sort of unintended installation and/or automatizing procedures has already been made. That following comes the evaluation process.

There is not much to say about evaluation with regards to the matter of non-interactive installation and updating a computer system. If evaluating at all then one has to look at the market, figure what might meet his requirements/desires best and if lucky then there is either just one solution which does so or none at all — no need to evaluate.

If the former is not true, two or more solutions might do the job and thus one has to evaluate the situation (one problem, n possible solutions to the problem with n being equal or greater two) and figure which of the n solutions is best for him.

Tools

Finally I came up with two possible choices. It will either be m23 or FAI. I choose FAI simply because it is exists longer and therefore it provides more features and looks more grown-up, has a greater user-base, is already integrated into main Debian. Another difference between m23 and FAI is, that m23 is more the GUI (Graphical User Interface) approach and FAI the CLI (Command Line Interface) approach which is what I like more.

However, please do not assume that m23 is of bad quality etc. — it is a good tool too but my personal favor is more towards FAI and how it does things. You should give both a try and then decide what meets your requirements or personal likings more.

I am not providing detailed information on my evaluation process between m23 and FAI since both are moving targets and then I have to admit I am a little bit biased in favor of FAI since I already knew it before I came a hardcore FAI user.

However, criticism is always good when it improves a situation. If I would be in power and FAI was just to be born, I would try to consolidate things towards a one-language for everything approach. Into that, what I find a bit cumbersome about FAI is that one needs to (can) deal with Shell, Perl, expect and cfengine scripts plus can also write his specific code in any other language e.g. python and use a hook to run it. Well, that sounds more dramatic then it is since for the usual setup one does not have to go into the code at all — it is just that a little consolidation might help to not scare folks away from FAI because it looks a bit complex at first sight. That is where the GUI approach of m23 definitely makes ground compared to FAI. In the end FAI is what it is — a tool so powerful and nice that once you made it through your first week, you will never want to miss it.

FAI Introduction

This section is intended to give the first time reader a notion about what FAI is, how it works, what kind of person makes use of it and what it might be used for. There will also be a glance behind the curtain, telling about the internals, which are of no importance for the one who just wants to use FAI in order to either perform non-interactive installation or non-interactive updating of some computer system(s).

One thought can be spelled out instantly — the user-base of FAI is so diverse that I think it cannot be narrowed down to the DC (Data Center) staff for example. To engage with FAI also makes sense for the random user@home even if he just manages his two computers plus the robo-puppy. In business or scientific environments, FAI just makes so much sense that I do not even feel the need to provide a use-case example — the reader will be able to tell a dozens when he finished reading that page assuming his background is in IT (Information Technology) and the like.

FAI Features

• A fully automated, unattended non-interactive installation can be performed. No matter how many clients need to be installed, FAI scales with O(1).
• Perform softupdates — update and update running systems without re-installation
• Clients can boot from NIC (Network Interface Card), CD, USB (Universal Serial Bus) stick or floppy.
• If not booting via NIC, easy creation of the CD, USB stick or floppy boot media is supported.
• PXE (Preboot Execution Environment) with DHCP (Dynamic Host Configuration Protocol) and BOOTP (Bootstrap Protocol) boot methods are supported.
• Lilo and GNU GRUB (GRand Unified Bootloader) support
• Next to many others, ReiserFS, ext3 and XFS file systems are supported
• Automatic hardware detection
• Remote login via SSH (Secure Shell) during installation process possible.
• Additional virtual terminals available during installation
• Making use of classes — similar configurations are shared among all install clients.
• Log files for all installed clients kept not matter if installing a client fails or succeeds. Those logs are saved onto the log server.
• Shell, Perl, expect and cfengine scripts are supported for the configuration setup. Custom made software can be written as desired. Therefore any language e.g. python might be used since, for the most part, FAI deals with plain-text files.
• Access to a Debian mirror via NFS (Network File System), FTP (File Transfer Protocol) or HTTP (Hypertext Transfer Protocol) during install possible.
• Keyboard layout selectable
• Can be used as a rescue system
• Tested on SUN SPARC hardware running Linux or Solaris. Works out of the box with any mainstream of-the-shelf hardware i.e. x86, x86-64, PPC (PowerPC), etc.
• Flexible system through easy class concept
• Predefined classes included e.g. classes to install a Beowulf cluster
• Disk-less client support
• Easily add your own functions via hooks and tailor things to specific needs
• Easily change the default behavior via hooks

How does FAI work?

The client(s) which will be installed using FAI, is/are booted from floppy disk or via network card. Every clients gets an IP address assigned and subsequently boots a Linux kernel which mounts its root file system via NFS (Network File System) from the install-server. After the OS (Operating System) is up and running, the FAI startup script performs the automatic installation which does not need any human interaction. First, the HDDs (Hard Disk Drives) respectively SSDs (Solid State Drives) etc. will be partitioned, file systems are created and then Debian software packages are installed. After that, the new installed operating system on any client is configured to local needs using some scripts. Finally the new operating system (client) will be rebooted, using its now on-disk OS i.e. Linux kernel image and basic layout that has been installed during the former stage.

The details of how to install a client, namely the configuration data, reside in the configuration space on the install-server. A concept of classes (groups of configuration files) are shared among groups of clients if they are intended to carry out similar tasks later on. So you need not create a configuration for every new client — the effort stays in O(1) no matter how much clients to install. Hence, FAI is a scalable method to install a big cluster with a great number of nodes.

FAI can also be used as a network rescue system. You can boot your computer, but it will not perform an installation. Instead it will run a fully functional Debian GNU/Linux without using the local hard disks. Then you can do a remote login and backup or restore a disk partition, check a filesystem, inspect the hardware or do any other task. Furthermore, FAI can be used to set up disk-less clients, perform software updates on numerous clients at once i.e. this for example equals apt-get update && apt-get upgrade on a normal system.

Picturing a typical Install Process with FAI

The image below show how FAI works, of what parts it is made of and how a typical installation procedure looks like. The subsequent explanation of terms will give you further notions in detail.

Terms

This section describes the components of which FAI is made of and which can also be seen in the picture above.

Install-Server

The computer system where the package fai-server is installed. It provides several services and data (e.g. Debian package repository, configuration data to set up clients, etc.) for all clients to install. There is usually one Install-server and one or more clients — it thus becomes a typical centralized structure with the install-server being the focal point when performing non-interactive installation with FAI.

Configuration Data

Detailed information of how the installation of clients should be performed. This includes information about:

• HDD, SSD, <other_storage_media> layout i.e. partitioning
• Local file systems, their types, mount points and mount options
• Debian software packages, third party software, user specific data
• Keyboard layout, time zone, Xorg configuration, remote file systems, user accounts, printers...

Class

A class reflects the grouping of numerous configuration files, similar in type, into one superset. As an example, a class for a workstation featuring a text-editor, drawing and email applications etc. might be called WORKSTATION or DESKTOP, containing dozens of configuration files in order to set up a workstation. This, however is fully customizable and so anything is possible. You want to rapidly install, configure and set up OS etc. on your robot bee swarm? No problem, once you are done with putting together all data and configuration you create a class e.g. ROBOBEES and there you go. All names of classes, except the hostname, are written in uppercase. They must not contain a hyphen, a hash or a dot, but may contain underscores. A description of all classes can be found in /usr/share/doc/fai-doc/classes_description.txt.

nfsroot

A (chroot) file system located on the install-server. It is the complete file system for the install clients during the installation process. All clients share the same nfsroot, which they mount read only.

Client

One or more computer(s) which will be installed using FAI and the data (e.g. Debian packages, configuration, etc.) from the install-server.

Log Server

When all installation tasks are finished, the log files are written to the just installed client and to the install-server. It is also possible to specify another computer as log saving destination. Thus the log-server is more a choice of where to save the log files than a distinct server. Basically there are three choices next to saving the logs on the just installed clients

• Copy each client logs to the install-server
• Copy them to another (not the install-server) physical/virtual machine
• Not copying them to a remote location at all

Debian Repository - Package Mirror

At some point during the installation, clients need be provided with those Debian packages which are intended to be installed onto the clients. Now, what is no problem with a single workstation is practically impossible or at least not the smartest way of doing it with dozens of clients to be installed from scratch.

If we install dozens of computers, all from scratch, even if we have high bandwidth available, the sheer amount of data needed to be downloaded¹ makes it worth to set up a local Debian mirror. This is beneficial since we only need to download data once² to our local mirror plus the local bandwidth and latency is much better compared as to using an official but remote Debian mirror. However, using Debians official mirrors with FAI is no problem. In between those two extremes there are some other possibilities called either fai-mirror or apt-proxy.

Part of FAI, is fai-mirror. It is a command that creates a partial mirror of Debian packages which will contain all packages of all classes used in the configuration space.

apt-proxy on the other hand is not in any way special to FAI but an official Debian package anyway, that is to be said is a really good choice if you do not want to keep a fully fledged local mirror but nonetheless want to have Debian packages quickly at hands for all your computers in your LAN (Local Area Network)

,----[ apt-cache show apt-proxy | sed -n '/^Description/,/ \./p' - ]
| Description: Debian archive proxy and partial mirror builder
|  apt-proxy automatically builds a Debian HTTP mirror based
|  on requests which pass through the proxy.  It's great for
|  multiple Debian machines on the same network with a slower
|  internet link.
|  .
`----

Prerequisites

This section tells about the things one should know of, that should be in place and the software that needs to be acquired in order to use FAI for non-interactive installing and/or updating computer systems of any sort³.

Knowledge and Skills

I am a very strong believer that time is the only limiting factor in life. That said, if you feel you do not have enough knowledge about basics of how to deal with a Unix-like OS (Operating System) (e.g. DebianGNU/Linux) just take your time to read, ask, think and play around with it. Normally, a person with a two year experience in Unix-like OSs should be able to figure how to use FAI within a two weeks or less. For all the wizards out there, FAI is probably as easy to handle as the TV set for normal folks.

Anyhow, this page should provide the reader familiar with the CLI (Command Line Interface), a text-editor of choice and basic networking knowledge with enough information to set up and manage computer systems with FAI — no matter if he already is an old Debian head or just started out with Debian a year ago.

Hardware

Client

A client computer must have a NIC (Network Interface Card). Unless a disk-less installation should be performed a local storage media e.g. SSD (Solid State Drive) or HDD (Hard Disk Drive) is also needed. No floppy disk, CD-ROM, keyboard or graphic card is needed.

Install-server

The install-server in your LAN (Local Area Network) also needs a NIC, usually has a HDD and keyboard and monitor attached to it. It can also be the master-node of an HPC (High Performance Computing) cluster located thousands of miles away in some DC (Data Center) which is equipped with nifty remote access and control hardware.

Networking Equipment

If the number of clients to install outnumbers the NICs on the install-server which is likely the case, then some sort of network switch]] is also needed to connect one install-server to many clients.

Cabling

Make sure you have enough cables. In cases where many cables have direct contact (i.e. a bundle) I strongly recommend to use shielded]] cables in order to gain good signal quality on any wire and thus avoid any "weird" errors that have been reported to busy folks for some days until the figured that the cabling had been the culprit.

Software

DHCP or BOOTP Server

DHCP (Dynamic Host Configuration Protocol) or BOOTP (Bootstrap Protocol), the clients need one of these daemons to obtain boot information. But it is also possible to put all this information onto the boot floppy.

TFTP (Trivial File Transfer Protocol) Server

The TFTP daemon is used for transferring the kernel to the clients. It is only needed when booting from a NIC with a boot PROM (Programmable read-only memory).

NFS (Network File System) -Root

It is a mountable directory which contains the whole file system for the clients during installation. It will be created during the setup of the FAI package and is also called nfsroot.

Debian Mirror

Access to a Debian mirror is needed. A local mirror of all Debian packages or an apt-proxy is recommended if you install several computers.

Install Kernel

The kernel image and the initial ramdisk that is used for booting the clients. It mounts its root file system via NFS.

Configuration Space

This directory tree which contains the configuration data is mounted via NFS by default. But you can also get this directory from a revision control system like SVN (Subversion), CVS or GIT (= a random three-letter combination).

The TFTP daemon and NFS server will be enabled automatically when installing the FAI package.

Setup and Configure FAI

This section is about how to install and set up all FAI components — it is not about using FAI to install clients. In essence, to set up FAI it takes the following steps:

1. Installation of the packages needed i.e. fai-quickstart is a metapackage and thus it suffices.
2. Edit the main configuration file called fai.conf to fit individual needs.
3. The next step is to edit the file make-fai-nfsroot.conf to point to the desired location of the nfsroot and configure some other settings related to the nfsroot.
4. Create a local Debian mirror. This is step optional but is strongly recommended.
5. Finally the command fai-setup is used to set up FAI on the install-server.

Variables and Files used by FAI

During the installation and later use you may come along a slew of files and variables used by FAI which you might not know at that point. This subsection shall make things easier by providing the reader with some quick overview for the files and variables used by FAI.

Files

/etc/fai/fai.conf

The configuration for the FAI package (not the configuration data for the clients).

/etc/fai/make-fai-nfsroot.conf

Definitions that are only used for creating the nfsroot are located in /etc/fai/make-fai-nfsroot.conf.

/etc/fai/NFSROOT

A list of Debian packages for creating the nfsroot.

/etc/fai/apt/sources.list

The contents of /etc/fai/apt/sources.list and/or the debian mirror specified by the variable FAI_DEBMIRROR are used by the install-server and also by clients.

/etc/fai/menu.lst

Grub menu.lst for fai-cd. See man 8 fai-cd for more information.

/etc/netgroup and /etc/exports

During the initial setup of FAI (not the installation of clients — that comes later), those files are altered in order to set the right permissions and the like so clients can mount the nfsroot and configuration space from the install-server into their own file system tree.

/etc/ethers and /etc/hosts

Used to assign MAC (Media Access Control) addresses to hostnames and IP addresses.

Variables

The FAI community keeps a dedicated page to inform about the meaning and their use.

FAI Components

How to install and set up the FAI components for later use.

Local Debian Mirror

apt-proxy

apt-mirror

fai-mirror

mkdebmirror

Preparations for booting the Clients

Security

Using gpg-authenticated debian-archives

• http://wiki.fai-project.org/index.php/Using_gpg-authenticated_debian-archives

Encrypting confidential Files on the Install-Server

• http://wiki.fai-project.org/index.php/Encrypting_confidential_files_on_the_install_server

Logging the install process

• http://wiki.fai-project.org/index.php/Getting_fai-logs_a_better_way

Miscellaneous

Fai Templates

• http://wiki.fai-project.org/index.php/FaiTemplates

Working remote

• http://wiki.fai-project.org/index.php/Using_FAI_to_install_a_root_Server
• installing FAI remote
• install remote systems with FAI

Use FAI to Install Clients

Tools and Functions Summary

fai-do-scripts

fcopy

ftar

fai-chboot

fai-divert

Add or remove a file to the list of diversions and replace the file with a dummy script. This is useful when a postinst script needs manual input. At the end of the installation all diversions are removed.

skiptask

This given list of tasks are skipped. For use e.g. in partition.DISKLESS.

The Plan - Bootstrapping the Infrastructure

Before starting your installation, you should spend much time in planning your installation. When you're happy with your installation concept, FAI can do all the boring, repetitive tasks to turn your plans into practice. FAI can't do good installations if your concept is imperfect or lacks some important details. Start planning the installation by answering the following questions:

Will I create a Beowulf cluster, or do I have to install some desktop machines?

How does my LAN topology looks like?

Do I have uniform hardware? Will the hardware stay uniform in the future?

Does the hardware need a special kernel?

How should the hosts be named?

How should the local hard disks be partitioned?

Which applications will be run by the users?

Do the users need a queueing system?

What software should be installed?

Which daemons should be started, and what should the configuration for these look like?

Which remote filesystems should be mounted?

How should backups be performed?

Do you have sufficient power supply?

How much heat do the cluster nodes produce and how are they cooled?

Take into account distribution specific things http://www.fai-project.org/fai/fai-guide/ch-arch.html

If you want to serve multiple nfsroot directories on one FAI server, you need to create specific config directories in /etc for fai, like /etc/fai-sarge and /etc/fai-etch. Then you need to set the NFSROOT variables to different directories and run make-fai-nfsroot -c /etc/fai-sarge.

Take into account chapter 10.2.3, "How to write a configuration suitable for softupdates".

Use fai-divert -a if a postinst script calls a configuration program, e.g. the postinst script for package apache calls apacheconfig, which needs manual input. You can fake the configuration program so the installation can be fully automatic. But don't forget to use fai-divert -R to remove all faked scripts.

You also have to think about user accounts, printers, a mail system, cron jobs, graphic cards, dual boot, NIS, NTP, timezone, keyboard layout, exporting and mounting directories via NFS and many other things. So, there is a lot to do before starting an installation. And remember that knowledge is power, and it is up to you to use it. Installation and administration is a process, not a product. FAI can't do things you don't tell it to do.

But you need not to start from scratch. Look at all files and scripts in the configuration space. There are a lot of things you can use for your own installation. A good paper with more aspects of building an infrastructure is http://www.infrastructures.org/papers/bootstrap/ "Bootstrapping an Infrastructure".

The Install Sequence in Detail

After the kernel has booted, it mounts the root filesystem via NFS from the install server and init(8) starts the script /usr/sbin/fai. This script controls the sequence of the installation. No other scripts in /etc/init.d/ are used.

After the install client has booted, only the script /usr/sbin/fai is executed. This is the main script which controls the sequence of tasks for FAI.

What happens step by step

Installing clients using DHCP and Netboot via PXE (Preboot Execution Environment):

• Clients boot for the fist time — this boot is used to gather the MAC addresses from all clients in the network. Subsequently, a human has to edit dhcp.conf on the install-server.
• All clients reboot again (now with fresh and appropriate dhcp.conf on the install-server).
• Clients get their FAI kernels via TFTP (Trivial File Transfer Protocol).
• The FAI kernel currently active at this point mounts the nfsroot from the install-server and the init process starts the installation script /usr/sbin/fai. This script controls the sequence of the installation. The installation script uses many subroutines, which are defined in /usr/share/fai/subroutines. Also, no other scripts in /etc/init.d/ are used at this point — only /usr/sbin/fai.
• Clients mount or receive configuration space from a CVS/SVN/GIT etc. repository on the install-server.
• The install process on the clients is performed using classes, the Debian package mirror etc.
• Clients submit their logs (or not) to the log-server (by default, the install-server is also the log-server).
• Finally, again, clients are rebooted. Afterwards clients are in their final state, ready to carry out productive work.

Monitoring the installation

You can monitor the installation of all install clients with the command man 8 faimond. All clients check if this daemon is running on the install server (or the machine defined by the variable monserver. Then, a message is sent when a task starts and ends. The fai monitor daemon prints this messages to standard output on the CLI (Command Line Interface). In the future, there will be a graphical frontend available.

The Configuration Space in Detail

The configuration is the collection of information about how exactly to install a computer. The central configuration space for all install clients is located on the install-server in /srv/fai/config and its sub-directories. This will be mounted by the install clients to /var/lib/fai/config. It is also possible to receive all the configuration data from a SCM (Software Configuration Management) repository e.g. SVN (Subversion). The following subdirectories are present and include several files:

class/

Scripts and files to define classes and variables and to load kernel modules.

disk_config/

Configuration files for disk partitioning and filesystem creation.

debconf/

This directory holds all debconf(8) data. The format is the same that is used by debconf-set-selections(8).

package_config/

File with lists of software packages to be installed or removed.

scripts/

Script for local site customization.

files/

Files used by customization scripts. Most files are located in a subtree structure which reflects the ordinary directory tree. For example, the templates for nsswitch.conf are located in /fai/files/etc/nsswitch.conf and are named according to the classes that they should match: /fai/ files/etc/nsswitch.conf/NIS is the version of /etc/nsswitch.conf to use for the NIS class. Note that the contents of the files directory are not automatically copied to the target machine, rather they must be explicitly copied by customization scripts using the fcopy(8) command.

basefiles/

Normally the file /var/tmp/base.tgz is extracted on the install client after the new file systems are created and before package are installed. This is a minimal base image, created right after calling debootstrap during the make-fai-nfsroot process on the install server. If you want to install another distribution than the nfsroot is, you can put a tar file into the subdirectory basefiles/ and name it after a class. Then the command ftar(8) is used to extract the tar file based on the classes defined. This is done in task extrbase.

hooks/

Hooks are user defined programs or scripts, which are called during the installation process.

Hooks

In the first part of fai, all hooks with prefix confdir are called. Since the configuration directory /fai is mounted in the default task confdir, the hooks for this task are the only hooks located in $nfsroot/fai/hooks on the install server. All other hooks are found in $FAI_CONFIGDIR/hooks on the install server. All hooks that are called before classes are defined can only use the following classes: DEFAULT $HOSTNAME LAST.

Some examples for what hooks could be used:

• Use ssh in the very beginning to verify that you mounted the configuration from the correct server and not a possible spoofing host.
• Do not mount the configuration directory, instead get a compressed archive via HTTP or from floppy disk and extract it into a new ram disk, then redefine $FAI_LOCATION.
• Load kernel modules before classes are defined in /fai/class.
• Send an email to the administrator if the installation is finished.
• Install a diskless client and skip local disk partitioning. See hooks/partition.DISKLESS.
• Partition the hard disk on an IA64 system, which needs a special partition table type that must be created with parted(8).

Softupdates (updating already existing Installations)

See chapter 10.2

Fai-updater - Perform FAI softupdates on many machines at the same time

• http://wiki.fai-project.org/index.php/Fai-updater

Xen and FAI

Using FAI to setup Xen Domains

• http://wiki.fai-project.org/index.php/Using_FAI_to_set_up_XEN_domains

Xen Tools and FAI Softupdates