Fully Automatic Installation
Status: Just notes so far. FAI version used: 3.2.2
Last changed: Sunday 2014-11-23 08:52 UTC
Fully Automatic Installation (FAI) is a non-interactive system to install a DebianGNU/Linux operating system unattended on a single, a variety of computers or a whole cluster (which is intended to work as a single logical unit) at once. Also, if you want to migrate your "system" from one hardware to another every now and then, then FAI is the tool to choose. FAI is even usable for system updates, using the same configuration as if initially installing thus making it easy to first do an installation and subsequently update what has been installed before at any times as often as desired. If there is a team of administrators (maybe geographically spread across the globe) involved or you just want to maintain several branches or do other nifty things a revision control/source-code management allows to do then just do it. Revision control systems like SVN (Subversion), CVS (Concurrent Versions System), GNU Arch, GIT, etc. can be used to maintain the FAI code. FAI uses the DebianGNU/Linux distribution and a collection of shell and Perl scripts for the installation process. Changes to the configuration files of the operating system can be made by cfengine, shell, Perl and expect scripts. Since everything is done by dealing with plain text/source-code files, setting up and maintaining systems with FAI is a blissful thing -- one does not have to deal with nonsense GUIs (Graphical User Interfaces).
Table of Contents
Note: some sections are literal transcripts from
http://fai-project.org which have been modeled to suit my personal
needs at the time.
This section is a brief brainstorming and just pictures my point of
view respectively thoughts initial to a first hands-on with some
software tool or something unknown so far in general (whatever that
may be; its a common habit that I am used to).
Why want somebody use FAI or something similar
In short. It is about time. We want to sustain O(1) even with
increasing number of computer systems to administer. Not using a tool
like FAI (Fully Automatic Installation) means this process scales with
O(n) or even worse.
In other words — with FAI, if it takes you 1 hour to setup a computer
manually, it may take you a little longer to configure FAI but then
you can also install 100 computers in 1 hour once FAI is
configured. Installing 100 computers manually may take you 100 hours.
- With FAI O(1) plus time to set up FAI (this is what we want)
- Without FAI O(n) (we do not want that)
Note, that in all cases the Landau notation is a rough approximation
to what might then become true but that is fine since it is exactly
what it is used for.
Adding to the time considerations, the reason to use FAI might be to,
ensure 100% repetitive tasks on any number of systems, ease
administration on any number of systems and do disaster recovery on
Evaluating the Situation
This step assumes that the decision to use some sort of unintended
installation and/or automatizing procedures has already been
made. That following comes the evaluation process.
There is not much to say about evaluation with regards to the matter
of non-interactive installation and updating a computer system. If
evaluating at all then one has to look at the market, figure what
might meet his requirements/desires best and if lucky then there is
either just one solution which does so or none at all — no need to
If the former is not true, two or more solutions might do the job and
thus one has to evaluate the situation (one problem, n possible
solutions to the problem with n being equal or greater two) and figure
which of the n solutions is best for him.
Finally I came up with two possible choices. It will either be m23 or
FAI. I choose FAI simply because it is exists longer and therefore it
provides more features and looks more grown-up, has a greater
user-base, is already integrated into main Debian. Another difference
between m23 and FAI is, that m23 is more the GUI (Graphical User
Interface) approach and FAI the CLI (Command Line Interface) approach
which is what I like more.
However, please do not assume that m23 is of bad quality etc. — it is
a good tool too but my personal favor is more towards FAI and how it
does things. You should give both a try and then decide what meets
your requirements or personal likings more.
I am not providing detailed information on my evaluation process
between m23 and FAI since both are moving targets and then I have to
admit I am a little bit biased in favor of FAI since I already knew it
before I came a hardcore FAI user.
However, criticism is always good when it improves a situation. If I
would be in power and FAI was just to be born, I would try to
consolidate things towards a one-language for everything
approach. Into that, what I find a bit cumbersome about FAI is that
one needs to (can) deal with Shell, Perl, expect and cfengine scripts
plus can also write his specific code in any other language
e.g. python and use a hook to run it. Well, that sounds more dramatic
then it is since for the usual setup one does not have to go into the
code at all — it is just that a little consolidation might help to
not scare folks away from FAI because it looks a bit complex at first
sight. That is where the GUI approach of m23 definitely makes ground
compared to FAI. In the end FAI is what it is — a tool so powerful
and nice that once you made it through your first week, you will never
want to miss it.
This section is intended to give the first time reader a notion about
what FAI is, how it works, what kind of person makes use of it and
what it might be used for. There will also be a glance behind the
curtain, telling about the internals, which are of no importance for
the one who just wants to use FAI in order to either perform
non-interactive installation or non-interactive updating of some
One thought can be spelled out instantly — the user-base of FAI is so
diverse that I think it cannot be narrowed down to the DC (Data
Center) staff for example. To engage with FAI also makes sense for the
random user@home even if he just manages his two computers plus the
robo-puppy. In business or scientific environments, FAI just makes so
much sense that I do not even feel the need to provide a use-case
example — the reader will be able to tell a dozens when he finished
reading that page assuming his background is in IT (Information
Technology) and the like.
- A fully automated, unattended non-interactive installation can be
performed. No matter how many clients need to be installed, FAI
scales with O(1).
- Perform softupdates — update and update running systems without
- Clients can boot from NIC (Network Interface Card), CD, USB
(Universal Serial Bus) stick or floppy.
- If not booting via NIC, easy creation of the CD, USB stick or
floppy boot media is supported.
- PXE (Preboot Execution Environment) with DHCP (Dynamic Host
Configuration Protocol) and BOOTP (Bootstrap Protocol) boot methods
- Lilo and GNU GRUB (GRand Unified Bootloader) support
- Next to many others, ReiserFS, ext3 and XFS file systems are supported
- Automatic hardware detection
- Remote login via SSH (Secure Shell) during installation process
- Additional virtual terminals available during installation
- Making use of classes — similar configurations are shared among
all install clients.
- Log files for all installed clients kept not matter if installing a
client fails or succeeds. Those logs are saved onto the log server.
- Shell, Perl, expect and cfengine scripts are supported for the
configuration setup. Custom made software can be written as
desired. Therefore any language e.g. python might be used since,
for the most part, FAI deals with plain-text files.
- Access to a Debian mirror via NFS (Network File System), FTP (File
Transfer Protocol) or HTTP (Hypertext Transfer Protocol) during
- Keyboard layout selectable
- Can be used as a rescue system
- Tested on SUN SPARC hardware running Linux or Solaris. Works out of
the box with any mainstream of-the-shelf hardware i.e. x86, x86-64,
PPC (PowerPC), etc.
- Flexible system through easy class concept
- Predefined classes included e.g. classes to install a
- Disk-less client support
- Easily add your own functions via hooks and tailor things to
- Easily change the default behavior via hooks
How does FAI work?
The client(s) which will be installed using FAI, is/are booted from
floppy disk or via network card. Every clients gets an IP address
assigned and subsequently boots a Linux kernel which mounts its root
file system via NFS (Network File System) from the
install-server. After the OS (Operating System) is up and running, the
FAI startup script performs the automatic installation which does not
need any human interaction. First, the HDDs (Hard Disk Drives)
respectively SSDs (Solid State Drives) etc. will be partitioned, file
systems are created and then Debian software packages are
installed. After that, the new installed operating system on any
client is configured to local needs using some scripts. Finally the
new operating system (client) will be rebooted, using its now on-disk
OS i.e. Linux kernel image and basic layout that has been installed
during the former stage.
The details of how to install a client, namely the configuration data,
reside in the configuration space on the install-server. A concept of
classes (groups of configuration files) are shared among groups of
clients if they are intended to carry out similar tasks later on. So
you need not create a configuration for every new client — the effort
stays in O(1) no matter how much clients to install. Hence, FAI is a
scalable method to install a big cluster with a great number of nodes.
FAI can also be used as a network rescue system. You can boot your
computer, but it will not perform an installation. Instead it will run
a fully functional Debian GNU/Linux without using the local hard
disks. Then you can do a remote login and backup or restore a disk
partition, check a filesystem, inspect the hardware or do any other
task. Furthermore, FAI can be used to set up disk-less clients,
perform software updates on numerous clients at once i.e. this for
apt-get update && apt-get upgrade on a normal system.
Picturing a typical Install Process with FAI
The image below show how FAI works, of what parts it is made of and
how a typical installation procedure looks like. The subsequent
explanation of terms will give you further notions in detail.
This section describes the components of which FAI is made of and
which can also be seen in the picture above.
The computer system where the package
fai-server is installed. It
provides several services and data (e.g. Debian package repository,
configuration data to set up clients, etc.) for all clients to
install. There is usually one Install-server and one or more clients
— it thus becomes a typical centralized structure with the
install-server being the focal point when performing non-interactive
installation with FAI.
Detailed information of how the installation of clients should be
performed. This includes information about:
- HDD, SSD, <other_storage_media> layout i.e. partitioning
- Local file systems, their types, mount points and mount options
- Debian software packages, third party software, user specific data
- Keyboard layout, time zone, Xorg configuration, remote file
systems, user accounts, printers...
A class reflects the grouping of numerous configuration files, similar
in type, into one superset. As an example, a class for a workstation
featuring a text-editor, drawing and email applications etc. might be
DESKTOP, containing dozens of configuration
files in order to set up a workstation. This, however is fully
customizable and so anything is possible. You want to rapidly install,
configure and set up OS etc. on your robot bee swarm? No problem, once
you are done with putting together all data and configuration you
create a class e.g.
ROBOBEES and there you go. All names of classes,
except the hostname, are written in uppercase. They must not contain a
hyphen, a hash or a dot, but may contain underscores. A description of
all classes can be found in
A (chroot) file system located on the install-server. It is the
complete file system for the install clients during the installation
process. All clients share the same nfsroot, which they mount read
One or more computer(s) which will be installed using FAI and the data
(e.g. Debian packages, configuration, etc.) from the install-server.
When all installation tasks are finished, the log files are written to
the just installed client and to the install-server. It is also
possible to specify another computer as log saving destination. Thus
the log-server is more a choice of where to save the log files than a
distinct server. Basically there are three choices next to saving the
logs on the just installed clients
- Copy each client logs to the install-server
- Copy them to another (not the install-server) physical/virtual machine
- Not copying them to a remote location at all
Debian Repository - Package Mirror
At some point during the installation, clients need be provided with
those Debian packages which are intended to be installed onto the
clients. Now, what is no problem with a single workstation is
practically impossible or at least not the smartest way of doing it
with dozens of clients to be installed from scratch.
If we install dozens of computers, all from scratch, even if we have
high bandwidth available, the sheer amount of data needed to be
downloaded makes it worth to set up a local Debian mirror. This is
beneficial since we only need to download data once to our local
mirror plus the local bandwidth and latency is much better compared as
to using an official but remote Debian mirror. However, using Debians
official mirrors with FAI is no problem. In between those two extremes
there are some other possibilities called either
Part of FAI, is
fai-mirror. It is a command that creates a partial
mirror of Debian packages which will contain all packages of all
classes used in the configuration space.
apt-proxy on the other hand is not in any way special to FAI but an
official Debian package anyway, that is to be said is a really good
choice if you do not want to keep a fully fledged local mirror but
nonetheless want to have Debian packages quickly at hands for all your
computers in your LAN (Local Area Network)
,----[ apt-cache show apt-proxy | sed -n '/^Description/,/ \./p' - ]
| Description: Debian archive proxy and partial mirror builder
| apt-proxy automatically builds a Debian HTTP mirror based
| on requests which pass through the proxy. It's great for
| multiple Debian machines on the same network with a slower
| internet link.
This section tells about the things one should know of, that should be
in place and the software that needs to be acquired in order to use
FAI for non-interactive installing and/or updating computer systems of
Knowledge and Skills
I am a very strong believer that time is the only limiting factor in
life. That said, if you feel you do not have enough knowledge about
basics of how to deal with a Unix-like OS (Operating System) (e.g.
DebianGNU/Linux) just take your time to read, ask, think and play
around with it. Normally, a person with a two year experience in
Unix-like OSs should be able to figure how to use FAI within a two
weeks or less. For all the wizards out there, FAI is probably as easy
to handle as the TV set for normal folks.
Anyhow, this page should provide the reader familiar with the CLI
(Command Line Interface), a text-editor of choice and basic networking
knowledge with enough information to set up and manage computer
systems with FAI — no matter if he already is an old Debian head or
just started out with Debian a year ago.
A client computer must have a NIC (Network Interface Card). Unless
a disk-less installation should be performed a local storage media
e.g. SSD (Solid State Drive) or HDD (Hard Disk Drive) is also
needed. No floppy disk, CD-ROM, keyboard or graphic card is
The install-server in your LAN (Local Area Network) also needs a
NIC, usually has a HDD and keyboard and monitor attached to it. It
can also be the master-node of an HPC (High Performance Computing)
cluster located thousands of miles away in some DC (Data Center)
which is equipped with nifty remote access and control hardware.
- Networking Equipment
If the number of clients to install outnumbers the NICs on the
install-server which is likely the case, then some sort of
network switch]] is also needed to connect one install-server to
Make sure you have enough cables. In cases where many cables have
direct contact (i.e. a bundle) I strongly recommend to use
shielded]] cables in order to gain good signal quality on any wire
and thus avoid any "weird" errors that have been reported to busy
folks for some days until the figured that the cabling had been
- DHCP or BOOTP Server
DHCP (Dynamic Host Configuration Protocol) or BOOTP (Bootstrap
Protocol), the clients need one of these daemons to obtain boot
information. But it is also possible to put all this information onto
the boot floppy.
- TFTP (Trivial File Transfer Protocol) Server
The TFTP daemon is used for transferring the kernel to the clients. It
is only needed when booting from a NIC with a boot PROM (Programmable
- NFS (Network File System) -Root
It is a mountable directory which contains the whole file system for
the clients during installation. It will be created during the setup
of the FAI package and is also called
- Debian Mirror
Access to a Debian mirror is needed. A local mirror of all Debian
packages or an apt-proxy is recommended if you install several
- Install Kernel
The kernel image and the initial ramdisk that is used for booting the
clients. It mounts its root file system via NFS.
- Configuration Space
This directory tree which contains the configuration data is mounted
via NFS by default. But you can also get this directory from a
revision control system like SVN (Subversion), CVS or GIT (= a random
The TFTP daemon and NFS server will be enabled automatically when
installing the FAI package.
Setup and Configure FAI
This section is about how to install and set up all FAI components —
it is not about using FAI to install clients. In essence, to set up
FAI it takes the following steps:
- Installation of the packages needed i.e.
fai-quickstart is a
metapackage and thus it suffices.
- Edit the main configuration file called
fai.conf to fit individual
- The next step is to edit the file
make-fai-nfsroot.conf to point
to the desired location of the nfsroot and configure some other
settings related to the nfsroot.
- Create a local Debian mirror. This is step optional but is
- Finally the command
fai-setup is used to set up FAI on the
Variables and Files used by FAI
During the installation and later use you may come along a slew of
files and variables used by FAI which you might not know at that
point. This subsection shall make things easier by providing the
reader with some quick overview for the files and variables used by
The configuration for the FAI package (not the configuration data for
Definitions that are only used for creating the nfsroot are located in
A list of Debian packages for creating the nfsroot.
The contents of
/etc/fai/apt/sources.list and/or the debian mirror
specified by the variable
FAI_DEBMIRROR are used by the install-server
and also by clients.
man 8 fai-cd for more information.
- /etc/netgroup and /etc/exports
During the initial setup of FAI (not the installation of clients —
that comes later), those files are altered in order to set the right
permissions and the like so clients can mount the nfsroot and
configuration space from the install-server into their own file system
- /etc/ethers and /etc/hosts
Used to assign MAC (Media Access Control) addresses to hostnames and
The FAI community keeps a dedicated page to inform about the meaning
and their use.
How to install and set up the FAI components for later use.
Local Debian Mirror
Preparations for booting the Clients
Using gpg-authenticated debian-archives
Encrypting confidential Files on the Install-Server
Logging the install process
Use FAI to Install Clients
Tools and Functions Summary
Add or remove a file to the list of diversions and replace the file
with a dummy script. This is useful when a postinst script needs
manual input. At the end of the installation all diversions are
This given list of tasks are skipped. For use e.g. in
The Plan - Bootstrapping the Infrastructure
Before starting your installation, you should spend much time in planning
your installation. When you're happy with your installation concept, FAI can
do all the boring, repetitive tasks to turn your plans into practice. FAI
can't do good installations if your concept is imperfect or lacks some
important details. Start planning the installation by answering the
Will I create a Beowulf cluster, or do I have to install some desktop
How does my LAN topology looks like?
Do I have uniform hardware? Will the hardware stay uniform in the
Does the hardware need a special kernel?
How should the hosts be named?
How should the local hard disks be partitioned?
Which applications will be run by the users?
Do the users need a queueing system?
What software should be installed?
Which daemons should be started, and what should the configuration for
these look like?
Which remote filesystems should be mounted?
How should backups be performed?
Do you have sufficient power supply?
How much heat do the cluster nodes produce and how are they
Take into account distribution specific things
If you want to serve multiple nfsroot directories on one FAI
server, you need to create specific config directories in /etc for
fai, like /etc/fai-sarge and /etc/fai-etch. Then you need to set
the NFSROOT variables to different directories and run
make-fai-nfsroot -c /etc/fai-sarge.
Take into account chapter 10.2.3, "How to write a configuration
suitable for softupdates".
Use fai-divert -a if a postinst script calls a configuration
program, e.g. the postinst script for package apache calls
apacheconfig, which needs manual input. You can fake the
configuration program so the installation can be fully
automatic. But don't forget to use fai-divert -R to remove all
You also have to think about user accounts, printers, a mail system, cron
jobs, graphic cards, dual boot, NIS, NTP, timezone, keyboard layout,
exporting and mounting directories via NFS and many other things. So,
there is a lot to do before starting an installation. And remember that
knowledge is power, and it is up to you to use it. Installation and
administration is a process, not a product. FAI can't do things you don't
tell it to do.
But you need not to start from scratch. Look at all files and scripts in the
configuration space. There are a lot of things you can use for your own
installation. A good paper with more aspects of building an infrastructure
is http://www.infrastructures.org/papers/bootstrap/ "Bootstrapping an
The Install Sequence in Detail
After the kernel has booted, it mounts the root filesystem via NFS from the
install server and init(8) starts the script /usr/sbin/fai. This script
controls the sequence of the installation. No other scripts in /etc/init.d/
After the install client has booted, only the script
executed. This is the main script which controls the sequence of tasks
What happens step by step
- Installing clients using DHCP and Netboot via PXE (Preboot Execution Environment):
- Clients boot for the fist time — this boot is used to gather the
MAC addresses from all clients in the network. Subsequently, a
human has to edit
dhcp.conf on the install-server.
- All clients reboot again (now with fresh and appropriate
on the install-server).
- Clients get their FAI kernels via TFTP (Trivial File Transfer
- The FAI kernel currently active at this point mounts the nfsroot
from the install-server and the
init process starts the
/usr/sbin/fai. This script controls the
sequence of the installation. The installation script uses many
subroutines, which are defined in
no other scripts in
/etc/init.d/ are used at this point — only
- Clients mount or receive configuration space from a CVS/SVN/GIT
etc. repository on the install-server.
- The install process on the clients is performed using classes, the
Debian package mirror etc.
- Clients submit their logs (or not) to the log-server (by default,
the install-server is also the log-server).
- Finally, again, clients are rebooted. Afterwards clients are in
their final state, ready to carry out productive work.
Monitoring the installation
You can monitor the installation of all install clients with the
man 8 faimond. All clients check if this daemon is running on
the install server (or the machine defined by the variable
monserver. Then, a message is sent when a task starts and ends. The
fai monitor daemon prints this messages to standard output on the CLI
(Command Line Interface). In the future, there will be a graphical
The Configuration Space in Detail
The configuration is the collection of information about how exactly
to install a computer. The central configuration space for all install
clients is located on the install-server in
/srv/fai/config and its
sub-directories. This will be mounted by the install clients to
/var/lib/fai/config. It is also possible to receive all the
configuration data from a SCM (Software Configuration Management)
repository e.g. SVN (Subversion). The following subdirectories are
present and include several files:
Scripts and files to define classes and variables and to load kernel
Configuration files for disk partitioning and filesystem creation.
This directory holds all debconf(8) data. The format is the same that
is used by debconf-set-selections(8).
File with lists of software packages to be installed or removed.
Script for local site customization.
Files used by customization scripts. Most files are located in a
subtree structure which reflects the ordinary directory tree. For
example, the templates for nsswitch.conf are located in
/fai/files/etc/nsswitch.conf and are named according to the classes
that they should match: /fai/ files/etc/nsswitch.conf/NIS is the
version of /etc/nsswitch.conf to use for the NIS class. Note that the
contents of the files directory are not automatically copied to the
target machine, rather they must be explicitly copied by customization
scripts using the fcopy(8) command.
Normally the file /var/tmp/base.tgz is extracted on the install client
after the new file systems are created and before package are
installed. This is a minimal base image, created right after calling
debootstrap during the make-fai-nfsroot process on the install
server. If you want to install another distribution than the nfsroot
is, you can put a tar file into the subdirectory basefiles/ and name
it after a class. Then the command ftar(8) is used to extract the tar
file based on the classes defined. This is done in task extrbase.
Hooks are user defined programs or scripts, which are called during
the installation process.
In the first part of fai, all hooks with prefix confdir are
called. Since the configuration directory /fai is mounted in the
default task confdir, the hooks for this task are the only hooks
located in $nfsroot/fai/hooks on the install server. All other hooks
are found in $FAI_CONFIGDIR/hooks on the install server. All hooks
that are called before classes are defined can only use the following
classes: DEFAULT $HOSTNAME LAST.
Some examples for what hooks could be used:
- Use ssh in the very beginning to verify that you mounted the
configuration from the correct server and not a possible spoofing
- Do not mount the configuration directory, instead get a compressed
archive via HTTP or from floppy disk and extract it into a new ram
disk, then redefine $FAI_LOCATION.
- Load kernel modules before classes are defined in /fai/class.
- Send an email to the administrator if the installation is finished.
- Install a diskless client and skip local disk partitioning. See
- Partition the hard disk on an IA64 system, which needs a special
partition table type that must be created with parted(8).
Softupdates (updating already existing Installations)
See chapter 10.2
Fai-updater - Perform FAI softupdates on many machines at the same time
Xen and FAI
Using FAI to setup Xen Domains
Xen Tools and FAI Softupdates