SystemImager Installation

The NIFLHEIM cluster uses the SystemImager toolkit on a central server to create an image of a Golden Client node's hard disk that has been installed in the usual way using a distribution on CD-ROM (CentOS Linux in our case). The SystemImager is subsequently used to install identical disk images of the Golden Client on all of the nodes (changing of course hostname and network parameters).

SystemImager documentation

Documentation is available at http://www.systemimager.org/documentation/.

There's a wiki at http://wiki.systemimager.org containing a Quick Start HowTo.

The Sisuite-users mailing list is an excellent place to search for information or get help.

RHEL6/CentOS6 status of SystemImager

When attempting to build SystemImager on RHEL6/CentOS6 systems we have found some problems, the solution of which would require SystemImager developers to carry out extra work. I tried without luck to solve the problem in this sisuite-users thread [sisuite-users] Building SystemImager on CentOS 6.2?.

SystemImager status as of July 2012:

  1. The SystemImager trunk version 4.3.0 rev. 4568 from SVN cannot build on RHEL6.
  2. You may be able to find missing source tar-balls at http://olivier.lahaye1.free.fr/SOURCES/
  3. On RHEL6 the kernel config stage will ask you many questions about modules (guess: just press Enter to select the defaults).
  4. The parted-2.3 package cannot build on RHEL6, aborts with the error checking for dm_task_create in -ldevmapper... no.

Next I tried to build the older SystemImager version 4.1.7 (which we used for years on RHEL5), but this version also crashes during the kernel build stage (probably the RHEL6 gcc 4.4 is too recent).

Conclusion: RHEL6 differs so much from RHEL5 that some amount of porting and testing of SystemImager would be required. Unfortunately, hardly anyone seems to be working actively on SystemImager at this time :-(

Workarounds on RHEL6

Build the old SystemImager 4.1.7 on a RHEL5 system. The resulting noarch packages can install successfully on RHEL6.

Further testing required.

RHEL5/CentOS5 status of SystemImager

SystemImager status as of July 2012:

  1. The SystemImager trunk version 4.3.0 rev. 4568 from SVN cannot build on RHEL5.
  2. The build crashes when building gzip-1.4.tar.xz, because the .xz extension is unknown. A solution would be to replace gzip-1.4.tar.xz by gzip-1.4.tar.gz in the SVN.

Downloading SystemImager

The source and binary files can be downloaded from SystemImager at SourceForge.

Another version maintained by Bernard Li can be downloaded from the OSCAR project site.

Systemconfigurator

You also need to install the tool systemconfigurator (not needed in the build process of SystemImager). >From this page you can get RPM packages for systemconfigurator and perl-AppConfig.

For further information see the Wiki page http://wiki.systemimager.org and http://wiki.systemimager.org/index.php/System_Configurator.

Systemconfigurator has a prerequisite perl-AppConfig which is available in many YUM repositories and can possibly be installed by yum install perl-AppConfig. RPMs can be downloaded from http://pkgs.repoforge.org/perl-AppConfig/.

Subversion trunk

The development version of SystemImager is kept in http://systemimager.svn.sourceforge.net/ An updated version can be downloaded from SVN by:

svn checkout https://systemimager.svn.sourceforge.net/svnroot/systemimager/trunk

Building SystemImager

Building RPMs for SystemImager is described in http://wiki.systemimager.org/index.php/RPM_Packaging.

One has to build the RPMs as the root superuser, or use the fakeroot method. One can alternatively use noarch RPMs from SystemImager at SourceForge.

As a prerequisite the docbook-utils and PyXML are required; if not available they must first be installed by root:

yum install docbook-utils PyXML dos2unix e2fsprogs-devel openssl-devel python-devel

Build process

Now build the RPMs (takes a long time) as specified in http://wiki.systemimager.org/index.php/RPM_Packaging. For the Subversion trunk (see above) the instructions are:

cd trunk/
make source_tarball # Ignore messages about GPG keys
cd tmp
rpmbuild -ta systemimager-4.?.?.tar.bz2

This process will download many source packages from the net to rpmbuild's build tree (assuming you are building as root) /usr/src/redhat/BUILD/systemimager-*/src/.

Post-building instructions

In order to avoid repeated downloading, copy these packages as follows:

cd /usr/src/redhat/BUILD/systemimager-*/src
cp *.gz *.bz2 *.tgz *.tar /usr/src/
cd /usr/src/redhat/BUILD/systemimager-*/initrd_source/src
cp *.gz *.bz2 /usr/src/

Other packages needed

You also need to build the systemconfigurator (not needed in the build process of SystemImager) RPMs systemconfigurator and perl-AppConfig.

In addition the RPM package perl-XML-Simple should be installed. On Yum-based systems you can simply do:

yum install http://pkgs.repoforge.org/perl-XML-Simple/perl-XML-Simple-2.18-3.el6.rfx.noarch.rpm

where the appropriate package URL is from http://dag.wieers.com/rpm/packages/perl-XML-Simple/.

Installing SystemImager on the server

The following subset of RPM packages should be installed on the SystemImager image server:

systemimager-i386boot-standard    (if you have this architecture)
systemimager-x86_64boot-standard  (if you have this architecture)
systemimager-common
systemimager-server
systemconfigurator
perl-AppConfig
perl-XML-Simple

You may use this command:

yum localinstall --nogpgcheck systemconfigurator*.rpm perl-AppConfig*.rpm perl-XML-Simple*.rpm
yum localinstall --nogpgcheck systemimager-common-*.rpm systemimager-server-*.rpm systemimager-x86_64boot-standard-*.rpm

Installing BitTorrent transport with SystemImager

Still needs to be tested:

If you want to use the BitTorrent transport to distribute images, please read first:

First you must enable RPMFORGE, please see https://rpmrepo.org/RPMforge/Using. There are also detailed instructions in http://wiki.centos.org/AdditionalResources/Repositories/RPMForge.

For RHEL6/CentOS6 we can download an RPM built for EL6, see for example http://code.google.com/p/atyu30/downloads/list?q=label:rhel6

For CentOS5 we used to do this:

rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

You should disable the repository by default in the file /etc/yum.repos.d/rpmforge.repo to read:

enabled = 0

Install the yum utilities:

yum install yum-utils

Download the bittorrent RPM by:

yumdownloader --resolve --enablerepo=rpmforge bittorrent

and install the 3 packages by:

yum localinstall bittorrent-*.rpm python-crypto-*.rpm python-khashmir-*.rpm

Then install:

yum localinstall --nogpgcheck systemimager-bittorrent-*.rpm

You must configure the Bittorrent transport in /etc/systemimager/bittorrent.conf, especially using the correct values for BT_INTERFACE and BT_IMAGES.

Installing SystemImager on the clients

The following subset of RPM packages should be installed on the SystemImager Golden Client:

systemimager-i386initrd_template    (if you have this architecture)
systemimager-x86_64initrd_template  (if you have this architecture)
systemimager-common
systemimager-client
systemconfigurator
perl-AppConfig

Then install packages by:

yum localinstall --nogpgcheck systemconfigurator*rpm
yum localinstall --nogpgcheck systemimager-common*rpm systemimager-client*rpm systemimager-x86_64initrd_template*rpm

Golden Client image

The disk image of the Golden Client is done by first preparing an rsync process on the client:

si_prepareclient --server <image-server-name> --yes

There are some useful flags that may be added to the above command, see the si_prepareclient man-page:

--my-modules  # Make a smaller initrd.img file with only the relevant kernel modules.

--no-uyok     # Do not create kernel/initrd.img - useful if you just want to update the image.

--autodetect-disks # Try to detect available disks on the client instead of using devices from autoinstallscript.conf.

When this is completed, the Golden Client disk image is downloaded to the image server by:

si_getimage --golden-client <golden-client-name> --image <image-name> --exclude-file /root/exclude.getimage --post-install reboot [--update-script NO]

The -update-script flag controls whether the image-name.master script in /var/lib/systemimager/scripts/ should be updated or not. The exclude file /root/exclude.getimage defines files that you want to omit from the image, for example:

/scratch/*/*
/tmp/*
/tmp/*/
/mnt/*
/mnt/*/
/var/cache/yum/*/packages
/var/log/*log
/var/log/messages*
/var/log/maillog*
/var/log/ksyms*
/var/log/cron*
/var/log/rpmpkgs*
/var/log/secure*
/var/log/spooler*
/var/log/wtmp*
/var/log/XFree86*
/var/log/ntp.log*
/var/log/boot.log*
/var/log/*.0
/var/log/*.1
/var/log/*.2
/var/log/*.3
/var/log/*.4
/var/log/*.5
/var/log/*.6
/var/log/XFree*
/var/tmp/*
/var/lib/slocate/*
/var/spool/mqueue/*
/var/spool/lpd/*
/var/spool/cups/tmp/*
/var/spool/torque/mom_logs/*
/var/spool/torque/spool/*
/var/spool/torque/undelivered/*
/var/spool/torque/checkpoint/*
/var/spool/torque/aux/*
/var/spool/torque/mom_priv/jobs/*
/var/spool/torque/mom_priv/mom.lock
/var/spool/at/spool/*
/var/spool/at/a*
/var/spool/at/b*
/var/spool/mail/*

After the si_getimage has completed, you may want to kill the rsync process on the client for security reasons:

GoldenClient# skill -v rsync

However, it seems that Systemimager 3.X kills the rsync daemon on the Golden Client upon completion of the si_getimage command, so there is no need to worry about rsync daemons being left behind.

Copy UYOK kernel and initrd for this client

SystemImager allows/recommends you to Use Your Own Kernel (UYOK) for PXE-booting the clients. This must be used if there is any special hardware in the client.

Following the UYOK instructions, copy the kernel and initrd.img for this client to the TFTP area:

cd /var/lib/systemimager/images/<image-name>/etc/systemimager/boot
cp * /tftpboot/<nodetype>/

where the <nodetype> is a hardware-specific subdirectory under /tftpboot for this particular type of client.

Configuring the boot server

The following prerequisites are needed before you run si_mkbootserver, so install them if they are absent:

rpm -q tftp-server tftp

Now set up your DHCP/TFTP boot server daemons:

si_mkbootserver

This command will also invoke the DHCP configuration command si_mkdhcpserver.

You need to check the configuration files before proceeding:

/etc/dhcpd.conf
/etc/xinetd.d/tftp

Make sure that TFTP uses the /tftpboot directory.

Copy the SystemImager message file:

cp /etc/systemimager/pxelinux.cfg/message.txt /tftpboot/

SystemImager monitoring

You may want to set up the SystemImager monitoring service as described in http://wiki.systemimager.org/index.php/Monitoring. The RHEL5 RPM package libxml2 may perhaps contain the libxml-perl package referred to in this page (?).

You need some additional Perl modules, install on RHEL5 by:

yum install --enablerepo=rpmforge perl-Tk

If you use SystemImager monitoring (see the MONITOR_SERVER below) you must start the monitoring service:

chkconfig systemimager-server-monitord on
service systemimager-server-monitord start

If you use the iptables firewall you will have to configure monitoring access. Add to /etc/sysconfig/iptables:

# SystemImager monitor server
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8181 -j ACCEPT

(see DEFAULT_PORT in /usr/sbin/si_monitor).

Define client nodes

You define client nodes to SystemImager by the command:

si_addclients

This will create links to image master scripts in /var/lib/systemimager/scripts. The master scripts are created by the above si_getimage command.

Configure image service

In order for clients to download an image from the SystemImager server, you need to start one or more image services on the server.

To use the standard rsync method:

chkconfig systemimager-server-rsyncd on
service systemimager-server-rsyncd start

If you use the iptables firewall you will have to configure rsync access. Add to /etc/sysconfig/iptables:

# SystemImager rsync server
-A INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j ACCEPT

and restart the iptables service.

To use the bittorrent method:

chkconfig systemimager-server-bittorrent on
service systemimager-server-bittorrent start

When starting the systemimager-server-bittorrent service, a tar-ball and .torrent file is made of the desired images (BT_IMAGES in /etc/systemimager/bittorrent.conf) and placed in /var/lib/systemimager/torrents/, along with a torrent for the boel_binaries.tar.gz.

Beware: This means that if you update your image (using si_getimage), the Bittorrent tar-ball image will remain unchanged until you restart the systemimager-server-bittorrent service.

Configure network PXE booting of clients

Install pxeconfig and PXE menu tools

We prefer to control the network PXE booting of clients by means of the external PXEconfig package. There are installation instructions for pxeconfig.

It is also a good idea to configure the PXE menu tools from the PXEconfig package. This enables to control the booting of networked computers directly from the computer's console at the BIOS level before any operating system has been loaded. You get an on-screen hierarchical menu system which you navigate with the keyboard, and this allows you to select dynamically the tool or disk image to be booted.

Configure PXE booting

For each hardware type of client, which in this example we call x326m, we assume you have created the directory /tftpboot/x326m/ containing kernel and initrd.img as described above. The PXE boot file is described in the UYOK PXE instructions, but we use a slightly different setup.

We create the file /tftpboot/pxelinux.cfg/default.node_install.x326m containing:

DEFAULT systemimager
LABEL systemimager
DISPLAY message.txt
PROMPT 1
TIMEOUT 50
KERNEL x326m/kernel
APPEND vga=extended initrd=x326m/initrd.img root=/dev/ram MONITOR_SERVER=10.1.128.2 MONITOR_CONSOLE=yes ramdisk_size=80000 tmpfs_size=2500M

Here x326m/ refers to the mentioned subdirectory of /tftpboot/ and the monitoring server in this example is 10.1.128.2.

Examining the initrd.img image

If you need to look inside the initrd.img (initial RAM-disk image) this can be done as follows:

mkdir temp; cd temp
cp .../initrd.img initrd.img.gz  # Make a copy from somewhere
gunzip initrd.img.gz
cpio -id < initrd.img

Now the current directory .../temp/ contains the unpacked image.

Changing disk size in SystemImager

If one or more of your nodes has a different disk size than that of your Golden Client machine, the SI master-script in will not have the correct partitioning information. The partitioning information is created on the Golden Client and stored in the client's /etc/systemimager/autoinstallscript.conf file and used by the si_mkautoinstallscript script.

Since we use CentOS/RHEL our disks are partitioned using Logical Volumes. See http://wiki.systemimager.org/index.php/LVM for LVM support in SI.

To grow the last partition on a disk to maximum size, use the size parameter in the /etc/systemimager/autoinstallscript.conf file (see man autoinstallscript.conf):

  • Partition: Size can also hold the special value of "*". This is used to indicate that the specified partition should be growable to the end of the disk. If the last partition is a primary partition, then only that partition should have "*" specified. If the last partition is a logical partition, then both the logical partition, and the extended partition that contains it, should have "*" specified.
  • LVM: size - The size of the LVM logical volume. A size suffix (k for kilobytes up to t for terabytes) is optional, megabytes is the default if no suffix is present. The special value "*" can be used to assign all the free space available in the volume group.

Warning: The file /etc/systemimager/autoinstallscript.conf gets overwritten every time you run the si_prepareclient command ! So you need to edit the file after starting si_prepareclient.

Modifying the master script

We can alternatively modify the SI *.master script in /var/lib/systemimager/scripts to use the entire disk space.

Locate the lines using parted for partitioning and change the last partition's fixed size into the size of the disk:

logmsg "Creating partition ${DISK0}2."
START_MB=$END_OF_LAST_PRIMARY
END_MB=$(( $DISK_SIZE - 0 ))

Change the lines for the /scratch partition like:

logmsg "lvcreate  -L178552832K -n lv_scratch VolGroup00 || shellout"
lvcreate  -L178552832K -n lv_scratch VolGroup00 || shellout

into using the 100%FREE flag (see man lvcreate or the CentOS lvcreate docs):

logmsg "lvcreate  -l 100%FREE -n lv_scratch VolGroup00 || shellout"
lvcreate  -l 100%FREE -n lv_scratch VolGroup00 || shellout

This will cause the /scratch partition to occupy 100% of the remaining free space, which is what we would like to have.

An additional optimization is to reduce the number of inodes on the /scratch disk by adding -i 65536 to the mke2fs command:

logmsg "mke2fs -q -j -i 65536 /dev/VolGroup00/LogVol02 || shellout"
mke2fs -q -j -i 65536 /dev/VolGroup00/LogVol02 || shellout

Niflheim: SystemImager_Installation (last edited 2018-02-14 11:37:10 by OleHolmNielsen)