Login

How to Build a Netboot Server, Part 2

<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/12/how-to-build-a-netboot-server-part-2.png" width="1024" height="819" title="" alt="" /></div><div>The article <a href="https://fedoramagazine.org/how-to-build-a-netboot-server-part-1/">How to Build a Netboot Server, Part 1</a> showed you how to create a netboot image with a “liveuser” account whose home directory lives in volatile memory. Most users probably want to preserve files and settings across reboots, though. So this second part of the netboot series shows how to reconfigure the netboot image from part one so that <a href="https://en.wikipedia.org/wiki/Active_Directory" target="_blank" rel="noopener">Active Directory</a> user accounts can log in and their home directories can be automatically mounted from a NFS server.
Part 3 of this series will show how to make an interactive and centrally-configurable iPXE boot menu for the netboot clients.

<h2>Setup NFS4 Home Directories with KRB5 Authentication</h2>
Follow the directions from the previous post “<a href="https://fedoramagazine.org/secure-nfs-home-directories-kerberos" target="_blank" rel="noopener">Share NFS Home Directories Securely with Kerberos</a>,” then return here.
<h2>Remove the Liveuser Account</h2>
Remove the “liveuser” account created in part one of this series:
<pre>$ sudo -i # sed -i '/automaticlogin/Id' /fc28/etc/gdm/custom.conf # rm -f /fc28/etc/sudoers.d/liveuser # for i in passwd shadow group gshadow; do sed -i '/^liveuser:/d' /fc28/etc/$i; done</pre>
<h2>Configure NTP, KRB5 and SSSD</h2>
Next, we will need to duplicate the NTP, KRB5, and SSSD configuration that we set up on the server in the client image so that the same accounts will be available:
<pre># MY_HOSTNAME=$(</etc/hostname) # MY_DOMAIN=${MY_HOSTNAME#*.} # dnf -y --installroot=/fc28 install ntp krb5-workstation sssd # cp /etc/ntp.conf /fc28/etc # chroot /fc28 systemctl enable ntpd.service # cp /etc/krb5.conf.d/${MY_DOMAIN%%.*} /fc28/etc/krb5.conf.d # cp /etc/sssd/sssd.conf /fc28/etc/sssd</pre>
Reconfigure sssd to provide authentication services, in addition to the identification service already configured:
<pre># sed -i '/services =/s/$/, pam/' /fc28/etc/sssd/sssd.conf</pre>
Also, ensure none of the clients attempt to update the computer account password:
<pre># sed -i '/id_provider/a \ \ ad_maximum_machine_account_password_age = 0' /fc28/etc/sssd/sssd.conf </pre>
Also, copy the nfsnobody definitions:
<pre># for i in passwd shadow group gshadow; do grep "^nfsnobody:" /etc/$i >> /fc28/etc/$i; done</pre>
<h2>Join Active Directory</h2>
Next, you’ll perform a chroot to join the client image to Active Directory. Begin by deleting any pre-existing computer account with the same name your netboot image will use:
<pre># MY_USERNAME=jsmith # MY_CLIENT_HOSTNAME=$(</fc28/etc/hostname) # adcli delete-computer "${MY_CLIENT_HOSTNAME%%.*}" -U "$MY_USERNAME"</pre>
Also delete the krb5.keytab file from the netboot image if it exists:
<pre># rm -f /fc28/etc/krb5.keytab</pre>
Perform a chroot into the netboot image:
<pre># for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc28/$i; done # chroot /fc28 /usr/bin/bash --login</pre>
Perform the join:
<pre># MY_USERNAME=jsmith # MY_HOSTNAME=$(</etc/hostname) # MY_DOMAIN=${MY_HOSTNAME#*.} # MY_REALM=${MY_DOMAIN^^} # MY_OU="cn=computers,dc=${MY_DOMAIN//./,dc=}" # adcli join $MY_DOMAIN --login-user="$MY_USERNAME" --computer-name="${MY_HOSTNAME%%.*}" --host-fqdn="$MY_HOSTNAME" --user-principal="host/$MY_HOSTNAME@$MY_REALM" --domain-ou="$MY_OU"</pre>
Now log out of the chroot and clear the root user’s command history:
<pre># logout # for i in run sys proc dev/shm dev/pts dev; do umount /fc28/$i; done # > /fc28/root/.bash_history</pre>
<h2>Install and Configure PAM Mount</h2>
We want our clients to automatically mount the user’s home directory when they log in. To accomplish this, we’ll use the “pam_mount” module. Install and configure pam_mount:
<pre># dnf install -y --installroot=/fc28 pam_mount # cat << END > /fc28/etc/security/pam_mount.conf.xml <?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd"> <pam_mount> <debug enable="0" /> <volume uid="1400000000-1499999999" fstype="nfs4" server="$MY_HOSTNAME" path="/home/%(USER)" mountpoint="/home/%(USER)" options="sec=krb5" /> <mkmountpoint enable="1" remove="0" /> <msg-authpw>Password:</msg-authpw> </pam_mount> END</pre>
Reconfigure PAM to use pam_mount:
<pre># dnf install -y patch # cp -r /fc28/usr/share/authselect/default/sssd /fc28/etc/authselect/custom # echo 'initgroups: files' >> /fc28/etc/authselect/custom/sssd/nsswitch.conf # patch /fc28/etc/authselect/custom/sssd/system-auth << END @@ -12 +12,2 @@ -auth sufficient pam_sss.so forward_pass +auth requisite pam_mount.so {include if "with-pammount"} +auth sufficient pam_sss.so {if "with-pammount":use_first_pass|forward_pass} @@ -35,2 +36,3 @@ session required pam_unix.so +session optional pam_mount.so {include if "with-pammount"} session optional pam_sss.so END # patch /fc28/etc/authselect/custom/sssd/password-auth << END @@ -9 +9,2 @@ -auth sufficient pam_sss.so forward_pass +auth requisite pam_mount.so {include if "with-pammount"} +auth sufficient pam_sss.so {if "with-pammount":use_first_pass|forward_pass} @@ -32,2 +33,3 @@ session required pam_unix.so +session optional pam_mount.so {include if "with-pammount"} session optional pam_sss.so END # chroot /fc28 authselect select custom/sssd with-pammount --force</pre>
Also ensure the NFS server’s hostname is always resolvable from the client:
<pre># MY_IP=$(host -t A $MY_HOSTNAME | awk '{print $4}') # echo "$MY_IP $MY_HOSTNAME ${MY_HOSTNAME%%.*}" >> /fc28/etc/hosts</pre>
Optionally, allow all users to run sudo:
<pre># echo '%users ALL=(ALL) NOPASSWD: ALL' > /fc28/etc/sudoers.d/users</pre>
<h2>Convert the NFS Root to an iSCSI Backing-Store</h2>
Current versions of nfs-utils may have difficulty establishing a second connection from the client back to the NFS server for home directories when an nfsroot connection is already established. The client hangs when attempting to access the home directory. So, we will work around the problem by using a different protocol (iSCSI) for sharing our netboot image.
First chroot into the image to reconfigure its initramfs for booting from an iSCSI root:
<pre># for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc28/$i; done # chroot /fc28 /usr/bin/bash --login # dnf install -y iscsi-initiator-utils # sed -i 's/nfs/iscsi/' /etc/dracut.conf.d/netboot.conf # echo 'omit_drivers+=" qedi "' > /etc/dracut.conf.d/omit-qedi.conf # echo 'blacklist qedi' > /etc/modprobe.d/blacklist-qedi.conf # KERNEL=$(ls -c /lib/modules | head -n 1) # INITRD=$(find /boot -name 'init*' | grep -m 1 $KERNEL) # dracut -f $INITRD $KERNEL # logout # for i in run sys proc dev/shm dev/pts dev; do umount /fc28/$i; done # > /fc28/root/.bash_history</pre>
The qedi driver broke iscsi during testing, so it’s been disabled here.
Next, create a fc28.img <a href="https://en.wikipedia.org/wiki/Sparse_file" target="_blank" rel="noopener">sparse file</a>. This file serves as the iSCSI target’s backing store:
<pre># FC28_SIZE=$(du -ms /fc28 | cut -f 1) # dd if=/dev/zero of=/fc28.img bs=1MiB count=0 seek=$(($FC28_SIZE*2))</pre>
(If you have one available, a separate partition or disk drive can be used instead of creating a file.)
Next, format the image with a filesystem, mount it, and copy the netboot image into it:
<pre># mkfs -t xfs -L NETROOT /fc28.img # TEMP_MNT=$(mktemp -d) # mount /fc28.img $TEMP_MNT # cp -a /fc28/* $TEMP_MNT # umount $TEMP_MNT</pre>
During testing using SquashFS, the client would occasionally stutter. It seems that SquashFS does not perform well when doing random I/O from a multiprocessor client. (See also <a href="https://chrisdown.name/2018/04/17/kernel-adventures-the-curious-case-of-squashfs-stalls.html" target="_blank" rel="noopener">The curious case of stalled squashfs reads</a>.) If you want to improve throughput performance with filesystem compression, <a href="https://en.wikipedia.org/wiki/ZFS" target="_blank" rel="noopener">ZFS</a> is probably a better option.
If you need extremely high throughput from the iSCSI server (say, for hundreds of clients), it might be possible to <a href="https://en.wikipedia.org/wiki/Load_balancing_(computing)" target="_blank" rel="noopener">load balance</a> a <a href="http://docs.ceph.com/docs/mimic/rbd/iscsi-overview/" target="_blank" rel="noopener">Ceph</a> cluster. For more information, see <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ceph_example" target="_blank" rel="noopener">Load Balancing Ceph Object Gateway Servers with HAProxy and Keepalived</a>.
<h2>Install and Configure iSCSI</h2>
Install the scsi-target-utils package which will provide the iSCSI daemon for serving our image out to our clients:
<pre># dnf install -y scsi-target-utils</pre>
Configure the iSCSI daemon to serve the fc28.img file:
<pre># MY_REVERSE_HOSTNAME=$(echo $MY_HOSTNAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_HOSTNAME}) # cat << END > /etc/tgt/conf.d/fc28.conf <target iqn.$MY_REVERSE_HOSTNAME:fc28> backing-store /fc28.img readonly 1 </target> END</pre>
The leading iqn. is expected by /usr/lib/dracut/modules.d/40network/net-lib.sh.
Add an exception to the firewall and enable and start the service:
<pre># firewall-cmd --add-service=iscsi-target # firewall-cmd --runtime-to-permanent # systemctl enable tgtd.service # systemctl start tgtd.service</pre>
You should now be able to see the image being shared with the tgtadm command:
<pre># tgtadm --mode target --op show</pre>
The above command should output something similar to the following:
<pre>Target 1: iqn.edu.example.server-01:fc28 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 10488 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: Yes SWP: No Thin-provisioning: No Backing store type: rdwr Backing store path: /fc28.img Backing store flags: Account information: ACL information: ALL</pre>
We can now remove the NFS share that we created in part one of this series:
<pre># rm -f /etc/exports.d/fc28.exports # exportfs -rv # umount /export/fc28 # rmdir /export/fc28 # sed -i '/^\/fc28 /d' /etc/fstab</pre>
You can also delete the /fc28 filesystem, but you may want to keep it for performing future updates.
<h2>Update the ESP to use the iSCSI Kernel</h2>
Ipdate the ESP to contain the iSCSI-enabled initramfs:
<pre>$ rm -vf $HOME/esp/linux/*.fc28.* $ MY_KRNL=$(ls -c /fc28/lib/modules | head -n 1) $ cp $(find /fc28/lib/modules -maxdepth 2 -name 'vmlinuz' | grep -m 1 $MY_KRNL) $HOME/esp/linux/vmlinuz-$MY_KRNL $ cp $(find /fc28/boot -name 'init*' | grep -m 1 $MY_KRNL) $HOME/esp/linux/initramfs-$MY_KRNL.img </pre>
Update the boot.cfg file to pass the new root and netroot parameters:
<pre>$ MY_NAME=server-01.example.edu $ MY_EMAN=$(echo $MY_NAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_NAME}) $ MY_ADDR=$(host -t A $MY_NAME | awk '{print $4}') $ sed -i "s! root=[^ ]*! root=/dev/disk/by-path/ip-$MY_ADDR:3260-iscsi-iqn.$MY_EMAN:fc28-lun-1 netroot=iscsi:$MY_ADDR::::iqn.$MY_EMAN:fc28!" $HOME/esp/linux/boot.cfg</pre>
Now you just need to copy the updated files from your $HOME/esp/linux directory out to the ESPs of all your client systems. You should see results similar to what is shown in the below screenshot:
<img class="alignnone wp-image-23701 size-large" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/12/how-to-build-a-netboot-server-part-2.png" alt="" width="616" height="493" />
<h2>Upgrading the Image</h2>
First, make a copy of the current image:
<pre># cp -a /fc28 /fc29</pre>
Chroot into the new copy of the image:
<pre># for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc29/$i; done # chroot /fc29 /usr/bin/bash --login</pre>
Allow updating the kernel:
<pre># sed -i 's/^exclude=kernel-\*$/#exclude=kernel-*/' /etc/dnf/dnf.conf</pre>
Perform the upgrade:
<pre># dnf distro-sync -y --releasever=29</pre>
Prevent the kernel from being updated:
<pre># sed -i 's/^#exclude=kernel-\*$/exclude=kernel-*/' /etc/dnf/dnf.conf</pre>
The above command is optional, but saves you from having to copy a new kernel out to the clients if you add or update a few packages in the image at some future time.
Clean up dnf’s package cache:
<pre># dnf clean all </pre>
Exit the chroot and clear root’s command history:
<pre># logout # for i in run sys proc dev/shm dev/pts dev; do umount /fc29/$i; done # > /fc29/root/.bash_history</pre>
Create the iSCSI image:
<pre># FC29_SIZE=$(du -ms /fc29 | cut -f 1) # dd if=/dev/zero of=/fc29.img bs=1MiB count=0 seek=$(($FC29_SIZE*2)) # mkfs -t xfs -L NETROOT /fc29.img # TEMP_MNT=$(mktemp -d) # mount /fc29.img $TEMP_MNT # cp -a /fc29/* $TEMP_MNT # umount $TEMP_MNT</pre>
Define a new iSCSI target that points to our new image and export it:
<pre># MY_HOSTNAME=$(</etc/hostname) # MY_REVERSE_HOSTNAME=$(echo $MY_HOSTNAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_HOSTNAME}) # cat << END > /etc/tgt/conf.d/fc29.conf <target iqn.$MY_REVERSE_HOSTNAME:fc29> backing-store /fc29.img readonly 1 </target> END # tgt-admin --update ALL</pre>
Add the new kernel and initramfs to the ESP:
<pre>$ MY_KRNL=$(ls -c /fc29/lib/modules | head -n 1) $ cp $(find /fc29/lib/modules -maxdepth 2 -name 'vmlinuz' | grep -m 1 $MY_KRNL) $HOME/esp/linux/vmlinuz-$MY_KRNL $ cp $(find /fc29/boot -name 'init*' | grep -m 1 $MY_KRNL) $HOME/esp/linux/initramfs-$MY_KRNL.img</pre>
Update the boot.cfg in the ESP:
<pre>$ MY_DNS1=192.0.2.91 $ MY_DNS2=192.0.2.92 $ MY_NAME=server-01.example.edu $ MY_EMAN=$(echo $MY_NAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_NAME}) $ MY_ADDR=$(host -t A $MY_NAME | awk '{print $4}') $ cat << END > $HOME/esp/linux/boot.cfg #!ipxe kernel --name kernel.efi \${prefix}/vmlinuz-$MY_KRNL initrd=initrd.img ro ip=dhcp rd.peerdns=0 nameserver=$MY_DNS1 nameserver=$MY_DNS2 root=/dev/disk/by-path/ip-$MY_ADDR:3260-iscsi-iqn.$MY_EMAN:fc29-lun-1 netroot=iscsi:$MY_ADDR::::iqn.$MY_EMAN:fc29 console=tty0 console=ttyS0,115200n8 audit=0 selinux=0 quiet initrd --name initrd.img \${prefix}/initramfs-$MY_KRNL.img boot || exit END</pre>
Finally, copy the files from your $HOME/esp/linux directory out to the ESPs of all your client systems and enjoy!
</div>

xSicKxBot