pxe_boot_alma/homelab/docs/virtual-lab.md

Virtual Lab Build

This document records the exact server1 setup performed in the current KVM/libvirt lab.

Lab Topology

Host machine:

• Ubuntu 24.04 with libvirt/KVM

Libvirt networks:

• default: NAT management network
• pxe-lab: isolated PXE network with no libvirt DHCP

Host-side bridge access:

• virbr1: manually assigned 192.168.50.254/24 on the KVM host for direct access to PXE-lab guests

server1 VM:

• 2 vCPU
• 4 GB RAM
• 40 GB qcow2 disk
• UEFI firmware
• NIC 1 on default
• NIC 2 on pxe-lab

Inside server1:

• enp1s0: management NIC on libvirt NAT
• enp2s0: PXE NIC set to 192.168.50.1/24

1. Base Server Access

After AlmaLinux was installed and your bootstrap scripts were run, SSH access was available as valherun, but codex public key auth was not accepted even though the codex account existed.

To repair codex access in the lab, the VM disk was edited offline from the libvirt host:

sudo virsh destroy server1
CODEx_KEY="$(ssh-add -L | awk '/ codex$/{print; exit}')"
sudo virt-customize -a /var/lib/libvirt/images/server1.qcow2 \
  --run-command "install -d -m 700 -o codex -g codex /home/codex/.ssh && \
  printf '%s\n' '$CODEx_KEY' > /home/codex/.ssh/authorized_keys && \
  chown codex:codex /home/codex/.ssh/authorized_keys && \
  chmod 600 /home/codex/.ssh/authorized_keys && \
  restorecon -Rv /home/codex/.ssh || true"
sudo virsh start server1

After the reboot, SSH as codex worked and sudo -n was available.

2. Install PXE Packages On server1

Commands run:

ssh [email protected]
sudo dnf install -y dnsmasq nginx syslinux rsync

3. Set Hostname And PXE NIC Address

Commands run:

sudo hostnamectl set-hostname server1
sudo nmcli connection modify enp2s0 \
  ipv4.method manual \
  ipv4.addresses 192.168.50.1/24 \
  ipv6.method disabled \
  connection.autoconnect yes
sudo nmcli connection up enp2s0

4. Create Content Directories

Commands run:

sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg
sudo mkdir -p /var/lib/tftpboot/EFI/BOOT
sudo mkdir -p /var/www/html/alma/10/BaseOS/x86_64/os
sudo mkdir -p /var/www/html/kickstarts
sudo mkdir -p /opt/pxe_boot_alma
sudo chown -R codex:codex /opt/pxe_boot_alma

5. Copy Repo And ISO To server1

Commands run from the libvirt host:

rsync -az --delete --exclude '.git' --exclude '*.iso' \
  /usr/local/valherun/pxe_boot_alma/ \
  [email protected]:/opt/pxe_boot_alma/

rsync -az \
  /usr/local/valherun/pxe_boot_alma/AlmaLinux-10.1-x86_64-minimal.iso \
  [email protected]:/opt/pxe_boot_alma/

6. Extract Alma Install Tree

Run on server1:

cd /opt/pxe_boot_alma
sudo ./scripts/sync_alma_tree.sh \
  --iso ./AlmaLinux-10.1-x86_64-minimal.iso \
  --http-root /var/www/html/alma/10/BaseOS/x86_64/os

7. Copy PXE Boot Files

Run on server1:

sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/vmlinuz /var/lib/tftpboot/vmlinuz-alma10
sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/initrd.img /var/lib/tftpboot/initrd-alma10.img

sudo cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/libutil.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/ldlinux.c32 /var/lib/tftpboot/

sudo cp /boot/efi/EFI/BOOT/BOOTX64.EFI /var/lib/tftpboot/EFI/BOOT/
sudo cp /boot/efi/EFI/almalinux/grubx64.efi /var/lib/tftpboot/EFI/BOOT/
sudo cp /boot/efi/EFI/almalinux/grubx64.efi /var/lib/tftpboot/grubx64.efi
sudo cp /opt/pxe_boot_alma/boot/grub2/grub.cfg /var/lib/tftpboot/grub.cfg
sudo cp /opt/pxe_boot_alma/boot/grub2/grub.cfg /var/lib/tftpboot/EFI/BOOT/grub.cfg
sudo cp /opt/pxe_boot_alma/boot/pxelinux.cfg/default /var/lib/tftpboot/pxelinux.cfg/default
sudo chmod 644 /var/lib/tftpboot/EFI/BOOT/BOOTX64.EFI /var/lib/tftpboot/EFI/BOOT/grubx64.efi /var/lib/tftpboot/grubx64.efi

UEFI-specific note:

• shim requested /EFI/BOOT/BOOTX64.EFI
• then requested /grubx64.efi
• then loaded /EFI/BOOT/grub.cfg

Without the root-level grubx64.efi copy, UEFI PXE did not progress past shim.

8. Publish Kickstart And Post-Install Bootstrap

Run on server1:

sudo cp /opt/pxe_boot_alma/kickstarts/alma10-auto.ks /var/www/html/kickstarts/alma10-auto.ks
sudo cp /opt/pxe_boot_alma/scripts/post-install-bootstrap.sh /var/www/html/kickstarts/post-install-bootstrap.sh

9. Expose HTTP Content Through Alma's nginx Default Root

The stock AlmaLinux nginx package serves /usr/share/nginx/html, so these symlinks were created:

sudo mkdir -p /usr/share/nginx/html
sudo ln -sfn /var/www/html/alma /usr/share/nginx/html/alma
sudo ln -sfn /var/www/html/kickstarts /usr/share/nginx/html/kickstarts
sudo nginx -t
sudo systemctl reload nginx

10. Install dnsmasq Config And Enable Services

Run on server1:

sudo cp /opt/pxe_boot_alma/dnsmasq/dnsmasq.conf.example /etc/dnsmasq.d/pxe-alma.conf
sudo dnsmasq --test

sudo systemctl enable --now nginx
sudo systemctl enable --now dnsmasq
sudo systemctl restart nginx
sudo systemctl restart dnsmasq

11. Install squid For AppStream Proxying

The missing installer packages (git, vim-enhanced, and wget) came from AppStream, not BaseOS. To keep the PXE subnet isolated while still allowing those downloads, squid was installed on server1:

sudo dnf install -y squid
sudo sed -i '/^http_access allow localhost$/a http_access allow localnet' /etc/squid/squid.conf
sudo systemctl enable --now squid
sudo systemctl restart squid
curl -I -x http://127.0.0.1:3128 https://mirrors.almalinux.org/mirrorlist/10.1/appstream

Kickstart change:

url --url="http://192.168.50.1/alma/10/BaseOS/x86_64/os"
repo --name="appstream" --mirrorlist="https://mirrors.almalinux.org/mirrorlist/10.1/appstream" --proxy="http://192.168.50.1:3128"

The post-install bootstrap script was also updated to export:

http_proxy=http://192.168.50.1:3128
https_proxy=http://192.168.50.1:3128

During the first unattended server2 attempt, the install hung at Running post-install scripts. The cause was kickstart/bin/create_valherun.sh, which calls passwd valherun interactively. The PXE bootstrap was then changed to create valherun locally and non-interactively inside scripts/post-install-bootstrap.sh, while still cloning the kickstart and tools repos for the remaining bootstrap steps.

The final lab flow uses direct HTTPS clones from Gitea through the squid proxy on server1. That keeps the PXE subnet isolated without maintaining local tarball mirrors for kickstart and tools. The SSH helpers are run with --no-reload so the install does not rely on systemctl reload sshd inside the installer chroot.

The PXE bootstrap was also updated to pass the preferred codex public key explicitly into tools/bin/install_codex_access.sh. That avoids the older fallback codex@valh-mini key embedded in the shared helper and keeps unattended installs aligned with the preferred Bitwarden-managed codex key.

12. Firewall

firewalld was active on server1. The PXE interface should be trusted:

sudo firewall-cmd --permanent --zone=trusted --add-interface=enp2s0
sudo firewall-cmd --reload

If HTTP is restricted on the management side, also allow it:

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload

13. Validation

Run on server1:

ip -br a
sudo dnsmasq --test
curl -I http://192.168.50.1/kickstarts/alma10-auto.ks
curl -I http://192.168.50.1/alma/10/BaseOS/x86_64/os/
ls -l /var/lib/tftpboot

14. Host Access To pxe-lab

Direct host access to server2 was added later, because validating through server1 was misleading:

• server1 blocked ProxyJump with AllowTcpForwarding no
• server1 blocked agent forwarding for codex

So the host was given a direct IP on virbr1:

sudo ip addr add 192.168.50.254/24 dev virbr1
ip -brief addr show dev virbr1

After that, direct host-to-guest validation worked:

ping -c 2 192.168.50.150
ssh [email protected] 'id && sudo -n true'
ssh [email protected] 'ls -d /usr/local/valherun/kickstart /usr/local/valherun/tools'
ssh [email protected] 'sudo systemctl is-active sshd firewalld qemu-guest-agent'
ssh [email protected] 'id'

Observed behavior:

• codex SSH works directly and has passwordless sudo
• valherun SSH works directly
• valherun sudo prompts for a password

15. Server2 Boot Order

The reliable boot order for server2 in this lab is:

• hd first
• network second

Why:

• on a blank disk, firmware falls through to PXE and starts the install
• after install, the guest prefers the installed disk and does not re-enter Kickstart

Leaving PXE first caused server2 to start a second unattended reinstall after the first successful reboot.

Next Step

Customize the Kickstart so the installed hostname is not the generic alma10-auto, and decide whether valherun should keep password-required sudo or move to passwordless sudo.