pxe_boot_alma/homelab/docs/virtual-lab.md

# Virtual Lab Build

This document records the exact `server1` setup performed in the current KVM/libvirt lab.

## Lab Topology

Host machine:

- Ubuntu 24.04 with libvirt/KVM

Libvirt networks:

- `default`: NAT management network
- `pxe-lab`: isolated PXE network with no libvirt DHCP

Host-side bridge access:

- `virbr1`: manually assigned `192.168.50.254/24` on the KVM host for direct access to PXE-lab guests

`server1` VM:

- 2 vCPU
- 4 GB RAM
- 40 GB qcow2 disk
- UEFI firmware
- NIC 1 on `default`
- NIC 2 on `pxe-lab`

Inside `server1`:

- `enp1s0`: management NIC on libvirt NAT
- `enp2s0`: PXE NIC set to `192.168.50.1/24`

## 1. Base Server Access

After AlmaLinux was installed and your bootstrap scripts were run, SSH access was available as `valherun`, but `codex` public key auth was not accepted even though the `codex` account existed.

To repair `codex` access in the lab, the VM disk was edited offline from the libvirt host:

```bash
sudo virsh destroy server1
CODEx_KEY="$(ssh-add -L | awk '/ codex$/{print; exit}')"
sudo virt-customize -a /var/lib/libvirt/images/server1.qcow2 \
  --run-command "install -d -m 700 -o codex -g codex /home/codex/.ssh && \
  printf '%s\n' '$CODEx_KEY' > /home/codex/.ssh/authorized_keys && \
  chown codex:codex /home/codex/.ssh/authorized_keys && \
  chmod 600 /home/codex/.ssh/authorized_keys && \
  restorecon -Rv /home/codex/.ssh || true"
sudo virsh start server1
```

After the reboot, SSH as `codex` worked and `sudo -n` was available.

## 2. Install PXE Packages On server1

Commands run:

```bash
ssh [email protected]
sudo dnf install -y dnsmasq nginx syslinux rsync
```

## 3. Set Hostname And PXE NIC Address

Commands run:

```bash
sudo hostnamectl set-hostname server1
sudo nmcli connection modify enp2s0 \
  ipv4.method manual \
  ipv4.addresses 192.168.50.1/24 \
  ipv6.method disabled \
  connection.autoconnect yes
sudo nmcli connection up enp2s0
```

## 4. Create Content Directories

Commands run:

```bash
sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg
sudo mkdir -p /var/lib/tftpboot/EFI/BOOT
sudo mkdir -p /var/www/html/alma/10/BaseOS/x86_64/os
sudo mkdir -p /var/www/html/kickstarts
sudo mkdir -p /opt/pxe_boot_alma
sudo chown -R codex:codex /opt/pxe_boot_alma
```

## 5. Copy Repo And ISO To server1

Commands run from the libvirt host:

```bash
rsync -az --delete --exclude '.git' --exclude '*.iso' \
  /usr/local/valherun/pxe_boot_alma/ \
  [email protected]:/opt/pxe_boot_alma/

rsync -az \
  /usr/local/valherun/pxe_boot_alma/AlmaLinux-10.1-x86_64-minimal.iso \
  [email protected]:/opt/pxe_boot_alma/
```

## 6. Extract Alma Install Tree

Run on `server1`:

```bash
cd /opt/pxe_boot_alma
sudo ./scripts/sync_alma_tree.sh \
  --iso ./AlmaLinux-10.1-x86_64-minimal.iso \
  --http-root /var/www/html/alma/10/BaseOS/x86_64/os
```

## 7. Copy PXE Boot Files

Run on `server1`:

```bash
sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/vmlinuz /var/lib/tftpboot/vmlinuz-alma10
sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/initrd.img /var/lib/tftpboot/initrd-alma10.img

sudo cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/libutil.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/ldlinux.c32 /var/lib/tftpboot/

sudo cp /boot/efi/EFI/BOOT/BOOTX64.EFI /var/lib/tftpboot/EFI/BOOT/
sudo cp /boot/efi/EFI/almalinux/grubx64.efi /var/lib/tftpboot/EFI/BOOT/
sudo cp /boot/efi/EFI/almalinux/grubx64.efi /var/lib/tftpboot/grubx64.efi
sudo cp /opt/pxe_boot_alma/boot/grub2/grub.cfg /var/lib/tftpboot/grub.cfg
sudo cp /opt/pxe_boot_alma/boot/grub2/grub.cfg /var/lib/tftpboot/EFI/BOOT/grub.cfg
sudo cp /opt/pxe_boot_alma/boot/pxelinux.cfg/default /var/lib/tftpboot/pxelinux.cfg/default
sudo chmod 644 /var/lib/tftpboot/EFI/BOOT/BOOTX64.EFI /var/lib/tftpboot/EFI/BOOT/grubx64.efi /var/lib/tftpboot/grubx64.efi
```

UEFI-specific note:

- shim requested `/EFI/BOOT/BOOTX64.EFI`
- then requested `/grubx64.efi`
- then loaded `/EFI/BOOT/grub.cfg`

Without the root-level `grubx64.efi` copy, UEFI PXE did not progress past shim.

## 8. Publish Kickstart And Post-Install Bootstrap

Run on `server1`:

```bash
sudo cp /opt/pxe_boot_alma/kickstarts/alma10-auto.ks /var/www/html/kickstarts/alma10-auto.ks
sudo cp /opt/pxe_boot_alma/scripts/post-install-bootstrap.sh /var/www/html/kickstarts/post-install-bootstrap.sh
```

## 9. Expose HTTP Content Through Alma's nginx Default Root

The stock AlmaLinux `nginx` package serves `/usr/share/nginx/html`, so these symlinks were created:

```bash
sudo mkdir -p /usr/share/nginx/html
sudo ln -sfn /var/www/html/alma /usr/share/nginx/html/alma
sudo ln -sfn /var/www/html/kickstarts /usr/share/nginx/html/kickstarts
sudo nginx -t
sudo systemctl reload nginx
```

## 10. Install dnsmasq Config And Enable Services

Run on `server1`:

```bash
sudo cp /opt/pxe_boot_alma/dnsmasq/dnsmasq.conf.example /etc/dnsmasq.d/pxe-alma.conf
sudo dnsmasq --test

sudo systemctl enable --now nginx
sudo systemctl enable --now dnsmasq
sudo systemctl restart nginx
sudo systemctl restart dnsmasq
```

## 11. Install squid For AppStream Proxying

The missing installer packages (`git`, `vim-enhanced`, and `wget`) came from `AppStream`, not `BaseOS`. To keep the PXE subnet isolated while still allowing those downloads, `squid` was installed on `server1`:

```bash
sudo dnf install -y squid
sudo sed -i '/^http_access allow localhost$/a http_access allow localnet' /etc/squid/squid.conf
sudo systemctl enable --now squid
sudo systemctl restart squid
curl -I -x http://127.0.0.1:3128 https://mirrors.almalinux.org/mirrorlist/10.1/appstream
```

Kickstart change:

```kickstart
url --url="http://192.168.50.1/alma/10/BaseOS/x86_64/os"
repo --name="appstream" --mirrorlist="https://mirrors.almalinux.org/mirrorlist/10.1/appstream" --proxy="http://192.168.50.1:3128"
```

The post-install bootstrap script was also updated to export:

```bash
http_proxy=http://192.168.50.1:3128
https_proxy=http://192.168.50.1:3128
```

During the first unattended `server2` attempt, the install hung at `Running post-install scripts`. The cause was `kickstart/bin/create_valherun.sh`, which calls `passwd valherun` interactively. The PXE bootstrap was then changed to create `valherun` locally and non-interactively inside [`scripts/post-install-bootstrap.sh`](/usr/local/valherun/pxe_boot_alma/scripts/post-install-bootstrap.sh), while still cloning the `kickstart` and `tools` repos for the remaining bootstrap steps.

The final lab flow uses direct HTTPS clones from Gitea through the `squid` proxy on `server1`. That keeps the PXE subnet isolated without maintaining local tarball mirrors for `kickstart` and `tools`. The SSH helpers are run with `--no-reload` so the install does not rely on `systemctl reload sshd` inside the installer chroot.

The PXE bootstrap was also updated to pass the preferred `codex` public key explicitly into `tools/bin/install_codex_access.sh`. That avoids the older fallback `codex@valh-mini` key embedded in the shared helper and keeps unattended installs aligned with the preferred Bitwarden-managed `codex` key.

## 12. Firewall

`firewalld` was active on `server1`. The PXE interface should be trusted:

```bash
sudo firewall-cmd --permanent --zone=trusted --add-interface=enp2s0
sudo firewall-cmd --reload
```

If HTTP is restricted on the management side, also allow it:

```bash
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload
```

## 13. Validation

Run on `server1`:

```bash
ip -br a
sudo dnsmasq --test
curl -I http://192.168.50.1/kickstarts/alma10-auto.ks
curl -I http://192.168.50.1/alma/10/BaseOS/x86_64/os/
ls -l /var/lib/tftpboot
```

## 14. Host Access To pxe-lab

Direct host access to `server2` was added later, because validating through `server1` was misleading:

- `server1` blocked `ProxyJump` with `AllowTcpForwarding no`
- `server1` blocked agent forwarding for `codex`

So the host was given a direct IP on `virbr1`:

```bash
sudo ip addr add 192.168.50.254/24 dev virbr1
ip -brief addr show dev virbr1
```

After that, direct host-to-guest validation worked:

```bash
ping -c 2 192.168.50.150
ssh [email protected] 'id && sudo -n true'
ssh [email protected] 'ls -d /usr/local/valherun/kickstart /usr/local/valherun/tools'
ssh [email protected] 'sudo systemctl is-active sshd firewalld qemu-guest-agent'
ssh [email protected] 'id'
```

Observed behavior:

- `codex` SSH works directly and has passwordless `sudo`
- `valherun` SSH works directly
- `valherun` `sudo` prompts for a password

## 15. Server2 Boot Order

The reliable boot order for `server2` in this lab is:

- `hd` first
- `network` second

Why:

- on a blank disk, firmware falls through to PXE and starts the install
- after install, the guest prefers the installed disk and does not re-enter Kickstart

Leaving PXE first caused `server2` to start a second unattended reinstall after the first successful reboot.

## Next Step

Customize the Kickstart so the installed hostname is not the generic `alma10-auto`, and decide whether `valherun` should keep password-required `sudo` or move to passwordless `sudo`.