pxe_boot_alma/homelab/docs/replication.md

# Replication Guide

This document describes how to reproduce the AlmaLinux PXE + Kickstart environment on a fresh `server1`.

## Topology

- `server1`: PXE services
  - runs `dnsmasq`
  - serves TFTP boot files
  - serves Alma install tree and Kickstarts over HTTP
- `server2`: install target
  - configured to check local disk first and fall back to PXE

Example addresses used in the current lab:

- `server1`: `192.168.50.1`
- host on `pxe-lab`: `192.168.50.254`
- `server2`: DHCP client on the same isolated PXE subnet

Replace these with your real values everywhere below.

## Services

You need these on `server1`:

- `dnsmasq`
- an HTTP server such as `nginx`, `httpd`, or `caddy`
- TFTP root directory

Common paths used in this repo:

- TFTP root: `/var/lib/tftpboot`
- HTTP root: `/var/www/html`
- Alma install tree: `/var/www/html/alma/10/BaseOS/x86_64/os`
- Kickstarts: `/var/www/html/kickstarts`

## 1. Install Packages On Server 1

Example for AlmaLinux or RHEL-family hosts:

```bash
sudo dnf install -y dnsmasq nginx syslinux-tftpboot grub2-efi-x64 shim-x64
```

Example for Debian or Ubuntu:

```bash
sudo apt-get update
sudo apt-get install -y dnsmasq nginx pxelinux syslinux-common grub-efi-amd64-bin shim-signed
```

Package names vary slightly by distro. Adjust as needed for your host.

## 2. Prepare HTTP And TFTP Paths

```bash
sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg
sudo mkdir -p /var/lib/tftpboot/EFI/BOOT
sudo mkdir -p /var/www/html/alma/10/BaseOS/x86_64/os
sudo mkdir -p /var/www/html/kickstarts
```

## 3. Copy AlmaLinux ISO Contents To HTTP Root

Place the ISO in this repo, then run:

```bash
./scripts/sync_alma_tree.sh \
  --iso ./AlmaLinux-10.1-x86_64-minimal.iso \
  --http-root /var/www/html/alma/10/BaseOS/x86_64/os
```

This mounts the ISO read-only and copies the contents into the HTTP install tree.

## 4. Copy PXE Boot Files

From the extracted Alma tree, copy:

- `images/pxeboot/vmlinuz`
- `images/pxeboot/initrd.img`

Example:

```bash
sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/vmlinuz /var/lib/tftpboot/vmlinuz-alma10
sudo cp /var/www/html/alma/10/BaseOS/x86_64/os/images/pxeboot/initrd.img /var/lib/tftpboot/initrd-alma10.img
```

For BIOS clients, also copy PXELINUX assets:

```bash
sudo cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/libutil.c32 /var/lib/tftpboot/
sudo cp /usr/share/syslinux/ldlinux.c32 /var/lib/tftpboot/
```

For UEFI clients, install or copy the GRUB EFI loader that matches your distro packaging.

Typical examples:

- `/boot/efi/EFI/BOOT/BOOTX64.EFI`
- `/boot/efi/EFI/almalinux/grubx64.efi`
- `/boot/efi/EFI/almalinux/shimx64.efi`

## 5. Publish Kickstart Files

Copy the Kickstart file into the HTTP root:

```bash
sudo cp kickstarts/alma10-auto.ks /var/www/html/kickstarts/alma10-auto.ks
sudo cp scripts/post-install-bootstrap.sh /var/www/html/kickstarts/post-install-bootstrap.sh
```

The sample Kickstart expects to be reachable at:

```text
http://192.168.50.1/kickstarts/alma10-auto.ks
```

Adjust the IP or hostname in:

- `kickstarts/alma10-auto.ks`
- `kickstarts/post-install-bootstrap.sh`
- `boot/grub2/grub.cfg`
- `boot/pxelinux.cfg/default`

## 6. Install dnsmasq Config

Use [`dnsmasq/dnsmasq.conf.example`](/usr/local/valherun/pxe_boot_alma/dnsmasq/dnsmasq.conf.example) as the base.

Important fields to customize:

- interface name
- DHCP range
- router and DNS values
- `dhcp-boot`
- `enable-tftp`
- `tftp-root`

Then install and restart:

```bash
sudo cp dnsmasq/dnsmasq.conf.example /etc/dnsmasq.d/pxe-alma.conf
sudo systemctl enable --now dnsmasq
sudo systemctl restart dnsmasq
```

## 7. Install PXE Menu Files

BIOS:

```bash
sudo cp boot/pxelinux.cfg/default /var/lib/tftpboot/pxelinux.cfg/default
```

UEFI:

```bash
sudo cp boot/grub2/grub.cfg /var/lib/tftpboot/grub.cfg
sudo cp boot/grub2/grub.cfg /var/lib/tftpboot/EFI/BOOT/grub.cfg
sudo cp /boot/efi/EFI/almalinux/grubx64.efi /var/lib/tftpboot/grubx64.efi
sudo chmod 644 /var/lib/tftpboot/EFI/BOOT/BOOTX64.EFI /var/lib/tftpboot/EFI/BOOT/grubx64.efi /var/lib/tftpboot/grubx64.efi
```

Some UEFI layouts expect the config under:

- `/var/lib/tftpboot/EFI/BOOT/grub.cfg`
- `/var/lib/tftpboot/grub/grub.cfg`

Use the location expected by your GRUB EFI binary.

In the current AlmaLinux 10 UEFI lab, shim fetched `EFI/BOOT/BOOTX64.EFI` and then looked for:

- `/grubx64.efi`
- `/EFI/BOOT/grub.cfg`

So the root-level `grubx64.efi` copy above was required for successful UEFI PXE handoff.

## 8. Make Sure HTTP Is Serving The Tree

Verify from another host:

```bash
curl -I http://192.168.50.1/alma/10/BaseOS/x86_64/os/
curl -I http://192.168.50.1/kickstarts/alma10-auto.ks
```

On AlmaLinux, the stock `nginx` package serves `/usr/share/nginx/html` by default. In the current lab, the published content is exposed by symlinking:

```bash
sudo ln -sfn /var/www/html/alma /usr/share/nginx/html/alma
sudo ln -sfn /var/www/html/kickstarts /usr/share/nginx/html/kickstarts
sudo nginx -t
sudo systemctl reload nginx
```

## 9. Optional: Proxy AppStream Through server1

If your PXE subnet is isolated and you want the installer to download packages without direct outbound internet access from `server2`, install `squid` on `server1` and allow the PXE subnet to use it.

```bash
sudo dnf install -y squid
sudo sed -i '/^http_access allow localhost$/a http_access allow localnet' /etc/squid/squid.conf
sudo systemctl enable --now squid
sudo systemctl restart squid
```

Validate from `server1`:

```bash
curl -I -x http://127.0.0.1:3128 https://mirrors.almalinux.org/mirrorlist/10.1/appstream
```

The current Kickstart uses the local BaseOS tree and fetches `AppStream` through the proxy:

```kickstart
url --url="http://192.168.50.1/alma/10/BaseOS/x86_64/os"
repo --name="appstream" --mirrorlist="https://mirrors.almalinux.org/mirrorlist/10.1/appstream" --proxy="http://192.168.50.1:3128"
```

The sample Kickstart commits a salted SHA-512 root password hash with:

```kickstart
rootpw --iscrypted ...
```

Only the hash is stored in Git. If you rotate the password, generate a new hash with `openssl passwd -6` and replace the `rootpw` line.

## 10. Give The Host Direct Access To pxe-lab

In the working lab, the libvirt bridge `virbr1` already existed for `pxe-lab`, but the host itself did not have an IP on that subnet. Adding one made direct host-to-guest SSH validation possible without hopping through `server1`.

Runtime command used on the host:

```bash
sudo ip addr add 192.168.50.254/24 dev virbr1
```

Validation:

```bash
ip -brief addr show dev virbr1
ip route get 192.168.50.150
ping -c 2 192.168.50.150
```

If you want that to persist across host reboots, configure the bridge address through your host network manager instead of relying on the one-shot `ip addr add` command.

## 11. Boot Server 2

Set `server2` to:

- UEFI PXE boot, or BIOS PXE boot depending on hardware
- disk first, network second

When PXE works correctly:

1. `server2` first checks the local disk
2. on a blank disk, firmware falls through to PXE
3. it gets an IP from `dnsmasq`
4. it downloads the bootloader from TFTP
5. it loads kernel + initrd
6. installer fetches the install tree from HTTP
7. Kickstart runs unattended

This boot order is important. If network boot is left ahead of disk boot, the guest can re-enter PXE after the first successful install and start a second unattended reinstall.

## 12. Post-Install Bootstrap

The sample Kickstart calls [`scripts/post-install-bootstrap.sh`](/usr/local/valherun/pxe_boot_alma/scripts/post-install-bootstrap.sh) during `%post`.

That script:

1. installs `git`, `curl`, `sudo`, and related basics if missing
2. clones your `kickstart` repo to `/usr/local/valherun/kickstart`
3. clones your `tools` repo to `/usr/local/valherun/tools`
4. optionally runs:
   - a local non-interactive `valherun` account bootstrap that mirrors your current SSH keys, sudo policy, and a committed salted password hash
   - `kickstart/bash_profiles.sh --adduser`
   - `tools/bin/install_codex_access.sh --pubkey "$CODEX_PUBKEY" --no-reload`
   - `tools/bin/update_sshd_config.sh --no-reload`

This gives you a second-stage bootstrap after the base OS install is complete.

## 13. Post-Install Validation

In the working lab, `server2` came up on `192.168.50.150` and the following direct checks from the host succeeded:

```bash
ping -c 2 192.168.50.150
ssh [email protected] 'id && hostnamectl --static && sudo -n true'
ssh [email protected] 'ls -d /usr/local/valherun/kickstart /usr/local/valherun/tools'
ssh [email protected] 'sudo systemctl is-active sshd firewalld qemu-guest-agent'
ssh [email protected] 'id'
```

Current observed state:

- `codex` SSH works directly from the host and has passwordless `sudo`
- `valherun` SSH works directly from the host
- `valherun` sudo requires a password
- the installed hostname is still `alma10-auto`

The `valherun` step is handled locally in this repo instead of calling `kickstart/bin/create_valherun.sh` because that upstream helper runs `passwd valherun`, which blocks unattended Kickstart `%post`. The local helper sets `valherun` with a salted SHA-512 password hash and the expected SSH keys. The SSH helpers are run with `--no-reload` so the install does not depend on reloading `sshd` inside the installer chroot.

The PXE bootstrap also pins the preferred `codex` public key directly in [`scripts/post-install-bootstrap.sh`](/usr/local/valherun/pxe_boot_alma/scripts/post-install-bootstrap.sh) so unattended installs do not inherit the fallback key embedded in the shared `tools` helper. Those clones run over HTTPS through the proxy on `server1`, so no local tarball mirror is required.

The post-install bootstrap also exports proxy variables by default:

```bash
http_proxy=http://192.168.50.1:3128
https_proxy=http://192.168.50.1:3128
```

## Notes About Your Existing kickstart.sh

Your current [`kickstart.sh`](/usr/local/valherun/kickstart/kickstart.sh) is useful for interactive host bootstrap, but it is not ideal for unattended Kickstart `%post` as-is because:

- it prompts by default
- `--all` re-enables the Unraid path internally

For unattended Alma installs, this repo uses the reusable sub-scripts directly instead.

## Validation Checklist

Before attempting install:

- `dnsmasq --test`
- `curl -I` against the Alma tree
- `curl -I` against the Kickstart file
- confirm `vmlinuz` and `initrd.img` exist in TFTP root
- confirm `server2` can PXE on the intended NIC and VLAN

## Suggested Next Improvements

- add a host-specific Kickstart per target machine
- add static DHCP reservation for `server2`
- add UEFI HTTP Boot if the hardware supports it
- add a wrapper to sync repo files directly into `/srv/tftp` and `/srv/www`