pxe_boot_alma/homelab/docs/alma10-kickstart-changes.md

# AlmaLinux 10 Kickstart — Changes & Known Issues

**Created:** 2026-03-27
**Scope:** Production kickstart build — `work/KickStart/cfg/AlmaLinux-10.1-x86_64/`
**Environment:** Monolith at `10.1.1.1` provides dnsmasq (DHCP+TFTP), Squid proxy (`:3128`), and HTTP for kickstart and repo. Probes/servers have no other network access during install.

---

## Architecture

- KickStart delivered via CGI: `http://10.1.1.1/KickStart/cfg/ks.cgi?DIST=AlmaLinux-10.1-x86_64&DISKTYPE=ssd&TYPE=probe|server`
- Install tree: DVD ISO (not minimal/lite) — required for `@Server`, `@Development Tools`, and other groups
- tftpboot root: `/home/kickstart/tftpboot` on monolith
- Post-install scripts: modular `ks.d/` (N/P/S series), executed in sort order

---

## Install Source Change: Minimal → DVD

Switched from `AlmaLinux-10.1-x86_64-minimal.iso` to `AlmaLinux-10.1-x86_64-dvd.iso`.

**Why:** The minimal ISO does not include package groups (`@Server`, `@Development Tools`, `@Security Tools`, etc.) or many individual packages listed in `KS_Packages.cfg`. Using it caused anaconda to abort with missing package errors and also caused GDM/initial-setup failures on the minimal environment.

---

## KS_Packages.cfg — Removed Packages

The following packages were removed from `KS_Packages.cfg` with justification:

| Package | Reason |
|---|---|
| `perl-core` | Meta-package removed in RHEL 10; individual `perl-*` packages are preferred |
| `dracut-fips` | Hard-removed in RHEL 10 — FIPS is boot-time only (`fips=1` kernel param). This project does not use FIPS. |
| `dracut-fips-aesni` | Same as above |
| `libXScrnSaver` | Merged into `libX11` in RHEL 10; no longer a separate package |
| `usbguard` | `usbguard-selinux` scriptlet crashes anaconda's RPM transaction during kickstart. Moved to `S930A-security` (installed via `dnf` in `%post`). |
| `perl-NetAddr-IP` | EPEL-only on RHEL 10; not available from DVD. Moved to `S105A-extra-pkgs` EPELPKGS. |
| `perl-Net-DNS` | EPEL-only on RHEL 10; not available from DVD. Moved to `S105A-extra-pkgs` EPELPKGS. |

### lshw — Confirmed Present in DVD

`lshw` is available in the AlmaLinux 10.1 DVD AppStream repo. An earlier filename-based check produced a false miss because the package version string starts with a letter (`lshw-B.02.20...`), defeating a naive `sed 's/-[0-9].*//'` version strip. Confirmed by test VM install — `lshw` installs cleanly.

---

## S850A-network — dhclient and ifcfg Removed in RHEL 10

**Changes made:**

1. Removed the `dhclient` block entirely — `dhclient` package is removed in RHEL 10. NetworkManager uses its built-in DHCP client (`dhcp=internal`).
2. Removed the `cd /etc/sysconfig/network-scripts` block — `ifcfg` file format is removed in RHEL 10. NetworkManager uses keyfiles at `/etc/NetworkManager/system-connections/`.
3. Replaced ifcfg DNS overrides with a Python `configparser` approach that modifies the NM keyfile directly:

```python
import configparser, os, sys, glob
condir, first = sys.argv[1], sys.argv[2]
candidates = (
    glob.glob(os.path.join(condir, first + '.nmconnection')) +
    glob.glob(os.path.join(condir, '*.nmconnection'))
)
for path in candidates:
    cfg = configparser.RawConfigParser()
    cfg.optionxform = str
    cfg.read(path)
    if not cfg.has_section('ipv4'):
        continue
    if cfg.get('ipv4', 'method', fallback='') not in ('auto', 'dhcp'):
        continue
    cfg.set('ipv4', 'dns', '192.168.194.102;192.168.226.102;')
    cfg.set('ipv4', 'dns-search', 'iglass.net;')
    cfg.set('ipv4', 'dns-priority', '-100')
    with open(path, 'w') as fh:
        cfg.write(fh)
    os.chmod(path, 0o600)
    break
```

---

## S200A-systemctl — GDM / initial-setup Guards

Added `|| true` and file existence checks around service disabling and GDM config modification, because the DVD minimal environment may or may not install GDM or initial-setup. Without these guards, the script errored out and halted kickstart.

```bash
for _svc in initial-setup-graphical.service initial-setup-reconfiguration.service \
            initial-setup-text.service initial-setup.service; do
   systemctl disable "$_svc" 2>/dev/null && echo "Disabled $_svc" || true
done
if [ -f /etc/gdm/custom.conf ]; then
   perl -spi -e 's/\[daemon\]/\[daemon\]\nInitialSetupEnable=False/' /etc/gdm/custom.conf
fi
```

---

## S930A-security — usbguard Moved to Post-Install

**Problem:** `usbguard-selinux` contains a scriptlet that crashes the anaconda RPM transaction with:

```
Error in <unknown> scriptlet in rpm package usbguard-selinux
```

This does not happen in a running system — only during anaconda's install phase.

**Fix:** Removed `usbguard` from `KS_Packages.cfg`. `S930A-security` now installs it via `dnf` after boot:

```bash
dnf install -y usbguard 2>&1
if [ -e /bin/usbguard ]; then
   /bin/usbguard generate-policy > /etc/usbguard/rules.conf
   perl -spi -e 's/ImplicitPolicyTarget=\S+/ImplicitPolicyTarget=reject/' \
       /etc/usbguard/usbguard-daemon.conf
   systemctl disable usbguard.service
fi
```

---

## S940A-cis-hardening — New Script

Added CIS Level 1 hardening script. See [alma10-cis-hardening.md](alma10-cis-hardening.md) for full details on what was applied, deferred, and excluded.

---

## S105A-extra-pkgs — Known Alma 10 Compatibility Issues

This script has several items that need Alma 10 review before production use:

| Item | Issue | Action Needed |
|---|---|---|
| `dnf config-manager --set-enabled powertools` | `powertools` repo does not exist in RHEL 10 — superseded by `crb`. **Removed** (2026-03-27). CRB is already enabled by the lines that follow. | Done |
| MS repo URL (`rhel/9.0/prod.repo`) | Updated to `rhel/10/prod.repo` (2026-03-27). Package renamed: `mssql-tools` → `mssql-tools18`. `msodbcsql18` name unchanged. Confirmed present at `https://packages.microsoft.com/rhel/10/prod/`. | Done |
| `dnf -y module install nodejs:18/common` | DNF AppStream modules removed in RHEL 10; `nodejs` is a regular package. **Replaced** (2026-03-27) with `dnf -y install nodejs`. RHEL 10 AppStream ships nodejs 22. | Done |
| PowerShell `powershell-7.3.4-1.rh.x86_64.rpm` | RHEL 9 build retained; WS-Man support required. No RHEL 10 build available as of 2026-03-27. TODO comment added. | Verify RHEL 10 RPM when available at https://github.com/PowerShell/PowerShell/releases |
| `@Xfce` in `EPELPKGS` | Alma 10 snapshot stage `3` failed on 2026-03-29 with `Module or Group 'Xfce' is not available.` It is now optional in the Alma 10 script and skipped when unavailable. | Re-test stage `3` |
| `perl-NetAddr-IP`, `perl-Net-DNS` | Added to `EPELPKGS` (2026-03-27) — EPEL-only, removed from `KS_Packages.cfg`. | Done |

---

## Server RPM Dependency Coverage — Pending

Only the `probe` RPM was available for dependency analysis (2026-03-27). A `server` RPM exists but was not available in this repo at time of review. The following packages were moved from `PERLPKGS` (dnf) to `cpanm` because RHEL 10 has no RPM for them — they are retained in cpanm to cover potential server usage until a server RPM can be checked:

- `Crypt::SSLeay` — removed from RHEL 10 (superseded by `Net::SSLeay`); not used by probe but retained for server
- `HTML::Restrict` — no EPEL 10 package; used by probe and possibly server
- `Cache::FastMmap` — no EPEL 10 package; used by probe and possibly server

**Action:** When a server RPM is available, run the same `rpm -ql | xargs grep '^use '` analysis and cross-reference against the cpanm and dnf lists.

---

## EPEL Dependency Chain

The `S105A-extra-pkgs` script enables EPEL via `dnf -y install epel-release` before installing `EPELPKGS`. The following packages are EPEL-only and must remain in that section rather than `KS_Packages.cfg`:

- `perl-NetAddr-IP`
- `perl-Net-DNS`

These were confirmed EPEL-only by test VM install (2026-03-27): present in EPEL 10, not in DVD BaseOS or AppStream.

---

## net-snmp-perl Perl Dependencies

`net-snmp-perl` does **not** directly require `perl-NetAddr-IP` or `perl-Net-DNS`. They are likely pulled in by local monitoring Perl scripts (in the `iglass` application stack), not by RPM-level dependencies of `net-snmp-perl`. They must be installed; remove them only after confirming no monitoring scripts reference them.

---

## authselect / pam_faillock Post-IPA Note

`faillock.conf` is pre-configured by `S940A-cis-hardening`, but `pam_faillock` is not active until authselect enables it. This **must** run after IPA enrolment:

```bash
authselect enable-feature with-faillock
```

Do not run this in kickstart — `ipa-client-install` owns the authselect profile and will reset it.

---

## AIDE

AIDE is installed via `KS_Packages.cfg` but cron-based scanning is managed centrally (not locally). No local AIDE cron is configured in kickstart. A central machine runs AIDE against these hosts.

---

## Symlinks Restored (Windows Transit)

15 `V`/`P` variant scripts that are symlinks to their `A` counterparts were zeroed out during a Linux→Windows→Linux filesystem round-trip. All restored (2026-03-27/28):

| Symlink | Target |
|---|---|
| S105V-extra-pkgs | S105A-extra-pkgs |
| S150V-etc-mod | S150A-etc-mod |
| S200V-systemctl | S200A-systemctl |
| S350V-chrony | S350A-chrony |
| S570V-Postgres | S570A-Postgres |
| S650V-Crontabs | S650A-Crontabs |
| S701P-chroot-maps | S701A-chroot-maps |
| S701V-chroot-maps | S701A-chroot-maps |
| S800V-netsnmp | S800A-netsnmp |
| S856V-unbound | S856A-unbound |
| S870V-sshd | S870A-sshd |
| S875V-firewall | S875A-firewall |
| S960V-selinux | S960A-selinux |
| S980V-monitor | S980A-monitor |
| S990V-mailme | S990A-mailme |
| S104V-confluent | S104P-confluent |
| S400V-noc-iglass | S400P-noc-iglass |
| S700V-chroot-as400 | S700P-chroot-as400 |
| S900P-rmlogrotate | S900S-rmlogrotate |
| S900V-rmlogrotate | S900S-rmlogrotate |
| S910V-chmodHosts | S910P-chmodHosts |

---

## S000A-start — NIC Detection Fix

The original script hardcoded `eth0` as the fallback interface name. On RHEL 10 with predictable interface naming (`eno1`, `em1`, `p1p1`, etc.), this fails silently. Fixed to detect the first interface with a MAC address:

```bash
ST=$(ip -o link show | grep -v '\blo\b' | perl -ne 'if (/link\/ether\s+(\S+)/) { print "$1\n"; last }')
```

---

## S104P-confluent — Simplified: Standard RHEL 10 Repos Sufficient

**Status (2026-03-28):** Confluent publishes `librdkafka` for EL10 at `clients/rpm/centos/10/x86_64/`, but RHEL 10 ships `librdkafka` in AppStream and `librdkafka-devel` in CRB. The Confluent package (`librdkafka1`) conflicts with the AppStream package (`librdkafka`).

**Resolution:** S104P now enables CRB and installs from standard repos — no Confluent clients repo needed:

```bash
dnf config-manager --set-enabled crb
dnf install -y librdkafka librdkafka-devel
```

Verified on alma10-pkgtest (2026-03-28): `librdkafka-2.1.1-7.el10` from AppStream, `librdkafka-devel-2.1.1-7.el10` from CRB, `Net::Kafka` builds successfully.

The `confluent.repo` file (deployed via `etc.tgz`) still provides the `[Confluent]` platform repo (rpm/8.2) for potential future use, but `[Confluent-Clients]` was removed to avoid the package naming conflict.

---

## S105A-extra-pkgs — PERLPKGS and cpanm Changes

### PERLPKGS (dnf-installed Perl modules)

Packages removed (no RHEL 10 / EPEL 10 RPM available — moved to cpanm):
- `perl-Crypt-SSLeay` — superseded by `Net::SSLeay` in RHEL 10
- `perl-HTML-Restrict` — no EPEL 10 package
- `perl-Cache-FastMmap` — no EPEL 10 package

Package added:
- `perl-IPC-Run` — available in RHEL 10 AppStream

### EPELPKGS additions

Added to cover probe dependencies not in BaseOS/AppStream:
- `perl-NetAddr-IP`
- `perl-Net-DNS`
- `perl-Regexp-Common`
- `perl-WWW-Mechanize`
- `perl-Protocol-WebSocket`
- `perl-Crypt-Rijndael`
- `perl-Net-SSH-Perl`

### cpanm list changes

Removed (now covered by EPELPKGS above):
- `IO::Select` (stdlib)
- `HTML::Entities` (part of `HTML::Parser`, available as RPM)
- `Regexp::Common`
- `WWW::Mechanize`
- `Protocol::WebSocket`
- `Crypt::Rijndael`

Added (no RPM available, retained for probe and potential server use):
- `HTML::Restrict`
- `Cache::FastMmap`
- `Net::Address::IP::Local`
- `Net::Ping::External`
- `Net::SIP`
- `Net::SSH`
- `Net::Telnet`

### cpanm Smoke Test Results (2026-03-28, alma10-pkgtest)

22 modules tested via `cpanm -n` on AlmaLinux 10.1. Two failures:

| Module | Result | Action |
|---|---|---|
| `Crypt::SSLeay` | **Hard fail** — uses `SSLv2/SSLv3_client_method()` removed in OpenSSL 3.x | **Removed** from cpanm list. RHEL 10 ships OpenSSL 3.x; this module is incompatible and unmaintained (last release 2016). Use `Net::SSLeay` (available as `perl-Net-SSLeay` RPM) instead. |
| `Net::Kafka` | **Configure fail** — needs `librdkafka-devel` (`rdkafka.pc`) | **Resolved:** `librdkafka-devel` is in CRB, installed by S104P. `Net::Kafka` now builds cleanly. Conditional guard removed. |

All 22 modules install successfully once S104P has run (librdkafka-devel in place).

---

## S150A-etc-mod — systemctl --user Guards

`systemctl --user` commands for masking PulseAudio fail during kickstart post-install because no user session is running. Guarded with `2>/dev/null || true`:

```bash
systemctl --user mask pulseaudio             2>/dev/null || true
systemctl --user disable pulseaudio.socket   2>/dev/null || true
systemctl --user disable pulseaudio.service  2>/dev/null || true
```

---

## S400P-noc-iglass — init.d → systemd Migration

The `.init` copy loop was replaced with systemd unit file installation (2026-03-28). Key findings:

- `iFunctions.init` is a bash function library, not a service init script — it does not need to go in `/etc/init.d/`
- The iGlass RPM already ships 24 systemd unit files in `/opt/iglass/systemd/system/` (services + timers)
- `ctl-iglass.service` is the master controller that starts all configured iglass services via `properties.pl`

New block installs all `.service` and `.timer` files to `/etc/systemd/system/` and runs `daemon-reload`. Services are **not enabled** during kickstart — `ctl-iglass.service` requires probe configuration (customer/hostname) which is set post-install.

---

## S570A-Postgres — pgdg Repo and useradd Fix

Three changes:

1. **pgdg repo URL:** `EL-9-x86_64` → `EL-10-x86_64`
2. **Module disable:** `dnf -y module disable postgresql` commented out with TODO — DNF AppStream modules are removed in RHEL 10, this command would fail
3. **useradd flag:** `-n` (no user private group) changed to `-N` — the flag was renamed in RHEL 10

---

## S700P-chroot-as400 / S701A-chroot-maps — AlmaLinux 10 Version Check

Both scripts checked `^VERSION="9"` in `/etc/os-release`. Updated to accept RHEL 9 or 10:

```bash
if ! grep -qE '^VERSION="(9|10)' /etc/os-release; then
   echo "This script is for AlmaLinux 9 or 10"
```

**Note:** `mkjail.pl` and the full chroot setup have not been tested on RHEL 10. Test on first physical build.

---

## S750A-serialLogin — /etc/securetty Guard

`/etc/securetty` was removed in RHEL 9.2+ and does not exist in RHEL 10. Guarded the append:

```bash
if [ -f /etc/securetty ] && ! grep -q "$TTYS" /etc/securetty; then
   echo "$TTYS" >> /etc/securetty
fi
```

**Pending:** Verify that `rhgb quiet` is present in `GRUB_CMDLINE_LINUX` on RHEL 10 — the regex that removes it may not match the default RHEL 10 grub config. Check on first physical probe build.

---

## S900S-rmlogrotate — Timer-Based logrotate (RHEL 10)

In RHEL 10, logrotate runs via `logrotate.timer` (systemd timer), not just a cron job. Both are now disabled:

```bash
# Disable cron-based logrotate (RHEL 9 and earlier)
if [ -f /etc/cron.daily/logrotate ]; then
   sed -i '2s/^/exit\n/' /etc/cron.daily/logrotate
fi

# Disable timer-based logrotate (RHEL 10+)
systemctl disable --now logrotate.timer 2>/dev/null || true
rm -f /etc/systemd/system/timers.target.wants/logrotate.timer
```

S900P and S900V are symlinks to S900S (not separate scripts).

---

## S920V-vm-cleanup — dhclient Block Removed

The script previously migrated `/etc/dhcp/dhclient-em1.conf` from the installer environment into the installed system. `dhclient` is removed in RHEL 10 — NetworkManager uses its built-in DHCP client. The block was removed:

```bash
# dhclient removed in RHEL 10 — dhclient.conf migration no longer needed
```

The interactive `read -p 'Customer:'` / `read -p 'Hostname/Probe:'` prompts are intentional — this script is run interactively during VM provisioning, not as part of the automated kickstart sequence.

---

## S980A-monitor — SSSD enumerate Removed

SSSD `enumerate = true` is not supported for AD or IdM providers in RHEL 10 (removed from those backends). The option was removed from the sssd.conf block. A comment was left explaining why:

```bash
# NOTE: 'enumerate = true' was removed in RHEL 10 for AD/IdM SSSD providers.
```

---

## S990A-mailme — NIC Detection and SSH Protocol Fix

Two changes:

1. **NIC detection:** Replaced hardcoded `eth0` with a fallback chain covering predictable interface names, then `ip link` as a last resort:

```bash
ETH=
for _candidate in eth0 em1 p1p1 eno1; do
   if [ -e "/sys/class/net/$_candidate" ]; then
      ETH="$_candidate"
      break
   fi
done
if [ -z "$ETH" ]; then
   ETH=$(ip -o link show | awk -F': ' '!/lo/{print $2; exit}')
fi
```

2. **SSH config:** Removed deprecated `Protocol 2` directive from the inline SSH config block — `Protocol` is not a valid sshd_config option in OpenSSH 7.6+ and causes a parse error.

---

## S995A-firmware — Firmware Update Deferred

A TODO comment was added. Dell BIN format firmware update support on RHEL 10 needs verification before re-enabling. No functional change to the script.

---

## P200A-diskpart — NVMe and VirtIO Disk Support

**Problem:** The disk enumeration loop only scanned `sd*` and `hd*` device names. NVMe (`nvme*`) and VirtIO (`vd*`) disks were silently skipped, causing the script to log `"Error: No Disks Found!!!"` on any system without a SCSI/SATA disk.

**Fix 1 — disk enumeration (line 14):**

```bash
for DSK in $(cd /sys/block; ls -d sd* 2>/dev/null; ls -d hd* 2>/dev/null; ls -d nvme* 2>/dev/null; ls -d vd* 2>/dev/null); do
```

**Fix 2 — NVMe partition naming:** NVMe partitions use a `p` separator (`nvme0n1p1`) while SCSI/VirtIO do not (`sda1`, `vda1`). Added partition prefix variables after disk selection:

```bash
DEVAP=""; [[ "$DEVA" == nvme* ]] && DEVAP="p"
DEVBP=""; [[ "$DEVB" == nvme* ]] && DEVBP="p"
```

All `--onpart=` references updated to `${DEVA}${DEVAP}$PRT` and `${DEVB}${DEVBP}$PRT` (8 substitutions across EFI, MBR, and second-disk blocks).

---

## no.R010P-1passwd — /mnt/sysimage Path (Disabled)

This script is disabled (`no.` prefix). If ever re-enabled, note that Anaconda changed its chroot path from `/mnt/sysimage/` to `/mnt/sysroot/` in RHEL 9+. The script currently references `/mnt/sysimage/dev/shm/` which would not exist on RHEL 10.