pxe_boot_alma/homelab/docs/alma10-kickstart-changes.md

AlmaLinux 10 Kickstart — Changes & Known Issues

Created: 2026-03-27 Scope: Production kickstart build — work/KickStart/cfg/AlmaLinux-10.1-x86_64/ Environment: Monolith at 10.1.1.1 provides dnsmasq (DHCP+TFTP), Squid proxy (:3128), and HTTP for kickstart and repo. Probes/servers have no other network access during install.

---

Architecture

• KickStart delivered via CGI: http://10.1.1.1/KickStart/cfg/ks.cgi?DIST=AlmaLinux-10.1-x86_64&DISKTYPE=ssd&TYPE=probe|server
• Install tree: DVD ISO (not minimal/lite) — required for @Server, @Development Tools, and other groups
• tftpboot root: /home/kickstart/tftpboot on monolith
• Post-install scripts: modular ks.d/ (N/P/S series), executed in sort order

---

Install Source Change: Minimal → DVD

Switched from AlmaLinux-10.1-x86_64-minimal.iso to AlmaLinux-10.1-x86_64-dvd.iso.

Why: The minimal ISO does not include package groups (@Server, @Development Tools, @Security Tools, etc.) or many individual packages listed in KS_Packages.cfg. Using it caused anaconda to abort with missing package errors and also caused GDM/initial-setup failures on the minimal environment.

---

KS_Packages.cfg — Removed Packages

The following packages were removed from KS_Packages.cfg with justification:

Package	Reason
perl-core	Meta-package removed in RHEL 10; individual perl-* packages are preferred
dracut-fips	Hard-removed in RHEL 10 — FIPS is boot-time only (fips=1 kernel param). This project does not use FIPS.
dracut-fips-aesni	Same as above
libXScrnSaver	Merged into libX11 in RHEL 10; no longer a separate package
usbguard	usbguard-selinux scriptlet crashes anaconda's RPM transaction during kickstart. Moved to S930A-security (installed via dnf in %post).
perl-NetAddr-IP	EPEL-only on RHEL 10; not available from DVD. Moved to S105A-extra-pkgs EPELPKGS.
perl-Net-DNS	EPEL-only on RHEL 10; not available from DVD. Moved to S105A-extra-pkgs EPELPKGS.

lshw — Confirmed Present in DVD

lshw is available in the AlmaLinux 10.1 DVD AppStream repo. An earlier filename-based check produced a false miss because the package version string starts with a letter (lshw-B.02.20...), defeating a naive sed 's/-[0-9].*//' version strip. Confirmed by test VM install — lshw installs cleanly.

---

S850A-network — dhclient and ifcfg Removed in RHEL 10

Changes made:

1. Removed the dhclient block entirely — dhclient package is removed in RHEL 10. NetworkManager uses its built-in DHCP client (dhcp=internal).
2. Removed the cd /etc/sysconfig/network-scripts block — ifcfg file format is removed in RHEL 10. NetworkManager uses keyfiles at /etc/NetworkManager/system-connections/.
3. Replaced ifcfg DNS overrides with a Python configparser approach that modifies the NM keyfile directly:

import configparser, os, sys, glob
condir, first = sys.argv[1], sys.argv[2]
candidates = (
    glob.glob(os.path.join(condir, first + '.nmconnection')) +
    glob.glob(os.path.join(condir, '*.nmconnection'))
)
for path in candidates:
    cfg = configparser.RawConfigParser()
    cfg.optionxform = str
    cfg.read(path)
    if not cfg.has_section('ipv4'):
        continue
    if cfg.get('ipv4', 'method', fallback='') not in ('auto', 'dhcp'):
        continue
    cfg.set('ipv4', 'dns', '192.168.194.102;192.168.226.102;')
    cfg.set('ipv4', 'dns-search', 'iglass.net;')
    cfg.set('ipv4', 'dns-priority', '-100')
    with open(path, 'w') as fh:
        cfg.write(fh)
    os.chmod(path, 0o600)
    break

---

S200A-systemctl — GDM / initial-setup Guards

Added || true and file existence checks around service disabling and GDM config modification, because the DVD minimal environment may or may not install GDM or initial-setup. Without these guards, the script errored out and halted kickstart.

for _svc in initial-setup-graphical.service initial-setup-reconfiguration.service \
            initial-setup-text.service initial-setup.service; do
   systemctl disable "$_svc" 2>/dev/null && echo "Disabled $_svc" || true
done
if [ -f /etc/gdm/custom.conf ]; then
   perl -spi -e 's/\[daemon\]/\[daemon\]\nInitialSetupEnable=False/' /etc/gdm/custom.conf
fi

---

S930A-security — usbguard Moved to Post-Install

Problem: usbguard-selinux contains a scriptlet that crashes the anaconda RPM transaction with:

Error in <unknown> scriptlet in rpm package usbguard-selinux

This does not happen in a running system — only during anaconda's install phase.

Fix: Removed usbguard from KS_Packages.cfg. S930A-security now installs it via dnf after boot:

dnf install -y usbguard 2>&1
if [ -e /bin/usbguard ]; then
   /bin/usbguard generate-policy > /etc/usbguard/rules.conf
   perl -spi -e 's/ImplicitPolicyTarget=\S+/ImplicitPolicyTarget=reject/' \
       /etc/usbguard/usbguard-daemon.conf
   systemctl disable usbguard.service
fi

---

S940A-cis-hardening — New Script

Added CIS Level 1 hardening script. See alma10-cis-hardening.md for full details on what was applied, deferred, and excluded.

---

S105A-extra-pkgs — Known Alma 10 Compatibility Issues

This script has several items that need Alma 10 review before production use:

Item	Issue	Action Needed
dnf config-manager --set-enabled powertools	powertools repo does not exist in RHEL 10 — superseded by crb. Removed (2026-03-27). CRB is already enabled by the lines that follow.	Done
MS repo URL (rhel/9.0/prod.repo)	Updated to rhel/10/prod.repo (2026-03-27). Package renamed: mssql-tools → mssql-tools18. msodbcsql18 name unchanged. Confirmed present at https://packages.microsoft.com/rhel/10/prod/.	Done
dnf -y module install nodejs:18/common	DNF AppStream modules removed in RHEL 10; nodejs is a regular package. Replaced (2026-03-27) with dnf -y install nodejs. RHEL 10 AppStream ships nodejs 22.	Done
PowerShell powershell-7.3.4-1.rh.x86_64.rpm	RHEL 9 build retained; WS-Man support required. No RHEL 10 build available as of 2026-03-27. TODO comment added.	Verify RHEL 10 RPM when available at https://github.com/PowerShell/PowerShell/releases
@Xfce in EPELPKGS	Alma 10 snapshot stage 3 failed on 2026-03-29 with Module or Group 'Xfce' is not available. It is now optional in the Alma 10 script and skipped when unavailable.	Re-test stage 3
perl-NetAddr-IP, perl-Net-DNS	Added to EPELPKGS (2026-03-27) — EPEL-only, removed from KS_Packages.cfg.	Done

---

Server RPM Dependency Coverage — Pending

Only the probe RPM was available for dependency analysis (2026-03-27). A server RPM exists but was not available in this repo at time of review. The following packages were moved from PERLPKGS (dnf) to cpanm because RHEL 10 has no RPM for them — they are retained in cpanm to cover potential server usage until a server RPM can be checked:

• Crypt::SSLeay — removed from RHEL 10 (superseded by Net::SSLeay); not used by probe but retained for server
• HTML::Restrict — no EPEL 10 package; used by probe and possibly server
• Cache::FastMmap — no EPEL 10 package; used by probe and possibly server

Action: When a server RPM is available, run the same rpm -ql | xargs grep '^use ' analysis and cross-reference against the cpanm and dnf lists.

---

EPEL Dependency Chain

The S105A-extra-pkgs script enables EPEL via dnf -y install epel-release before installing EPELPKGS. The following packages are EPEL-only and must remain in that section rather than KS_Packages.cfg:

• perl-NetAddr-IP
• perl-Net-DNS

These were confirmed EPEL-only by test VM install (2026-03-27): present in EPEL 10, not in DVD BaseOS or AppStream.

---

net-snmp-perl Perl Dependencies

net-snmp-perl does not directly require perl-NetAddr-IP or perl-Net-DNS. They are likely pulled in by local monitoring Perl scripts (in the iglass application stack), not by RPM-level dependencies of net-snmp-perl. They must be installed; remove them only after confirming no monitoring scripts reference them.

---

authselect / pam_faillock Post-IPA Note

faillock.conf is pre-configured by S940A-cis-hardening, but pam_faillock is not active until authselect enables it. This must run after IPA enrolment:

authselect enable-feature with-faillock

Do not run this in kickstart — ipa-client-install owns the authselect profile and will reset it.

---

AIDE

AIDE is installed via KS_Packages.cfg but cron-based scanning is managed centrally (not locally). No local AIDE cron is configured in kickstart. A central machine runs AIDE against these hosts.

---

Symlinks Restored (Windows Transit)

15 V/P variant scripts that are symlinks to their A counterparts were zeroed out during a Linux→Windows→Linux filesystem round-trip. All restored (2026-03-27/28):

Symlink	Target
S105V-extra-pkgs	S105A-extra-pkgs
S150V-etc-mod	S150A-etc-mod
S200V-systemctl	S200A-systemctl
S350V-chrony	S350A-chrony
S570V-Postgres	S570A-Postgres
S650V-Crontabs	S650A-Crontabs
S701P-chroot-maps	S701A-chroot-maps
S701V-chroot-maps	S701A-chroot-maps
S800V-netsnmp	S800A-netsnmp
S856V-unbound	S856A-unbound
S870V-sshd	S870A-sshd
S875V-firewall	S875A-firewall
S960V-selinux	S960A-selinux
S980V-monitor	S980A-monitor
S990V-mailme	S990A-mailme
S104V-confluent	S104P-confluent
S400V-noc-iglass	S400P-noc-iglass
S700V-chroot-as400	S700P-chroot-as400
S900P-rmlogrotate	S900S-rmlogrotate
S900V-rmlogrotate	S900S-rmlogrotate
S910V-chmodHosts	S910P-chmodHosts

---

S000A-start — NIC Detection Fix

The original script hardcoded eth0 as the fallback interface name. On RHEL 10 with predictable interface naming (eno1, em1, p1p1, etc.), this fails silently. Fixed to detect the first interface with a MAC address:

ST=$(ip -o link show | grep -v '\blo\b' | perl -ne 'if (/link\/ether\s+(\S+)/) { print "$1\n"; last }')

---

S104P-confluent — Simplified: Standard RHEL 10 Repos Sufficient

Status (2026-03-28): Confluent publishes librdkafka for EL10 at clients/rpm/centos/10/x86_64/, but RHEL 10 ships librdkafka in AppStream and librdkafka-devel in CRB. The Confluent package (librdkafka1) conflicts with the AppStream package (librdkafka).

Resolution: S104P now enables CRB and installs from standard repos — no Confluent clients repo needed:

dnf config-manager --set-enabled crb
dnf install -y librdkafka librdkafka-devel

Verified on alma10-pkgtest (2026-03-28): librdkafka-2.1.1-7.el10 from AppStream, librdkafka-devel-2.1.1-7.el10 from CRB, Net::Kafka builds successfully.

The confluent.repo file (deployed via etc.tgz) still provides the [Confluent] platform repo (rpm/8.2) for potential future use, but [Confluent-Clients] was removed to avoid the package naming conflict.

---

S105A-extra-pkgs — PERLPKGS and cpanm Changes

PERLPKGS (dnf-installed Perl modules)

Packages removed (no RHEL 10 / EPEL 10 RPM available — moved to cpanm):

• perl-Crypt-SSLeay — superseded by Net::SSLeay in RHEL 10
• perl-HTML-Restrict — no EPEL 10 package
• perl-Cache-FastMmap — no EPEL 10 package

Package added:

• perl-IPC-Run — available in RHEL 10 AppStream

EPELPKGS additions

Added to cover probe dependencies not in BaseOS/AppStream:

• perl-NetAddr-IP
• perl-Net-DNS
• perl-Regexp-Common
• perl-WWW-Mechanize
• perl-Protocol-WebSocket
• perl-Crypt-Rijndael
• perl-Net-SSH-Perl

cpanm list changes

Removed (now covered by EPELPKGS above):

• IO::Select (stdlib)
• HTML::Entities (part of HTML::Parser, available as RPM)
• Regexp::Common
• WWW::Mechanize
• Protocol::WebSocket
• Crypt::Rijndael

Added (no RPM available, retained for probe and potential server use):

• HTML::Restrict
• Cache::FastMmap
• Net::Address::IP::Local
• Net::Ping::External
• Net::SIP
• Net::SSH
• Net::Telnet

cpanm Smoke Test Results (2026-03-28, alma10-pkgtest)

22 modules tested via cpanm -n on AlmaLinux 10.1. Two failures:

Module	Result	Action
Crypt::SSLeay	Hard fail — uses SSLv2/SSLv3_client_method() removed in OpenSSL 3.x	Removed from cpanm list. RHEL 10 ships OpenSSL 3.x; this module is incompatible and unmaintained (last release 2016). Use Net::SSLeay (available as perl-Net-SSLeay RPM) instead.
Net::Kafka	Configure fail — needs librdkafka-devel (rdkafka.pc)	Resolved: librdkafka-devel is in CRB, installed by S104P. Net::Kafka now builds cleanly. Conditional guard removed.

All 22 modules install successfully once S104P has run (librdkafka-devel in place).

---

S150A-etc-mod — systemctl --user Guards

systemctl --user commands for masking PulseAudio fail during kickstart post-install because no user session is running. Guarded with 2>/dev/null || true:

systemctl --user mask pulseaudio             2>/dev/null || true
systemctl --user disable pulseaudio.socket   2>/dev/null || true
systemctl --user disable pulseaudio.service  2>/dev/null || true

---

S400P-noc-iglass — init.d → systemd Migration

The .init copy loop was replaced with systemd unit file installation (2026-03-28). Key findings:

• iFunctions.init is a bash function library, not a service init script — it does not need to go in /etc/init.d/
• The iGlass RPM already ships 24 systemd unit files in /opt/iglass/systemd/system/ (services + timers)
• ctl-iglass.service is the master controller that starts all configured iglass services via properties.pl

New block installs all .service and .timer files to /etc/systemd/system/ and runs daemon-reload. Services are not enabled during kickstart — ctl-iglass.service requires probe configuration (customer/hostname) which is set post-install.

---

S570A-Postgres — pgdg Repo and useradd Fix

Three changes:

1. pgdg repo URL: EL-9-x86_64 → EL-10-x86_64
2. Module disable: dnf -y module disable postgresql commented out with TODO — DNF AppStream modules are removed in RHEL 10, this command would fail
3. useradd flag: -n (no user private group) changed to -N — the flag was renamed in RHEL 10

---

S700P-chroot-as400 / S701A-chroot-maps — AlmaLinux 10 Version Check

Both scripts checked ^VERSION="9" in /etc/os-release. Updated to accept RHEL 9 or 10:

if ! grep -qE '^VERSION="(9|10)' /etc/os-release; then
   echo "This script is for AlmaLinux 9 or 10"

Note: mkjail.pl and the full chroot setup have not been tested on RHEL 10. Test on first physical build.

---

S750A-serialLogin — /etc/securetty Guard

/etc/securetty was removed in RHEL 9.2+ and does not exist in RHEL 10. Guarded the append:

if [ -f /etc/securetty ] && ! grep -q "$TTYS" /etc/securetty; then
   echo "$TTYS" >> /etc/securetty
fi

Pending: Verify that rhgb quiet is present in GRUB_CMDLINE_LINUX on RHEL 10 — the regex that removes it may not match the default RHEL 10 grub config. Check on first physical probe build.

---

S900S-rmlogrotate — Timer-Based logrotate (RHEL 10)

In RHEL 10, logrotate runs via logrotate.timer (systemd timer), not just a cron job. Both are now disabled:

# Disable cron-based logrotate (RHEL 9 and earlier)
if [ -f /etc/cron.daily/logrotate ]; then
   sed -i '2s/^/exit\n/' /etc/cron.daily/logrotate
fi

# Disable timer-based logrotate (RHEL 10+)
systemctl disable --now logrotate.timer 2>/dev/null || true
rm -f /etc/systemd/system/timers.target.wants/logrotate.timer

S900P and S900V are symlinks to S900S (not separate scripts).

---

S920V-vm-cleanup — dhclient Block Removed

The script previously migrated /etc/dhcp/dhclient-em1.conf from the installer environment into the installed system. dhclient is removed in RHEL 10 — NetworkManager uses its built-in DHCP client. The block was removed:

# dhclient removed in RHEL 10 — dhclient.conf migration no longer needed

The interactive read -p 'Customer:' / read -p 'Hostname/Probe:' prompts are intentional — this script is run interactively during VM provisioning, not as part of the automated kickstart sequence.

---

S980A-monitor — SSSD enumerate Removed

SSSD enumerate = true is not supported for AD or IdM providers in RHEL 10 (removed from those backends). The option was removed from the sssd.conf block. A comment was left explaining why:

# NOTE: 'enumerate = true' was removed in RHEL 10 for AD/IdM SSSD providers.

---

S990A-mailme — NIC Detection and SSH Protocol Fix

Two changes:

1. NIC detection: Replaced hardcoded eth0 with a fallback chain covering predictable interface names, then ip link as a last resort:

ETH=
for _candidate in eth0 em1 p1p1 eno1; do
   if [ -e "/sys/class/net/$_candidate" ]; then
      ETH="$_candidate"
      break
   fi
done
if [ -z "$ETH" ]; then
   ETH=$(ip -o link show | awk -F': ' '!/lo/{print $2; exit}')
fi

2. SSH config: Removed deprecated Protocol 2 directive from the inline SSH config block — Protocol is not a valid sshd_config option in OpenSSH 7.6+ and causes a parse error.

---

S995A-firmware — Firmware Update Deferred

A TODO comment was added. Dell BIN format firmware update support on RHEL 10 needs verification before re-enabling. No functional change to the script.

---

P200A-diskpart — NVMe and VirtIO Disk Support

Problem: The disk enumeration loop only scanned sd* and hd* device names. NVMe (nvme*) and VirtIO (vd*) disks were silently skipped, causing the script to log "Error: No Disks Found!!!" on any system without a SCSI/SATA disk.

Fix 1 — disk enumeration (line 14):

for DSK in $(cd /sys/block; ls -d sd* 2>/dev/null; ls -d hd* 2>/dev/null; ls -d nvme* 2>/dev/null; ls -d vd* 2>/dev/null); do

Fix 2 — NVMe partition naming: NVMe partitions use a p separator (nvme0n1p1) while SCSI/VirtIO do not (sda1, vda1). Added partition prefix variables after disk selection:

DEVAP=""; [[ "$DEVA" == nvme* ]] && DEVAP="p"
DEVBP=""; [[ "$DEVB" == nvme* ]] && DEVBP="p"

All --onpart= references updated to ${DEVA}${DEVAP}$PRT and ${DEVB}${DEVBP}$PRT (8 substitutions across EFI, MBR, and second-disk blocks).

---

no.R010P-1passwd — /mnt/sysimage Path (Disabled)

This script is disabled (no. prefix). If ever re-enabled, note that Anaconda changed its chroot path from /mnt/sysimage/ to /mnt/sysroot/ in RHEL 9+. The script currently references /mnt/sysimage/dev/shm/ which would not exist on RHEL 10.