Start by getting OmniOS installed in a VM. Its default install is ok, however give it as many cores as you can spare and a sizable disk or a fast NFS share.
Create a space for the source and checkout git clone https://github.com/illumos/illumos-gate.git
The build instructions online will define a illumos.sh which contains a list of environment variables for the build system to grab the correct versions of python/perl, other packages and build options. However some of these distro-specific environment variables can be found at their latest version from the distro build script. In the case of using OmniOS its at /opt/onbld/env/omnios-illumos-gate. Other Illumos versions will have their own but most of it should be the same.
It can then be built using:
time ./usr/src/tools/scripts/nightly /opt/onbld/env/omnios-illumos-gate
A mail message with any issues will be sent, and on my computer this took about a few minutes to build on 12 cores of Ryzen 9 7900X. The next task is making this bootable.
In the majority of cases the boot_archive - illumos's ramdisk/ initramfs image - should be filled with the required files for the kernel to use and built as part of the build scripts. It can be created explicitly using:
pfexec bootadm update-archive -v -F cpio -R $PROTO_ROOT
Where $PROTO_ROOT is in the completed build of illumos-gate eg /code/illumos-gate/proto_rooti386, and this will create an archive in illumos-gate/proto/root_i386/platform/i86pc/amd64/boot_archive
This took a few tries to figure out and to make sure that it built a CPIO archive and filled the archive with the correct files.
root@omnios:~# cat /illumos/build.sh
pfexec dd if=/dev/zero of=boot_archive.img bs=1M count=$((2*1024))
pfexec lofiadm -la boot_archive.img
DISK=$(lofiadm boot_archive.img | sed 's/p0//g')
echo "lofi device: $DISK"
pfexec zpool create pool ${DISK}
# Copy in proto
cd /illumos/illumos-gate/proto/root_i386
find . | pfexec cpio -pdum /pool
# Fix driver config with host system
pfexec cp /etc/name_to_major /pool/etc/name_to_major
pfexec cp /etc/driver_aliases /pool/etc/driver_aliases
pfexec cp /etc/minor_perm /pool/etc/minor_perm
pfexec cp /etc/driver_classes /pool/etc/driver_classes
pfexec cp /etc/iu.ap /pool/etc/iu.ap
pfexec dd if=/dev/urandom bs=4 count=1 of=/pool/etc/hostid 2>/dev/null
# Create device nodes
CN=$(grep "^cn " /pool/etc/name_to_major | awk '{print $2}')
MM=$(grep "^mm " /pool/etc/name_to_major | awk '{print $2}')
SY=$(grep "^sy " /pool/etc/name_to_major | awk '{print $2}')
SYSMSG=$(grep "^sysmsg " /pool/etc/name_to_major | awk '{print $2}')
pfexec mknod /pool/dev/console c $CN 0
pfexec mknod /pool/dev/null c $MM 2
pfexec mknod /pool/dev/zero c $MM 12
pfexec mknod /pool/dev/sysmsg c $SYSMSG 0
pfexec mknod /pool/dev/systty c $SY 0
pfexec chmod 0600 /pool/dev/console
pfexec chmod 0666 /pool/dev/null /pool/dev/zero /pool/dev/systty
pfexec chmod 0620 /pool/dev/sysmsg
# Remove stale door files
pfexec rm -f /pool/etc/svc/volatile/*
pfexec rm -f /pool/etc/svc/door
pfexec rm -f /pool/var/run/svc_door 2>/dev/null
# Make sure volatile dir exists and is empty
pfexec mkdir -p /pool/etc/svc/volatile
pfexec mkdir -p /pool/var/run
# Copy the seed database
pfexec cp /lib/svc/seed/global.db /pool/etc/svc/repository.db
pfexec chmod 0600 /pool/etc/svc/repository.db
# Import all manifests into it using the host's svccfg
SVCCFG_REPOSITORY=/pool/etc/svc/repository.db \
pfexec svccfg import /pool/lib/svc/manifest/
ls -la /pool/etc/svc/repository.db
#Configure the console tty
SVCCFG_REPOSITORY=/pool/etc/svc/repository.db pfexec svccfg <<'EOF'
select svc:/system/console-login:default
addpg ttymon application
setprop ttymon/terminal_type = astring: vt100
setprop ttymon/device = astring: /dev/term/a
refresh
exit
EOF
SVCCFG_REPOSITORY=/pool/etc/svc/repository.db \
pfexec svccfg -s svc:/system/console-login:default listprop ttymon
# Create /dev/term/a device node
ASY=$(grep "^asy " /pool/etc/name_to_major | awk '{print $2}')
echo "asy major: $ASY"
pfexec mkdir -p /pool/dev/term
pfexec rm -f /pool/dev/term/a
pfexec mknod /pool/dev/term/a c $ASY 0
pfexec chmod 666 /pool/dev/term/a
# Also create /dev/term/b just in case
pfexec mknod /pool/dev/term/b c $ASY 1
pfexec chmod 666 /pool/dev/term/b
# Check current setting
grep CONSOLE /pool/etc/default/login
# Comment out the CONSOLE restriction to allow root on any terminal
pfexec sed -i 's/^CONSOLE=.*/#&/' /pool/etc/default/login
# Verify
grep CONSOLE /pool/etc/default/login
# Check current root entry, set the password and shell
grep ^root /pool/etc/shadow
pfexec cp /etc/shadow /pool/etc/shadow
pfexec sed -i 's|^root:\(.*\):/usr/bin/bash|root:\1:/sbin/sh|' /pool/etc/passwd
# Verify
grep ^root /pool/etc/shadow
grep ^root /pool/etc/passwd
# Check current pam.conf
grep "pam_unix" /pool/etc/pam.conf | head -5
# Boot config
pfexec mkdir -p /pool/boot/conf.d
printf 'console="ttya,text"\nos_console="ttya"\nttya-mode="115200,8,n,1,-"\n' | pfexec tee /pool/boot/conf.d/serial.conf
# Build boot archive and install bootloader
pfexec bootadm update-archive -v -F cpio -R /pool
pfexec zpool set bootfs=pool pool
RDISK=$(echo $DISK | sed 's/dsk/rdsk/g')
pfexec installboot -mf /illumos/illumos-gate/proto/root_i386/boot/pmbr /illumos/illumos-gate/proto/root_i386/boot/gptzfsboot ${RDISK}s0
pfexec zpool export pool
# Export cleanly
pfexec lofiadm -d /dev/lofi/1
This gets you most of the way there. Next time, I will need to fix the build to use PKG and test instead of doing the artisanal way above of manually moving things from the host env into this build.
However if anyone is doing what's been done here it should let you boot the image in your choice of vm to a prompt.
This was another attempt with a go at trying to use the pkg tools, however it got stuck resolving dependencies and would either pull the wrong components or fail to pull the correct ones into SMF, breaking the node on start. This time it was done fully in illumos using Omnios on hardware.
#Get rid of it if it was attached to debug
pfexec zpool export pool
pfexec lofiadm -d ./illumos.img
# Create the disk image
pfexec dd if=/dev/zero of=./illumos.img bs=1M count=4096
pfexec lofiadm -la ./illumos.img
DISK=$(lofiadm ./illumos.img | sed 's/p0//g')
pfexec zpool create -f pool ${DISK}
# Copy proto otherwise this would be where pkg install would happen
cd /home/ajk203/illumos-gate/proto/root_i386
pfexec find . | pfexec cpio -pdum /pool
cp -rP /lib/libxml* /pool/lib/
# Fix driver config with host system
pfexec cp /etc/name_to_major /pool/etc/name_to_major
pfexec cp /etc/driver_aliases /pool/etc/driver_aliases
pfexec cp /etc/minor_perm /pool/etc/minor_perm
pfexec cp /etc/driver_classes /pool/etc/driver_classes
pfexec cp /etc/iu.ap /pool/etc/iu.ap
pfexec dd if=/dev/urandom bs=4 count=1 of=/pool/etc/hostid 2>/dev/null
# Create device nodes
NTM=/pool/etc/name_to_major
CN=$(awk '/^cn / {print $2}' $NTM)
MM=$(awk '/^mm / {print $2}' $NTM)
SY=$(awk '/^sy / {print $2}' $NTM)
SYSMSG=$(awk '/^sysmsg / {print $2}' $NTM)
ASY=$(awk '/^asy / {print $2}' $NTM)
CRYPTO=$(awk '/^crypto / {print $2}' $NTM)
CRYPTOADM=$(awk '/^cryptoadm / {print $2}' $NTM)
OPENEEPR=$(awk '/^openeepr / {print $2}' $NTM)
pfexec mknod /pool/dev/openprom c $OPENEEPR 0 2>/dev/null || true
pfexec chmod 0644 /pool/dev/openprom
pfexec mknod /pool/dev/cryptoadm c $CRYPTOADM 0 2>/dev/null || true
pfexec chmod 0666 /pool/dev/cryptoadm
pfexec mknod /pool/dev/crypto c $CRYPTO 0 2>/dev/null || true
pfexec chmod 0666 /pool/dev/crypto
pfexec mknod /pool/dev/console c $CN 0
pfexec mknod /pool/dev/null c $MM 2
pfexec mknod /pool/dev/zero c $MM 12
pfexec mknod /pool/dev/sysmsg c $SYSMSG 0
pfexec mknod /pool/dev/systty c $SY 0
pfexec chmod 0600 /pool/dev/console
pfexec chmod 0666 /pool/dev/null /pool/dev/zero /pool/dev/systty
pfexec chmod 0620 /pool/dev/sysmsg
pfexec mkdir -p /pool/dev/term
pfexec mknod /pool/dev/term/a c $ASY 0
pfexec mknod /pool/dev/term/b c $ASY 1
pfexec chmod 666 /pool/dev/term/a /pool/dev/term/b
pfexec mkdir -p /pool/devices/pseudo
pfexec ln -sf ../devices/pseudo/cryptoadm@0:cryptoadm /pool/dev/cryptoadm
pfexec ln -sf ../devices/pseudo/openeepr@0:openprom /pool/dev/openprom
# Remove stale door files
pfexec rm -f /pool/etc/svc/volatile/*
pfexec rm -f /pool/etc/svc/door
pfexec rm -f /pool/var/run/svc_door 2>/dev/null
# Make sure volatile dir exists and is empty
pfexec mkdir -p /pool/etc/svc/volatile
pfexec mkdir -p /pool/var/run
#TODO: fix as my current bhyve vm makes noise for these
rm -f /pool/lib/svc/manifest/network/smtp-sendmail.xml
rm -f /pool/lib/svc/manifest/network/sendmail-client.xml
rm -f /pool/lib/svc/manifest/network/iscsi/iscsi-initiator.xml
rm -f /pool/lib/svc/manifest/network/iscsi/iscsi-target.xml
rm -f /pool/lib/svc/manifest/system/keymap.xml
rm -f /pool/lib/svc/manifest/system/intrd.xml
rm -f /pool/lib/svc/manifest/system/hal.xml
SVCCFG_REPOSITORY=/pool/etc/svc/repository.db svccfg list | grep -E "intrd|keymap|sendmail|hal|ssh"
# SMF seed
pfexec cp /pool/lib/svc/seed/global.db /pool/etc/svc/repository.db
pfexec chmod 0600 /pool/etc/svc/repository.db
SVCCFG_REPOSITORY=/pool/etc/svc/repository.db pfexec svccfg import /pool/lib/svc/manifest/
# Site profile for console
pfexec mkdir -p /pool/etc/svc/profile/site
pfexec cat > /pool/etc/svc/profile/site/console.xml <<'XMLEOF'
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type="profile" name="console">
<service name="system/console-login" version="1" type="service">
<instance name="default" enabled="true">
<property_group name="ttymon" type="application">
<propval name="device" type="astring" value="/dev/console"/>
<propval name="terminal_type" type="astring" value="vt100"/>
</property_group>
</instance>
</service>
</service_bundle>
XMLEOF
pfexec ln -sf generic_limited_net.xml /pool/etc/svc/profile/generic.xml
pfexec ln -sf ns_files.xml /pool/etc/svc/profile/name_service.xml
pfexec ln -sf platform_none.xml /pool/etc/svc/profile/platform.xml
#FMD Config as bhyve was letting it fault for missing a sas expander
pfexec rm -f /pool/usr/lib/fm/fmd/plugins/ses-log-transport.so
pfexec rm -f /pool/usr/lib/fm/fmd/plugins/ses-log-transport.conf
# Add smmsp user/group for sendmail
echo "smmsp:x:25:25:SendMail Message Submission Program:/:" >> /pool/etc/passwd
echo "smmsp:*LK*:::::::" >> /pool/etc/shadow
echo "smmsp::25:" >> /pool/etc/group
# Fix sendmail spool permissions in case anything references it
pfexec mkdir -p /pool/var/spool/clientmqueue
pfexec chown 25:25 /pool/var/spool/clientmqueue
pfexec chmod 0770 /pool/var/spool/clientmqueue
# Serial console
pfexec mkdir -p /pool/boot/conf.d
cat > /pool/boot/conf.d/serial.conf <<'EOF'
console="ttya,text"
os_console="ttya"
ttya-mode="115200,8,n,1,-"
EOF
# Root login
pfexec sed -i 's/^CONSOLE=.*/#&/' /pool/etc/default/login
pfexec cp /etc/shadow /pool/etc/shadow
pfexec sed -i 's|^root:\(.*\):/usr/bin/bash|root:\1:/sbin/sh|' /pool/etc/passwd
echo "illumos" > /pool/etc/nodename
# SSH host keys
pfexec mkdir -p /pool/etc/ssh
pfexec ssh-keygen -t rsa -f /pool/etc/ssh/ssh_host_rsa_key -N '' -q
pfexec ssh-keygen -t ecdsa -f /pool/etc/ssh/ssh_host_ecdsa_key -N '' -q
pfexec ssh-keygen -t ed25519 -f /pool/etc/ssh/ssh_host_ed25519_key -N '' -q
# Reconfigure for /dev
pfexec touch /pool/reconfigure
# Boot archive
pfexec bootadm update-archive -v -F cpio -R /pool
# Boot loader
pfexec zpool set bootfs=pool pool
RDISK=$(echo $DISK | sed 's/dsk/rdsk/g')
pfexec installboot -mf /pool/boot/pmbr /pool/boot/gptzfsboot ${RDISK}s0
cat > /pool/boot/loader.conf <<'EOF'
autoboot_delay="10"
boot_verbose="YES"
EOF
cd -
pfexec zpool export pool
pfexec lofiadm -d ./illumos.img
Attempt 3 is required in future, using the correct image-build tools, and packages for libxml2.