The RAT ( Redundant Array of Transistors) cluster is just a bunch of old computers I have lying around. There used to be more but long since given away. This is a leaning tower of 4 along with my current workstation and my laptop, the correct rectangle. To stay fresh I have mangled them into 'an' environment partly based on what I have grown up with. No doubt I will add more to this collection but for now some of the basics of building the environment.
The nodes have been largely on their own for a long time, with whatever ISO I've had to hand. The PI was running PXE whose manual setup is written - here - and was fine. However the nodes don't have LDAP, and point to the home router's DNS and DHCP over WIFI. Horrid.
Now they all exist on a 1Gbit switch. For now only 4 nodes is fine, and this is then plugged into the workstation as a gateway to the rest of the network.
What we end up with is Cheese2 which bridges this network and provides this IP forwarding, DNS and DHCP for the RAT cluster.
Cheese2 is the replacement for Cheese1 which was built over cheese and wine, is like its predecessor for playing games. When managing computers by day some form of config management is used and I still pine for LCFG and Machination. Specifically for its core concepts of declarative configuration and reconciling that state by modelling the current and then the desired states. Thus instead of spending time dual booting for games between windows and some RHEL thing, or trying WSL, its time to experiment with NixOS as one of the closest tools I can get to declarative configuration.
All in all using NixOS for games and other work on the GPU has been stable to warrant no more Windows in the house. Nix is flexible enough that we can add configuration specifically for the RAT cluster, disabling it if needed.
We can take this motley crew of computers and place a simple 1Gib Ethernet switch between them all and add Cheese2 as the router between the clusters network and the rest. This is an unmanaged PoE switch, if it does anything more clever than move packets it can be ignored for now. Cheese2 needs to have its interface setup correctly.
networking.interfaces."eno1" = {
useDHCP = false;
ipv4.addresses = [{
address = "10.0.0.254";
prefixLength = 24;
}];
};
Cheese2 has two interfaces, the WiFi on wlp8s0, and eno1 as its Ethernet. For this we can give it a static address and give it an ip. For addressing, we will give it a /24, and use 10.0.0.0/24 as its easy to type and for this work no other thing is using this address space for now. This can give us plenty of planing for overlay networks, kubernetes internal IPs, DHCP pools, load balancing and anything else we can think of. The rest of the homes network is split up into vlans each a /24 192.168.0.0 address. In future its be interesting to use v6 unique local addresses.
Next to forward packets from this network though cheese and out onto the rest of the network and out to the internet we set iptables.
networking.firewall.extraCommands = ''
# Set up SNAT on packets going from downstream to the wider internet
iptables -t nat -A POSTROUTING -o wlp8s0 -j MASQUERADE
iptables -A FORWARD -i eno1 -o wlp8s0 -s 10.0.0.0/8 -j ACCEPT
iptables -A FORWARD -i wlp8s0 -o eno1 -m state --state RELATED,ESTABLISHED -j ACCEPT
'';
Yes, nftables may be preferred by some, however the above is what I use and it will tell iptables to NAT packets and forward destinations from our cluster out, and any session traffic from outside back.
Now, giving a node an IP and sending a ping to 1.1.1.1 will give us a response. But we aren't going to manually set IPs for the network; we can do this with DHCP.
☞ N.B. one DHCP to rule them, one DHCP to bind them. If your computer has dnsmasq or some other daemon running and you want to configure it this way. Have a strong think. Even having dnsmasq by default off, libvirt for example will use it as a default for its VM network, and your DHCP server will not run.
DHCP must first be set to run after Cheese2 has started its network:
systemd.services.kea-dhcp4-server = {
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
};
Then the service gets configured.
services.kea.dhcp4 = {
enable = true;
settings = {
interfaces-config.interfaces = ["eno1"];
# this is a home network, a CSV file is enough to track DHCP leases
lease-database = {
name = "/var/lib/kea/dhcp4.leases";
persist = true;
type = "memfile";
};
.....
First we enable and set it to listen on our ethernet interface only. A CSV file is fine for this, however I understand you can point Kea at a Database for larger/complex networks.
....
renew-timer = 3600;
rebind-timer = 3600 * 2;
valid-lifetime = 3600 * 4;
....
This network will have statically assigned addresses explained later, and a small pool of addresses. The lease time can be long, the network is not changing for now and this can always be reduced later.
subnet4 = [
{
id = 1;
pools = [{pool = "10.0.0.1 - 10.0.0.10";}];
subnet = "10.0.0.0/24";
reservations = [
{
hw-address = "2c:cf:67:7c:e0:33";
ip-address = "10.0.0.250";
hostname = "head";
}
....
];
This tells DHCP about a subnet and how it should be configured for the network starting with the pool. This will be for any node that is plugged in and asks for an IP. The rest are reservations. This lets me register a node and hand out the same IP each time.
option-data = [
{
name = "routers";
data = "10.0.0.254";
}
{
name = "domain-name-servers";
data = "10.0.0.254";
}
{
name = "domain-name";
data = "rat.cluster";
}
{
name = "domain-search";
data = "rat.cluster";
}
];
This gives out all the extra information a DHCP server can provide alongside a request for an IP. This tells nodes where their router is, the DNS server and name information. Further we can tell it if there is a PXE server even custom options.
Without getting too excited this now gives the cluster DHCP and the next thing to implement is DNS.
Nodes can set /etc/resolve.conf to some external DNS or even the main house's DNS resolver, and so we can add DNS alongside our new DHCP. However this configuration will let Unbound become the main resolver for all of Cheese2's DNS traffic.
services.resolved.enable = false;
services.unbound = {
enable = true;
...
We tell systemd to no longer manage names, and give ourselves over to unbound.
resolveLocalQueries = true;
package = pkgs.unbound-full;
settings = {
remote-control.control-enable = true;
server = {
module-config = ''"iterator"''; #turn off dnssec
interface = [
"10.0.0.254"
"127.0.0.1"
];
access-control = [
"0.0.0.0/0 refuse"
"127.0.0.0/8 allow"
"10.0.0.0/8 allow"
];
private-domain = [ "rat.cluster" ];
local-zone = [
''"rat.cluster." static''
''"0.0.10.in-addr.arpa." static''
''"168.192.in-addr.arpa." transparent''
];
local-data = [
''"head.rat.cluster. A 10.0.0.250"''
''"n1-sw1.cluster. A 10.0.0.253"''
''"body.rat.cluster. A 10.0.0.29"''
];
local-data-ptr = [
''"10.0.0.250 head.rat.cluster"''
''"10.0.0.253 n1-sw1.rat.cluster."''
''"10.0.0.29 body.rat.cluster."''
];
};
forward-zone = [
{
name = ".";
forward-addr = [ "192.168.1.1" ];
}
];
};
};
We do turn off DNSSEC as we haven't set up any keys or certs. If you want to practise zero-trust networking, we can leave that for self-study. We only listen on eno1 our cluster network and any local asks, and do not allow any asks from networks other than that. The zone shall be rat.cluster and in it we define the names of our nodes and lastly make sure to forward anything outside of this zone to the home router, which in turn will forward out to the internet.
☞ N.B. Remember to tell the nodes firewall to allow traffic for these services. Assuming you have this turned on.
☞☞ N.N.B. The current names are different but the spirit of the machine is the same.
With nodes set to resolve this we can now finish this off with a bit of LDAP.
In this network there sits a dusty box. The box is a tepid Synology node rusting away with spare disks thrown in. Its network of 4GbE links is LACPed together and it makes noises like a diesel train. One day it will get upgraded but for now, it can host NFS shares and users. This makes our cluster not only have a flexible setup for users and groups, and no longer rely on local users. It also gives us coherent user IDs and group IDs for remote filesystems. Something large places still seem to struggle on. Setting up LDAP and NFS is for when I rebuild this NAS, however joining it can be done by doing in Nix:
users.ldap = {
enable = true;
base = "dc=diesel,dc=home";
server = "ldap://diesel.home";
useTLS = false;
loginPam = true;
nsswitch = true;
bind = {
policy = "soft";
};
daemon = {
enable = true;
extraConfig = ''
scope sub
filter passwd (objectclass=posixAccount)
base passwd cn=users,dc=diesel,dc=home
base group cn=groups,dc=diesel,dc=home
map passwd loginShell "''${loginShell:-/run/current-system/sw/bin/bash}"
'';
};
};
This will tell Nix to configure ldap for the node, currently TLS is not set and is on the todo list. The last rune in the extra config is a way to tell it to set a persons shell to what is in their map, else use some default. In this case its the Nix path to bash since NixOS has its own path for programs. For the other node being Illumos its ldap is setup using the ldapclient init config OmniOS has a page for how to use this.
With all of these services setup this is now the foundation of a very traditional unix environment. Now to do something with it.
The timeline now diverges: