The network rework project is finally completed, many months later.
- SuperMicro SC846 24-bay 4U chassis
- 1x SuperMicro PWS-920P-SQ power supply, second slot empty for now
- SuperMicro X9DRi-F motherboard
- 4x 16GB DDR3 memory
- 2x Intel Xeon E5-2630L v2 CPUs (6 cores/12 threads)
- 2x Noctua NH-D9DX i4 3U CPU coolers
- 2x Noctua NF A8 PWM 80mm fans to replace exhaust fans
- 3x Noctua NF-F12 iPPC 3000 PWM 120mm fans to replace fan wall
- Solarflare SFN7122F dual-port SFP+ NIC, connected at 10Gbit via DAC cable
- LSI 9211-4i SAS2 HBA
- 2x Kingston A400 128GB SSDs as a mirrored boot/system volume, useless for anything else due to terrible performance. Turns out they're cheap for a reason!
- 2x Samsung PM883 480GB SSDs as mirrored container/VM/homedir/scratch storage.
- 6x Western Digital Easystore 12TB external drives, removed from their cases (“shucked”) and configured as a 42.2TiB ZFS RAIDZ2 pool.
- 1x Seagate Ironwolf 10TB + 1x Western Digital Red 8TB HDDs in a janky 8TB mirror, currently unused and may just remove the drives entirely.
- Router: HP T730 thin client + Intel I350-T4V2 4-port 1GbE NIC
- Switch: Ruckus ICX 7150-C12P - 14x 1GbE (12 with PoE/PoE+), 2x 1/10Gb SFP+
Yup, it's some shelving on wheels. The chunky black case is the “old” server, currently turned off while I decide what on earth to do with all of the leftover hardware/components 😓 Stitch is the official server mascot.
Installed pfSense and spent a LOT of time breaking the network, annoying my wife no end. Things are now in a working state, honest.
- All DNS requests are forced to go through the router, and the router uses DNS over HTTPS to avoid various ISP shenanigans
- pfBlockerNG blocks ads network-wide without having to install an ad blocker on every device. This breaks some garbage like the Amazon Android app, I'll eventually get around to whitelisting their ads.
- A permanent OpenVPN connection with a VLAN that can only exit the network via it. Not entirely sure what I want to use this for.
I ran into a lot of issues trying to use pfSense's DHCP/DNS with multiple subnets, the devs seem to be vehemently opposed to letting you do this for some reason.
After going through the usual “finding a USB stick that still works” adventure, I installed FreeNAS and started trying to get a feel for it. It seems nice if you just want a NAS, or a NAS with a few extra bits, but I ultimately decided that it would make more sense to use the new monster server for everything. Compatibility is an issue (I write a lot of .NET Core lately), and FreeBSD's bhyve hypervisor is annoying at best.
Next adventure: Proxmox VE. ZFS/NFS is managed directly on the server, and everything else runs in a large pile of lightweight Linux containers and occasional heavier VMs. The only real issue so far has been how easy it is to completely break the Proxmox web UI while trying to get container UID/GID mapping working.
It currently runs 9 containers and 2 VMs, with a lot more planned as I get deeper into self-hosting things. The dnsmasq (multi-subnet DHCP/DNS) and Samba (Windows file sharing) containers are the only important ones, but hey, learning!
Having never used enterprise networking gear before, this ServeTheHome thread has been a MASSIVE help. I now have VLANs, broadcast forwarding (DHCP!), and some very basic ACLs! VLANs:
SECURE: desktops (wired), laptops (wireless via secondary SSID). Can access everything in
INSECURE: TVs/consoles/a whole lot of random garbage, mix of wired and wireless. Very limited access to
VPN: one toy container to make sure that it really does only work via the router's VPN.
SERVER: servers galore.
At some point I'll get around to having a separate VLAN for services (an internal DMZ?) to simplify ACLs, they're a bit horrifying right now.
There's always more things to do, but learning new things is at least half of the fun for me!
Yeah, these need to happen. Super looking forward to the initial process of syncing the important stuff (cat/dog pictures and videos) to THE CLOUD with Comcast's miserly upstream.
Everyone likes pretty graphs, right? I'd like to be able to monitor a variety of metrics, and it looks like Grafana (pretty dashboards) + InfluxDB (time series database) + Telegraf (metric collector) is the way to go.
- Temperatures, so many temperatures - drives/CPUs/etc across multiple machines
- CPU usage (Proxmox, all containers, all VMs)
- RAM usage (Proxmox, all containers, all VMs)
- Disk space (per pool? specific filesystems?)
- UPS power output/battery charge level/battery time remaining
- Server LAN bandwidth
- Internet bandwidth (from router)
- Latency to… somewhere?
- Switch stats? I think it has SNMP 🤔
- Wifi stats?
A large chunk of my work (and all of my weird side projects) use Docker/Docker Compose, but Docker for Windows is pretty bad. Probably makes sense to set up a Docker VM each for home/work. Will need to test to see how awful this will be via NFS.
Centralized user accounts
LDAP? FreeIPA? That other thing with a weird name? I'd like to not have to create user accounts over and over again.
Automate container/VM software updates
This is probably fairly easy to do, but it might as well make the list.
Probably need a proper DMZ setup for externally accessible services, even if that's just an SSH/VPN bastion host.
SERVER VLAN into
MANAGEMENT might make sense too.