Network Rework: Complete!

Technology
The Great Network Rework
Page content

The network rework project is finally completed, many months later.

Hardware

Server specs

Storage

Networking

“Rack”

Yup, it's some shelving on wheels. The chunky black case is the “old” server, currently turned off while I decide what on earth to do with all of the leftover hardware/components 😓 Stitch is the official server mascot.

“Rack”

Installation

Router

Installed pfSense and spent a LOT of time breaking the network, annoying my wife no end. Things are now in a working state, honest.

  • All DNS requests are forced to go through the router, and the router uses DNS over HTTPS to avoid various ISP shenanigans
  • pfBlockerNG blocks ads network-wide without having to install an ad blocker on every device. This breaks some garbage like the Amazon Android app, I'll eventually get around to whitelisting their ads.
  • A permanent OpenVPN connection with a VLAN that can only exit the network via it. Not entirely sure what I want to use this for.

I ran into a lot of issues trying to use pfSense's DHCP/DNS with multiple subnets, the devs seem to be vehemently opposed to letting you do this for some reason.

Server

After going through the usual “finding a USB stick that still works” adventure, I installed FreeNAS and started trying to get a feel for it. It seems nice if you just want a NAS, or a NAS with a few extra bits, but I ultimately decided that it would make more sense to use the new monster server for everything. Compatibility is an issue (I write a lot of .NET Core lately), and FreeBSD's bhyve hypervisor is annoying at best.

Next adventure: Proxmox VE. ZFS/NFS is managed directly on the server, and everything else runs in a large pile of lightweight Linux containers and occasional heavier VMs. The only real issue so far has been how easy it is to completely break the Proxmox web UI while trying to get container UID/GID mapping working.

It currently runs 9 containers and 2 VMs, with a lot more planned as I get deeper into self-hosting things. The dnsmasq (multi-subnet DHCP/DNS) and Samba (Windows file sharing) containers are the only important ones, but hey, learning!

Switch

Having never used enterprise networking gear before, this ServeTheHome thread has been a MASSIVE help. I now have VLANs, broadcast forwarding (DHCP!), and some very basic ACLs! VLANs:

  • 10 SECURE: desktops (wired), laptops (wireless via secondary SSID). Can access everything in SERVER.
  • 20 INSECURE: TVs/consoles/a whole lot of random garbage, mix of wired and wireless. Very limited access to SERVER services.
  • 30 VPN: one toy container to make sure that it really does only work via the router's VPN.
  • 40 SERVER: servers galore.

At some point I'll get around to having a separate VLAN for services (an internal DMZ?) to simplify ACLs, they're a bit horrifying right now.

Future tasks

There's always more things to do, but learning new things is at least half of the fun for me!

Backups

Yeah, these need to happen. Super looking forward to the initial process of syncing the important stuff (cat/dog pictures and videos) to THE CLOUD with Comcast's miserly upstream.

Monitoring

Everyone likes pretty graphs, right? I'd like to be able to monitor a variety of metrics, and it looks like Grafana (pretty dashboards) + InfluxDB (time series database) + Telegraf (metric collector) is the way to go.

  • Temperatures, so many temperatures - drives/CPUs/etc across multiple machines
  • CPU usage (Proxmox, all containers, all VMs)
  • RAM usage (Proxmox, all containers, all VMs)
  • Disk space (per pool? specific filesystems?)
  • UPS power output/battery charge level/battery time remaining
  • Server LAN bandwidth
  • Internet bandwidth (from router)
  • Latency to… somewhere?
  • Switch stats? I think it has SNMP 🤔
  • Wifi stats?

Docker VMs

A large chunk of my work (and all of my weird side projects) use Docker/Docker Compose, but Docker for Windows is pretty bad. Probably makes sense to set up a Docker VM each for home/work. Will need to test to see how awful this will be via NFS.

Centralized user accounts

LDAP? FreeIPA? That other thing with a weird name? I'd like to not have to create user accounts over and over again.

Automate container/VM software updates

This is probably fairly easy to do, but it might as well make the list.

Network hardening

Probably need a proper DMZ setup for externally accessible services, even if that's just an SSH/VPN bastion host.

Separating the SERVER VLAN into SERVICES and MANAGEMENT might make sense too.