The Great Network Rework

Technology
The Great Network Rework
Page content

My home network is a bit of a mess, this post documents my plans for a grand rework that will absolutely go exactly according to plan with no problems whatsoever 😅

Current setup

  • Netgear CM1000 cable modem for our Comcast “gigabit” internet
  • Netgear R7800 Nighthawk X4S router/WAP running OpenWRT
  • 8-port 1Gb switch for server and TV devices (TV, consoles, Raspberry Pi), connected to router
  • 5-port 1Gb switch for computers (my desktop, wife's desktop, printer), connected to router with a too-short 50ft cable
  • Server: ASRock E3V5 WS, Intel Xeon E3 1230 V5, 32GB DDR4, 500GB SSD, 2x 4TB HDD (mirror), 10TB HDD.

The existing server has to fill a lot of roles - NAS, development, home VMs, work VMs, and backup storage. All of the drives are full, there are no mirrors/backups for lots of data, and I find FreeBSD's hypervisor (bhyve) annoyingly wonky and difficult to use.

Plans!

This will take a fair while to finish due to 💸

Separate NAS server

Breaking network file storage while messing around with VMs is not exactly a family-friendly feature, it's time to split the NAS functionality out to a new host.

  • Buy a used enterprise rackmount machine. Currently leaning towards a system built around a SuperMicro SC846 24-bay 4U chassis.
  • Add 2 small (60GB?) SSDs as a mirrored boot volume, I definitely don't trust the FreeNAS-recommended $10 USB stick option.
  • Install FreeNAS, a FreeBSD-based NAS operating system. I'm tired of messing with the command line and would like things to mostly just work.
  • Add a second-hand 2 port SFP+ network card for 10Gb, connected to VM host and switch.
  • Move some (all?) of the existing drives over.

Storage

  • Purchase 6x 8 or 10TB drives, possibly during Black Friday/Cyber Monday sales. Shucking external drives (removing them from their external USB cases) is usually cheapest per TB.
  • Set up an exciting RAID-Z2 array (survives 2 disks failing), roughly 32TB usable space with 8TB disks or 40TB with 10TB disks. This is fairly easy to expand later by adding more 6 disk arrays due to ZFS magic.

Re-purpose existing server

The hardware is only a few years old and still perfectly fine. The motherboard doesn't support CPUs with more than 4 cores/8 threads, if that becomes a bottleneck in the future I can probably pick up a used SuperMicro motherboard + dual E5 Xeon CPUs from Ebay relatively cheaply.

  • Buy some sort of 4U chassis to re-home the existing server. A smaller case would also work since it doesn't have many drives, but cooling could be interesting. ATX power supply would be ideal!
    • Chenbro/iStarUSA/Norco are cheap and have middling reviews at best.
    • Rosewill (Newegg's brand I believe) are cheap, of wildly variable quality, and a huge pain to mount.
    • SuperMicro are high quality but also expensive, would also require a new PSU.
  • Add 2 small (60GB?) SSDs as a mirrored boot volume.
  • Install Proxmox VE, a Linux-based virtualization platform for containers and virtual machines.
  • Add a second-hand 1-2 port SFP+ network card for 10Gb, connected to NAS.
  • Use the NAS for VM storage. iSCSI volumes? NFS?

Network

  • Buy a rack-mount core switch! VLANs, 10Gb fiber, and various other fun things. Old enterprise gear can usually be found for cheap on Ebay, but they also tend to be SUPER LOUD. Some options:
  • Replace the current 50ft Cat6 cable from old router -> 5-port switch with a pair of 100ft cables so that it can actually reach the right part of the house - one for modem -> router, the other for core switch -> TV switch.
  • Buy some sort of useful router device to run pfSense on. Many options here: APU2, fitlet2, etc.
  • Move the existing router/WAP from the far end of the house to the new location, being closer to the middle should help with our signal woes.

Rack

Deciding on a rack to buy is proving quite difficult. An open 4 post rack is cheaper ($150+), but noise is possibly a problem. An enclosed cabinet rack is much more expensive ($400+), but dampens noise and conveniently has a top panel to sit things on.

  • Raising Electronics make 15U 4-post 31” deep racks ($136 pick-up but out of stock), unfortunately they don't seem to have a great reputation for quality.
  • StarTech make 12U 4-post adjustable depth racks ($203 shipped) that get much better reviews.
  • Various companies make good cabinets but $$$.
  • Picking up something second-hand on Craigslist/Ebay might work, but don't really have anywhere to put a huge full-height rack.