From eb235624a35cdb758c07af6ec405276d82712a81 Mon Sep 17 00:00:00 2001 From: Daniel McKnight Date: Wed, 20 Dec 2023 19:52:09 -0800 Subject: [PATCH] Add existing content --- 2023-11-29_Homelab-Upgrades.md | 89 ++++++++++++++++ 2023-12-01_Current-Software-Stack.md | 114 +++++++++++++++++++++ 2023-12-08_Optimizing-Power-Consumption.md | 65 ++++++++++++ 3 files changed, 268 insertions(+) create mode 100644 2023-11-29_Homelab-Upgrades.md create mode 100644 2023-12-01_Current-Software-Stack.md create mode 100644 2023-12-08_Optimizing-Power-Consumption.md diff --git a/2023-11-29_Homelab-Upgrades.md b/2023-11-29_Homelab-Upgrades.md new file mode 100644 index 0000000..1f73531 --- /dev/null +++ b/2023-11-29_Homelab-Upgrades.md @@ -0,0 +1,89 @@ +--- +date: 2023-11-29 +title: Planning a Homelab Upgrade +tags: + - homelab + - hardware + - networking + - unraid + - proxmox + - pfsense +--- + +I am just starting to plan a major overhaul of my homelab and decided where better to start than by documenting the process. +This is about how my setup has reached its current state and why this setup is starting to feel limiting. + +## A Brief History +The first home server I used was a Sunfire X4150 that I got for something like $50. Windows Home Server 2011 seemed +like a logical choice of OS (this is where I started; I didn't know what I was doing) so I grabbed a copy from my local Fry's (RIP). + +After some adventures in data recovery from a dead drive (read: data loss), I built an [unRaid](https://unraid.net/) box. +One of the main motivations here was that I could add drives over time and increase capacity easily without having to match +drive models or even sizes; this meant I could use all the random drives I had lying around and save some money. + +My current server is basically unchanged since then, with a few upgrades, namely a 3900X and some 8TB HDDs shucked out of +WD external drives when they went on sale. + +## Taking Inventory +The bulk of my setup lives in a 15U rack that I built into the wall of my office. I also have some other networking equipment, +a tower for work, and a personal machine I built into my desk but those aren't relevant to this project. +I'll work through my current setup from the top of the rack down. + +### Networking +- (2U) 48 port patch panel: This is where networking infrastructure like switches and access points, IP cameras, and other + rack equipment terminates +- (1U) 48 port Ubiquiti POE switch: This handles switching for most of the devices terminating at the patch panel +- (1U) 8 port Ubiquiti 10GbE switch: This connects other switches and provides a 10gig connection to my Unraid server +- (2U) pfSense router: This is just an old PC in a 2U case with an Intel Gigabit NIC + +### Compute +- (3U) Unraid server: This is where all of my storage and compute is +- (1U) X4150 server: I spin this up manually every month or so to manually back up data from my Unraid server. Its loud and power + hungry so I leave it shutdown as much as possible + +### Power +- (2U) TrippLite UPS: This provides enough power to run things for about 15 minutes, long enough to safely shutdown and even keep + networking up through a brief outage + +For anyone keeping score at home, that leaves 3U empty which does factor into my plan. + +## Current Shortcomings +Now that I've laid out what I have and how I got there, let's look at some pain points. + +### Power Consumption +Much of my setup has been built from surplus or repurposed systems. The pfSense router in particular is overbuilt for the minimal +compute that routing requires. The backup X4150 server is mostly powered off, but when running for a backup it takes a lot of power +and turns it into a lot of heat and noise in my office. + +### Availability and Maintenance +I have a pfSense VM defined on my Unraid server, so I can at least take the pfSense router down for maintenance without network +downtime but the same cannot be said about the Unraid server. If the Unraid server is down, Plex, HomeAssistant, NextCloud, this blog, +and a lot of other things go down. I also have some issues with my pfSense box resuming after a power outage with is particularly +annoying if I'm not home to turn it back on. + +### Compute +My 12-Core 3900X is generally more than enough compute for me now, but it is occasionally limiting and will not support all of the +services I want to deploy. I also need more than 32GB of RAM, but that's an easy upgrade. + +## The Plan +There are two main upgrades I want to make to address the current shortcomings. + +### pfSense Router Appliance +This one's easy, I just want to get a low-power box to run pfSense since it needs to run 24/7; +I'll likely pick up a used network appliance on eBay for this. + +### Clustered Compute +This still requires a bit more research. I've been becoming more acquanted with Kubernetes as +[part of my day job](https://github.com/NeonGeckoCom/neon-diana-utils) and am settled on setting up a cluster at home. +I will likely use [Proxmox](https://www.proxmox.com/en/) since it checks the boxes for being open source, having a web UI +for management, and being popular enough to find support online. + +I'll use my current pfSense router and Unraid server as nodes, possibly adding another new node with a GPU to play with some LLMs. + +### Storage +I am as of yet undecided on what to do here; I need to do some more reading up on GlusterFS, Ceph, iSCSI, +and probably some technologies I know even less about. + +## What Comes Next +This post is already getting a lot longer than I anticipated, so I'll wrap it up here. My next steps are to (1) find an appropriate +upgrade for my pfSense router and (2) decide on a storage solution. diff --git a/2023-12-01_Current-Software-Stack.md b/2023-12-01_Current-Software-Stack.md new file mode 100644 index 0000000..8e6c5bf --- /dev/null +++ b/2023-12-01_Current-Software-Stack.md @@ -0,0 +1,114 @@ +--- +date: 2023-12-01 +title: My Current Software Stack +tags: + - homelab + - software + - unraid +--- +As part of planning [some hardware upgrades](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/), +I think it is important to evaluate the software side of things to make sure everything is accounted for +in the new design. This is also a good opportunity to evaluate and prune what I have running before +deciding where to move things around. + +## Virtual Machines +I run a few VMs on my Unraid server; many are seldom used but a couple of them I use on a regular basis. +None of these need anything like five-nines availability, but they do need somewhere to run when needed. + +- Cura: I have an Ubuntu VM that is dedicated to running Cura for 3D printing. I use several diferent + computers throughout the day, so being able to slice and print things wherever I am (including from + my phone) is really handy. +- Windows 10: This Windows VM has a Bluray drive attached for working with my media collection or the + odd piece of software that I have on optical media. +- pfSense: This VM is configured with an Intel NIC to be manually spun up before taking my main pfSense + box down for maintenance. This VM isn't used often but it is essential to keeping the network up. + +## Container Services +This is where the bulk of my services exist. Unraid uses Docker with some specialized templating to +provide a nice Web UI for deployment, but I would be happy writing my own [Helm charts](https://helm.sh/) or +[Docker Compose](https://docs.docker.com/compose/) files to deploy things. There are a lot of containers here +so I'll try and organize them into some logical groups. + + +### Core Networking +These services are essential to keeping my network up and supporting other services. Outages here +can bring down other services or parts of my network. + +- DDClient: lets my domain registrar know when my public IP address changes +- LetsEncrypt: reverse-proxy for routing traffic to the appropriate container +- Unifi: controller and web UI for Ubiquiti networking +- Mosquitto: MQTT server used to connect ZWave devices to HomeAssistant + +### Database and Backend +These provide support for other services. An outage here will affect some specific +other services. + +- MariaDB: used by some other containers; I wouldn't mind eliminating or replacing this with Mongodb if I can. + I know practically no SQL and a little Mongodb +- zwavejs2mqtt: connects to zwave devices to HomeAssistant +- Frigate: security camera monitoring, object detection, and recording + +### User-Facing Services +These are the things you probably think of as what a home server is used for. Everything +here has a Web UI or is accessible via client software (or both). + +- hexo-blog: this. +- Vaultwarden: password manager for Bitwarden clients +- Plex: media server; I may move to Emby or Jellyfin as media tagging has been less reliable for me lately + and the recent [watch history debacle](https://www.techhive.com/article/2157803/plex-discover-together-privacy-concerns.html) + is another nudge to find something better. +- Nextcloud: cloud storage, contacts, calendar, client device backups +- Matrix: [matrix chat](https://matrix.org/) home server +- HomeAssistant: smart home controls/automation and camera monitoring +- Heimdall: web dashboard; I just played with this and don't find myself using it often so I'll probably drop it +- Grafana: monitoring dashboard +- GitLab: self-hosted git; I don't use this much but do like having a GitHub alternative available for private + projects (yes, I know you can have private repos on GitHub) + +### Monitoring +These services monitor the other services. If something goes wrong here I'm likely the only one to notice. + +- Influxdb: time-series database used to collect data for Grafana dashboards +- Tautulli: pulls Plex analytics, used to populate Grafana +- Varken: More Plex analytics middle-ware to push analytics to Influxdb +- UniFi-Poller: pulls Unifi analytics into Influxdb for Grafana +- Telegraf: pulls Unraid server analytics into Influxdb for Grafana + +### Mostly Unused +A few containers I have mostly spun-down. Some of these are for testing/evaluation and others I just haven't bothered removing. + +- ActualServer and Firefly-III: personal budgeting apps since [Mint](https://support.creditkarma.com/s/article/Intuit-Mint-and-Credit-Karma) + is going away; neither really does what Mind does for me but I need an alternative since Mint is shutting down at the end of the year and + CreditKarma doesn't appear to be getting budgets which is what Mint did for me. +- Handbrake: video encoder; I will probably run this in a VM if I move containers to k8s since I only run this as an Unraid container to + easily leverage all of my server's cores +- Genea: family tree builder, because that's the kind of data I won't put into a random webapp on the internet +- element-web: a Matrix client; I installed this with Matrix but never use it +- RabbitMQ: MQ server that I setup once in the early days of the [Diana backend](https://github.com/NeonGeckoCom/neon-diana-utils) I made + for work. +- Wireshark: packet sniffing and more network stuff; realistically I would run this on a workstation if I need it + +## Other +Aside from VMs and Containers, there is some basic functionality that needs to be addressed, namely data shares and VPN access. + +### Network Shares +Unraid exposes SMB shares with some basic user access control. Shares on Unraid can utilize separate Cache and Array storage pools; +both pools offer redundancy but the smaller Cache pool is faster than the larger Array pool. I'll go into storage in more detail when +I look at solutions for storage. + +### WireGuard +Unraid includes a WireGuard server which I use to remote into my network when travelling if I need to access the filesystem or a shell +directly. This is particularly helpful if I need to restart a container or find something on a network share that isn't served via Plex +or Nextloud. + +## Putting It All Together +This was all a long way of getting to what I really need in my software stack: +- Virtualization with device passthrough, i.e. a [KVM](https://www.linux-kvm.org/page/Main_Page) Hypervisor +- Container support, likely [k3s](https://k3s.io/) +- VPN access. I might delegate this to pfSense since that would allow for remote access in the event the cluster + is offline so long as the router is on +- Network shares + +I already mentioned [in my previous post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/) that upgrading my pfSense router +and picking a storage solution. I found some old network appliances repurposed to run pfSense [on ebay](https://www.ebay.com/itm/186181814221) +and plan on picking one up for a relatively inexpensive, low-power, and (hopefully) reliable solution. diff --git a/2023-12-08_Optimizing-Power-Consumption.md b/2023-12-08_Optimizing-Power-Consumption.md new file mode 100644 index 0000000..96e0803 --- /dev/null +++ b/2023-12-08_Optimizing-Power-Consumption.md @@ -0,0 +1,65 @@ +--- +date: 2023-12-08 +title: Optimizing Power Consumption +tags: + - homelab + - hardware + - power efficiency +--- +I mentioned in [an earlier post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/) that one of my concerns with my current setup is the +power consumption of everything in my rack. After writing, I realized that I know what the load is on my UPS, but I don't know how each component +draws on its own or what everything actually costs to run. I also haven't looked into tuning things for power consumption for some time, so lets see +if there are any savings to be had there. I picked up a couple of inexpensive ZWave switches that have current monitoring and added them to HomeAssistant +to monitor my Unraid server and pfSense router. I can approximate the network switches' power usage by subtracting those 2 measurements from the UPS' +measured usage. + +## Baseline Idle Power Usage +I connected my Unraid server to one of the monitoring outlets yesterday afternoon, so I have some data including minimal usage overnight. I see that +150-170W was used while streaming some media via Plex and then at bed time usage dropped to 140W with overnight load staying between 130-160W. There was +one peak at 165W, presumably when some scheduled tasks run at 4:30AM. + +Checking the UEFI, I saw the fans all set to "Full Speed". I updated all of them to "Balanced" profiles and that helped with noise in my office which is +a huge plus, but power usage is mostly unchanged. This is by no means a scientific test but does indicate there is little power efficiency to be gained +by messing with fan curves. + +For my pfSense router, things are more consistent with power draw right around 58W with a range of 57-60W. I went into the BIOS and disabled +overclocking and turned fans to "Eco mode" but saw very little change. + +Doing the math, my UPS measures about 250W of usage with about 150W and 60W going to my Unraid server and pfSense router, respectively. This leaves 40W +going to the 2 Ubiquiti switches in the rack. + +## Looking for Power Savings +There are a couple places I think I can save energy, so what can I do to validate that before investing more time and money? For the router, I +know there's not much room to optimize what I have, so I can compare my measured power draw against other hardware options. For the server, I think there +is some more investigation to do: + +### Spinning Rust +Looking at my Unraid array, I see that all 7 spinning disks are active. I am able to manually spin down 4 of them (one has activity which also +keeps the 2 parity disks active). This takes power draw from 154W to 134W; that's a full 20W savings! Guessing that Frigate would be the only +service that won't write to my cache pool, I spun that down and then manually spun down everything in the array again. I'm now down to 111W, +until the disks spin back up. + +This gives me some ideas about planning my storage solution. First, I'll want to try and get all of the services with regular disk IO using SSD-based +storage. Second, if I can keep file metadata in RAM or SSD storage, that will hopefully prevent the disk from needing to be spun up just to open a network +directory with the added benefit of not having to wait for disks to spin up just to look at directory contents. + +### pfSense Router +As mentioned [in my earlier post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/#pfSense-Router-Appliance), a more efficient router was part of +my initial upgrade plan. Looking at some of the [official Netgate options](https://www.pfsense.org/products/), I should be able to get the job done with +20W for a savings of 40W. + +## Conclusion +Adding up my potential savings, I optimistically have about 80W I can save at idle which comes to somewhere around 60kWh/month; that costs me a whole +$6.50/month. That's not even a trip to Starbucks in terms of energy costs. This doesn't match my expectations from reading forum threads about optimizing +every watt for cost savings, but according to [this random study](https://www.statista.com/statistics/263492/electricity-prices-in-selected-countries/), +this may be because energy costs in much of Europe are much higher (I would be looking at up to $32.80/month savings in Italy according to this study). + +This changes my perspective about where to spend time and effort in my upgrades; yes I can get a more efficient router, but I'm only going to save about +$3/month so that's not a good enough reason for me to spend $200. A new router could get me another 1U of available rack space, but I don't think I +will need that either in the immediate future. For the hard drives, I do still want to keep those spun down as much as possible just for longevity; also, +the issue of waiting for drives to spin up just to load network shares is a pretty common annoyance for me. + +This was a worthwhile investment of $50 and some time to actually do the math on what I stand to save with more efficient hardware. Another of many lessons +that you shouldn't believe everything you read on the internet and spending the time to at least sanity check your assumptions with concrete data is always +worth it. +