d_mcknight/blog-content
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add existing content

Browse source
This commit is contained in:
Daniel McKnight 2023-12-20 19:52:09 -08:00
commit eb235624a3
3 changed files with 268 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

89
2023-11-29_Homelab-Upgrades.md Normal file
View file

@ -0,0 +1,89 @@
---
date: 2023-11-29
title: Planning a Homelab Upgrade
tags:
- homelab
- hardware
- networking
- unraid
- proxmox
- pfsense
---
I am just starting to plan a major overhaul of my homelab and decided where better to start than by documenting the process.
This is about how my setup has reached its current state and why this setup is starting to feel limiting.
## A Brief History
The first home server I used was a Sunfire X4150 that I got for something like $50. Windows Home Server 2011 seemed
like a logical choice of OS (this is where I started; I didn't know what I was doing) so I grabbed a copy from my local Fry's (RIP).
After some adventures in data recovery from a dead drive (read: data loss), I built an [unRaid](https://unraid.net/) box.
One of the main motivations here was that I could add drives over time and increase capacity easily without having to match
drive models or even sizes; this meant I could use all the random drives I had lying around and save some money.
My current server is basically unchanged since then, with a few upgrades, namely a 3900X and some 8TB HDDs shucked out of
WD external drives when they went on sale.
## Taking Inventory
The bulk of my setup lives in a 15U rack that I built into the wall of my office. I also have some other networking equipment,
a tower for work, and a personal machine I built into my desk but those aren't relevant to this project.
I'll work through my current setup from the top of the rack down.
### Networking
- (2U) 48 port patch panel: This is where networking infrastructure like switches and access points, IP cameras, and other
rack equipment terminates
- (1U) 48 port Ubiquiti POE switch: This handles switching for most of the devices terminating at the patch panel
- (1U) 8 port Ubiquiti 10GbE switch: This connects other switches and provides a 10gig connection to my Unraid server
- (2U) pfSense router: This is just an old PC in a 2U case with an Intel Gigabit NIC
### Compute
- (3U) Unraid server: This is where all of my storage and compute is
- (1U) X4150 server: I spin this up manually every month or so to manually back up data from my Unraid server. Its loud and power
hungry so I leave it shutdown as much as possible
### Power
- (2U) TrippLite UPS: This provides enough power to run things for about 15 minutes, long enough to safely shutdown and even keep
networking up through a brief outage
For anyone keeping score at home, that leaves 3U empty which does factor into my plan.
## Current Shortcomings
Now that I've laid out what I have and how I got there, let's look at some pain points.
### Power Consumption
Much of my setup has been built from surplus or repurposed systems. The pfSense router in particular is overbuilt for the minimal
compute that routing requires. The backup X4150 server is mostly powered off, but when running for a backup it takes a lot of power
and turns it into a lot of heat and noise in my office.
### Availability and Maintenance
I have a pfSense VM defined on my Unraid server, so I can at least take the pfSense router down for maintenance without network
downtime but the same cannot be said about the Unraid server. If the Unraid server is down, Plex, HomeAssistant, NextCloud, this blog,
and a lot of other things go down. I also have some issues with my pfSense box resuming after a power outage with is particularly
annoying if I'm not home to turn it back on.
### Compute
My 12-Core 3900X is generally more than enough compute for me now, but it is occasionally limiting and will not support all of the
services I want to deploy. I also need more than 32GB of RAM, but that's an easy upgrade.
## The Plan
There are two main upgrades I want to make to address the current shortcomings.
### pfSense Router Appliance
This one's easy, I just want to get a low-power box to run pfSense since it needs to run 24/7;
I'll likely pick up a used network appliance on eBay for this.
### Clustered Compute
This still requires a bit more research. I've been becoming more acquanted with Kubernetes as
[part of my day job](https://github.com/NeonGeckoCom/neon-diana-utils) and am settled on setting up a cluster at home.
I will likely use [Proxmox](https://www.proxmox.com/en/) since it checks the boxes for being open source, having a web UI
for management, and being popular enough to find support online.
I'll use my current pfSense router and Unraid server as nodes, possibly adding another new node with a GPU to play with some LLMs.
### Storage
I am as of yet undecided on what to do here; I need to do some more reading up on GlusterFS, Ceph, iSCSI,
and probably some technologies I know even less about.
## What Comes Next
This post is already getting a lot longer than I anticipated, so I'll wrap it up here. My next steps are to (1) find an appropriate
upgrade for my pfSense router and (2) decide on a storage solution.

114
2023-12-01_Current-Software-Stack.md Normal file
View file

@ -0,0 +1,114 @@
---
date: 2023-12-01
title: My Current Software Stack
tags:
- homelab
- software
- unraid
---
As part of planning [some hardware upgrades](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/),
I think it is important to evaluate the software side of things to make sure everything is accounted for
in the new design. This is also a good opportunity to evaluate and prune what I have running before
deciding where to move things around.
## Virtual Machines
I run a few VMs on my Unraid server; many are seldom used but a couple of them I use on a regular basis.
None of these need anything like five-nines availability, but they do need somewhere to run when needed.
- Cura: I have an Ubuntu VM that is dedicated to running Cura for 3D printing. I use several diferent
computers throughout the day, so being able to slice and print things wherever I am (including from
my phone) is really handy.
- Windows 10: This Windows VM has a Bluray drive attached for working with my media collection or the
odd piece of software that I have on optical media.
- pfSense: This VM is configured with an Intel NIC to be manually spun up before taking my main pfSense
box down for maintenance. This VM isn't used often but it is essential to keeping the network up.
## Container Services
This is where the bulk of my services exist. Unraid uses Docker with some specialized templating to
provide a nice Web UI for deployment, but I would be happy writing my own [Helm charts](https://helm.sh/) or
[Docker Compose](https://docs.docker.com/compose/) files to deploy things. There are a lot of containers here
so I'll try and organize them into some logical groups.
### Core Networking
These services are essential to keeping my network up and supporting other services. Outages here
can bring down other services or parts of my network.
- DDClient: lets my domain registrar know when my public IP address changes
- LetsEncrypt: reverse-proxy for routing traffic to the appropriate container
- Unifi: controller and web UI for Ubiquiti networking
- Mosquitto: MQTT server used to connect ZWave devices to HomeAssistant
### Database and Backend
These provide support for other services. An outage here will affect some specific
other services.
- MariaDB: used by some other containers; I wouldn't mind eliminating or replacing this with Mongodb if I can.
I know practically no SQL and a little Mongodb
- zwavejs2mqtt: connects to zwave devices to HomeAssistant
- Frigate: security camera monitoring, object detection, and recording
### User-Facing Services
These are the things you probably think of as what a home server is used for. Everything
here has a Web UI or is accessible via client software (or both).
- hexo-blog: this.
- Vaultwarden: password manager for Bitwarden clients
- Plex: media server; I may move to Emby or Jellyfin as media tagging has been less reliable for me lately
and the recent [watch history debacle](https://www.techhive.com/article/2157803/plex-discover-together-privacy-concerns.html)
is another nudge to find something better.
- Nextcloud: cloud storage, contacts, calendar, client device backups
- Matrix: [matrix chat](https://matrix.org/) home server
- HomeAssistant: smart home controls/automation and camera monitoring
- Heimdall: web dashboard; I just played with this and don't find myself using it often so I'll probably drop it
- Grafana: monitoring dashboard
- GitLab: self-hosted git; I don't use this much but do like having a GitHub alternative available for private
projects (yes, I know you can have private repos on GitHub)
### Monitoring
These services monitor the other services. If something goes wrong here I'm likely the only one to notice.
- Influxdb: time-series database used to collect data for Grafana dashboards
- Tautulli: pulls Plex analytics, used to populate Grafana
- Varken: More Plex analytics middle-ware to push analytics to Influxdb
- UniFi-Poller: pulls Unifi analytics into Influxdb for Grafana
- Telegraf: pulls Unraid server analytics into Influxdb for Grafana
### Mostly Unused
A few containers I have mostly spun-down. Some of these are for testing/evaluation and others I just haven't bothered removing.
- ActualServer and Firefly-III: personal budgeting apps since [Mint](https://support.creditkarma.com/s/article/Intuit-Mint-and-Credit-Karma)
is going away; neither really does what Mind does for me but I need an alternative since Mint is shutting down at the end of the year and
CreditKarma doesn't appear to be getting budgets which is what Mint did for me.
- Handbrake: video encoder; I will probably run this in a VM if I move containers to k8s since I only run this as an Unraid container to
easily leverage all of my server's cores
- Genea: family tree builder, because that's the kind of data I won't put into a random webapp on the internet
- element-web: a Matrix client; I installed this with Matrix but never use it
- RabbitMQ: MQ server that I setup once in the early days of the [Diana backend](https://github.com/NeonGeckoCom/neon-diana-utils) I made
for work.
- Wireshark: packet sniffing and more network stuff; realistically I would run this on a workstation if I need it
## Other
Aside from VMs and Containers, there is some basic functionality that needs to be addressed, namely data shares and VPN access.
### Network Shares
Unraid exposes SMB shares with some basic user access control. Shares on Unraid can utilize separate Cache and Array storage pools;
both pools offer redundancy but the smaller Cache pool is faster than the larger Array pool. I'll go into storage in more detail when
I look at solutions for storage.
### WireGuard
Unraid includes a WireGuard server which I use to remote into my network when travelling if I need to access the filesystem or a shell
directly. This is particularly helpful if I need to restart a container or find something on a network share that isn't served via Plex
or Nextloud.
## Putting It All Together
This was all a long way of getting to what I really need in my software stack:
- Virtualization with device passthrough, i.e. a [KVM](https://www.linux-kvm.org/page/Main_Page) Hypervisor
- Container support, likely [k3s](https://k3s.io/)
- VPN access. I might delegate this to pfSense since that would allow for remote access in the event the cluster
is offline so long as the router is on
- Network shares
I already mentioned [in my previous post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/) that upgrading my pfSense router
and picking a storage solution. I found some old network appliances repurposed to run pfSense [on ebay](https://www.ebay.com/itm/186181814221)
and plan on picking one up for a relatively inexpensive, low-power, and (hopefully) reliable solution.

65
2023-12-08_Optimizing-Power-Consumption.md Normal file
View file

@ -0,0 +1,65 @@
---
date: 2023-12-08
title: Optimizing Power Consumption
tags:
- homelab
- hardware
- power efficiency
---
I mentioned in [an earlier post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/) that one of my concerns with my current setup is the
power consumption of everything in my rack. After writing, I realized that I know what the load is on my UPS, but I don't know how each component
draws on its own or what everything actually costs to run. I also haven't looked into tuning things for power consumption for some time, so lets see
if there are any savings to be had there. I picked up a couple of inexpensive ZWave switches that have current monitoring and added them to HomeAssistant
to monitor my Unraid server and pfSense router. I can approximate the network switches' power usage by subtracting those 2 measurements from the UPS'
measured usage.
## Baseline Idle Power Usage
I connected my Unraid server to one of the monitoring outlets yesterday afternoon, so I have some data including minimal usage overnight. I see that
150-170W was used while streaming some media via Plex and then at bed time usage dropped to 140W with overnight load staying between 130-160W. There was
one peak at 165W, presumably when some scheduled tasks run at 4:30AM.
Checking the UEFI, I saw the fans all set to "Full Speed". I updated all of them to "Balanced" profiles and that helped with noise in my office which is
a huge plus, but power usage is mostly unchanged. This is by no means a scientific test but does indicate there is little power efficiency to be gained
by messing with fan curves.
For my pfSense router, things are more consistent with power draw right around 58W with a range of 57-60W. I went into the BIOS and disabled
overclocking and turned fans to "Eco mode" but saw very little change.
Doing the math, my UPS measures about 250W of usage with about 150W and 60W going to my Unraid server and pfSense router, respectively. This leaves 40W
going to the 2 Ubiquiti switches in the rack.
## Looking for Power Savings
There are a couple places I think I can save energy, so what can I do to validate that before investing more time and money? For the router, I
know there's not much room to optimize what I have, so I can compare my measured power draw against other hardware options. For the server, I think there
is some more investigation to do:
### Spinning Rust
Looking at my Unraid array, I see that all 7 spinning disks are active. I am able to manually spin down 4 of them (one has activity which also
keeps the 2 parity disks active). This takes power draw from 154W to 134W; that's a full 20W savings! Guessing that Frigate would be the only
service that won't write to my cache pool, I spun that down and then manually spun down everything in the array again. I'm now down to 111W,
until the disks spin back up.
This gives me some ideas about planning my storage solution. First, I'll want to try and get all of the services with regular disk IO using SSD-based
storage. Second, if I can keep file metadata in RAM or SSD storage, that will hopefully prevent the disk from needing to be spun up just to open a network
directory with the added benefit of not having to wait for disks to spin up just to look at directory contents.
### pfSense Router
As mentioned [in my earlier post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/#pfSense-Router-Appliance), a more efficient router was part of
my initial upgrade plan. Looking at some of the [official Netgate options](https://www.pfsense.org/products/), I should be able to get the job done with
20W for a savings of 40W.
## Conclusion
Adding up my potential savings, I optimistically have about 80W I can save at idle which comes to somewhere around 60kWh/month; that costs me a whole
$6.50/month. That's not even a trip to Starbucks in terms of energy costs. This doesn't match my expectations from reading forum threads about optimizing
every watt for cost savings, but according to [this random study](https://www.statista.com/statistics/263492/electricity-prices-in-selected-countries/),
this may be because energy costs in much of Europe are much higher (I would be looking at up to $32.80/month savings in Italy according to this study).
This changes my perspective about where to spend time and effort in my upgrades; yes I can get a more efficient router, but I'm only going to save about
$3/month so that's not a good enough reason for me to spend $200. A new router could get me another 1U of available rack space, but I don't think I
will need that either in the immediate future. For the hard drives, I do still want to keep those spun down as much as possible just for longevity; also,
the issue of waiting for drives to spin up just to load network shares is a pretty common annoyance for me.
This was a worthwhile investment of $50 and some time to actually do the math on what I stand to save with more efficient hardware. Another of many lessons
that you shouldn't believe everything you read on the internet and spending the time to at least sanity check your assumptions with concrete data is always
worth it.