2024-01-12 Storage Server

2024-01-11_Storage-Server.md

@@ -0,0 +1,113 @@
+---
+date: 2024-01-11
+title: Building a Storage Server
+tags: 
+  - homelab
+  - hardware
+  - truenas
+  - storage
+---
+
+Since [my first post](https://blog.mcknight.tech/2023/11/29/Homelab-Upgrades/#Storage), I've done the research and settled on a storage solution.
+I did a fair bit of reading up on different file systems, SAN (Storage Area Network) and NAS (Network Attached Storage) architectures, and backup/
+failover strategies. In the end I've decided to go with TrueNAS SCALE to serve up shares on a ZFS file system. I read a lot of articles and watched a
+lot of YouTube videos and I'll include links to some of the resources I found helpful at the end of this post.
+
+## Why ZFS?
+There are a lot of reasons to like ZFS and even more resources online explaining why you should use ZFS, but I'll highlight the ones that are important
+to me for my use case.
+- Expandability: you can add drives to an existing pool (not as easily as in Unraid, but its possible)
+- Data Integrity: the way ZFS handles writes does a lot to avoid data corruption
+- Snapshots: snapshots (aka shadow copies) allow for keeping backups at the file-system level with minimal effort and disk usage; 
+  they also enable easy replication for external backups.
+- Special Devices: a pool of hard drives can be augmented with a small amount of SSD storage for caching, file metadata, and more
+
+## Why TrueNAS SCALE?
+One of my goals in my homelab is to make administration and maintenance easy since I'm doing this in my "free time". Having a good Web UI means there's
+a shorter learning curve compared to needing to drop into a shell for things. TrueNAS SCALE is also a popular solution with an active community forum
+as well as a number of tutorials and how-to articles/videos. I chose TrueNAS SCALE over TrueNAS CORE primarily because it is based on Linux instead of BSD,
+so in the event I do need to drop into a terminal for something I will be more at home. To summarize, it checks a lot of boxes:
+- Understandable Management Web UI
+- Built on Linux for familiar permissions management and CLI
+- Supports SMB and NFS shares
+- Includes built-in backup (replication) services
+- Supports iSCSI for exposing raw storage (this [video from Craft Computing](https://www.youtube.com/watch?v=9JL-RVUHj6o) explained to me why I 
+  would care about iSCSI)
+- *Bonus*: The web UI supports 2FA
+
+## The Hardware
+Most of the hardware for this storage server came from ebay, so I'll list model numbers here (I don't want this to be another blog with a bunch of dead
+ebay links). I've excluded incidentals like power cables and screws; I also already had the storage on hand to put in this.
+
+|Component|Model|Description|Price|
+|---------|-----|-----------|-----|
+|Chassis|Supermicro SC826|2U Server with 12 3.5" bays|$219|
+|CPU/MB|Supermicro X9SCL with Xeon E3-1230V2|Motherboard and CPU|$46|
+|HBA|LSI 9207-8i|SAS Controller Card and cables|$84|
+|RAM|Crucual ECC Unbuffered UDIMM PC3L|32GB RAM Kit|$50|
+|Network Card|Intel E10G42BTDA|10GbE Network Card|$35|
+|PSU|PWS-1K28P-SQ|2x Quiet PSU|$100|
+|Fans|Nocuta NF-A8|3x fans|$56|
+
+The core components come in just under $350 with another $250 for some upgrades for a total of around $600. I initially deployed my server without those
+upgrades but ended up making them sooner rather than later for a few reasons:
+- More RAM means more cache for ZFS which can significantly help with accessing commonly used files
+- I already had a 10GbE network card in my old server, so adding one here means I can migrate data at faster than Gigabit speeds
+- The stock PSUs can get loud and this server lives in my office, about 3 feet from my head when I'm at my desk.
+- The stock fans can also get loud so I swapped in some appropriately sized Noctua replacements
+
+### Hard Drives
+For the storage, I'm using some schucked WD 8TB external drives. If you're doing the same, these drives won't work when 3V3 power is supplied; I found a good 
+[illustrated guide](https://imgur.com/a/BFdmB) for how to address this. I did this awhile ago with my drives and I did notice some of the tape was starting 
+to peel; if I did it again I might consider using conformal coating, but these are about to be permanently installed in my server so I didn't bother doing
+anything since they all worked immediately.
+
+![HDD with taped over pin](./hdd_mod.jpg)
+
+### 2.5" SSDs
+I have 2 1TB SSDs that I'll be using for a metadata VDEV to speed up my Hard Drive array. In short, this 1TB will be used to store file metadata (file names,
+access times, etc) while file contents will be stored on hard drives. This should significantly speed up browsing network shares and navigating the file
+system. 
+> I plan on re-doing this setup with 3 drives here so the redundancy matches my data VDEV (RAIDZ2) to resolve the warning in TrueNAS. It is also important to
+  note that warning exists because if the metadata VDEV fails, then the data is inaccessible, just as it would be if the data VDEV failed. Realistically, I
+  think its less likely for 2 concurrent SSD failures than it is for 2 concurrent HDD failures and rebuilding 1TB is faster than 8TB but its only another $50
+  to add a drive and I have the capacity in my chassis to accomodate it.
+
+I found a [model on thingiverse](https://www.thingiverse.com/thing:262035) to mount these in drive sleds. I also noticed while doing this
+that some of the drive sleds I got have 2.5" drive mounting holes and others don't.
+
+![Drive sled comparisons](./drive_sleds.jpg)
+
+
+### Noctua Fans
+If you have ever used a rack-mounted server, you know that noise is not really a consideration in their cooling solutions. The stock fans in these servers idle
+fairly quietly, but ramp up fast with any appreciable CPU load (i.e. data compression). I found a printable fan carrier 
+[on Thingiverse](https://www.thingiverse.com/thing:4919403) and printed out 3 of them to install Noctua fans in my server. The Thingiverse project notes 
+how the fan connector needs to be modified slightly to fit the fan connectors and the fit is nearly perfect and things are *MUCH* quieter.
+
+## OS Installation
+If you're building with the X9SCL board I'm using, I learned the hard way that PCIe storage devices will show up in BIOS, but you can *NOT* boot from PCIe. I
+also learned (and I think there may be a warning in the installer) that it is *NOT* recommended to install TrueNAS SCALE to a USB flash drive. For the moment,
+I've attached a 2.5" SSD to the internal USB port, but longer-term I intend on migrating to a SATA Disk-on-Module (DOM) to resolve the warnings in the Web UI
+(and to remove the SSD installed with 3M VHB tape).
+
+## Wrap Up
+There are a lot of considerations for how to structure ZFS VDEVs and implement shares and I'm still actively working out a solution for myself. Initially, I
+have created a RAIDZ2 Data VDEV with a RAIDZ1 Metadata VDEV and that is working well. I already see that my storage pool isn't big enough though and
+I might want to change some of the encryption and compression options so I'll wait to document things until I have a better idea of how I want this configured.
+
+For the moment, I have duplicated data from my Unraid server and will keep things mostly in sync until I finalize the new setup. I also have an offline
+backup of really important stuff, just in case.
+
+## References
+Perhaps my first introduction to ZFS was from a Level1Techs video [on YouTube](https://www.youtube.com/c/Level1Techs). Wendell also wrote a couple of
+forum posts about [TrueNAS SCALE](https://forum.level1techs.com/t/truenas-scale-ultimate-home-setup-incl-tailscale/186444) and
+[ZFS Metadata devices](https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954).
+
+As I got closer to implementation, I found Lawrence Systems on YouTube has a 
+[playlist](https://www.youtube.com/playlist?list=PLjGQNuuUzvmv_LHMcaub8QfrLD7Lc1nMW) of TrueNAS explainers and how-to videos. There's also some other videos
+on the channel about ZFS and general network design. I also previously used this channel as a resource for pfSense setup, but that's a topic for another day.
+
+There are many other great resources online so I encourage searching around and doing other research into alternatives; ZFS seems a good fit for me, but
+it may not be the best option for everyone.
+