Love

From Sixsigma
(Redirected from Server)
Jump to: navigation, search

This is my latest server, 'love'. It is destined to be the primary file server and VM host on my network. It will store archives, backups, and shared data (such as images, packages, downloads, media, etc.) and run a bunch of network critical VM guests in VirtualBox. Backups will be processed on love for now, one day we'll commission happiness as the primary backup server (happiness will be able to store many more backups than love can).

This page is a bit of a mess of notes from various stages of research and planning. The actual final specs of the components we actually purchased are the components listed in the Scorptec Computers section.

Configuration

I'm going to build the system with dedup and compression on, then do some testing. Then rebuild the system with dedup and compression off, and compare. My guess is that I want dedup and compression both off. That will save CPU time and RAM at the cost of some magnetic storage (which we have more than enough of).

ZFS config

See the ZFS documentation or the ZFS FAQ.

See love-disk.hml for info on disk partitions.

THINK: should /fast be ZFS or ext4 on MD-raid?

These partition commands adapted from here. They might need revision...

# gpart create -s gpt sde
# gpart create -s gpt sdf
# gpart add -t freebsd-zfs -b 2048 -a 4k -l log0 -s 20G sde
# gpart add -t freebsd-zfs -b 2048 -a 4k -l log1 -s 20G sdf
# gpart add -t freebsd-zfs -a 4k -l fast0 sde
# gpart add -t freebsd-zfs -a 4k -l fast1 sdf

Note: with the above, if we can't put the /boot partition on the NVMe drive, the logs will become 18GB and /boot will be 2GB.

Note: /data:logs are mirrored, not striped, for 20GB usable. See Setting Up Separate ZFS Log Devices which says the log should be 1/2 the size of RAM (i.e. 16GB).

# apt install zfsutils-linux
# zpool create fast mirror /dev/sde2 /dev/sdf2
# zpool add fast log /dev/nvme0n1p2
# zpool create data mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd
# zpool add data log /dev/sde1 /dev/sdf1
# zpool add data cache /dev/nvme0n1p3

See How to improve ZFS performance and ZFS Compression: lz4 VS lzjb.

# zfs set compression=lz4 fast
# zfs set compression=lz4 data

According to How to improve ZFS performance you need to upgrade ZFS from time-to-time, like this:

# zfs upgrade -a
# zpool upgrade -a

Also according to How to improve ZFS performance the ARC and L2ARC settings should be tuned. THINK: can we use the / SSD for L2ARC?

See Expanding a zpool and adding ZIL (log) and L2ARC (cache) for info on how to configure L2ARC and logs.

Also according to How to improve ZFS performance we should disable deduping for better performance and after reading ZFS: Stripe & Parity, or Stripe & Mirror? Also... SLOG & L2ARC? and Inappropriate use of deduplication I'm gonna leave dedup off:

# zfs set dedup=off fast
# zfs set dedup=off data

According to ZFS FAULTED / kernel: (ada0:ahcich4:0:0:0): lost device in the BIOS configure AHCI instead of IDE.

Disable atime:

# zfs set atime=off fast
# zfs set atime=off data

And storing xattrs as system attributes significantly decreases the amount of disk I/O:

# zfs set xattr=sa fast
# zfs set xattr=sa data

From StackOverflow some handy commands:

  • ls: Shows the uncompressed size on files, but the compressed size on folders
  • zfs get used <pool>: Shows the compressed space of all files in the pool
  • zfs get logicalused <pool>: Shows the uncompressed space that all the files in the pool would use
  • zfs get compressratio <pool>: Shows the average compressratio of the pool
  • du -h --apparent-size: Shows the uncompressed size of the given files/folders
  • du -h: Shows the compressed size of the given files/folders

Note: I considered redundant_metadata=most, but I didn't entirely understand it, and it sounded safer to use the default setting 'all'.

Note: I need to do more reading on ARC config. I'm not certain but I'm thinking 4GB to 8GB RAM reserved for ARC. Actually, no, now I'm thinking 12GB for ARC leaving 20GB for other purposes.

Note: I need to think more about if I want to use sync=always. Might be good for safety if it doesn't kill perf too much.

Note: as we don't have a UPS on this host we're gonna use sync=always until we do.

# zfs set sync=always fast
# zfs set sync=always data

See ZFS Configuration Part 2: ZVols, LZ4, ARC, and ZILs Explained for info about configuring ARC via GRUB. I'm thinking I will configure ARC with 12GB and keep 4GB for dedup leaving 16GB left for the system (and its VM guests).

Upgrades

Airgap

On 2019-08-16 'AIRGAP 1' started throwing errors in 'orac'. Ordered replacement WD WD4003FFBX 4TB Red PRO 3.5” 7200RPM 256MB Cache SATA3 NAS Hard Drive for $276.45 (after discount).

32GB RAM

Get 2x16GB ECC Buffered RAM. CPU supports up to 2400Mhz.

RAM upgrade options:

Fast disk

When it comes time to upgrade disk capacity use M.2 drives and adapter, e.g.:

Best option for $647:

The above SATA options are probably best, here are some NVMe considerations:

Or consider:

Optical drive

Do we care for an optical drive?

Shopping

WASD Keyboards

Component Notes Unit Price Quantity Price Arrived
CODE V3 104-Key Mechanical Keyboard - Cherry MX Green Black case, no sound dampeners US$175.00 1 US$175.00
17" Soft Foam Wrist Pad US$14.00 1 US$14.00
Freight US$69.50
Total US$258.50

Memory.NET

Component Warranty Unit Price Quantity Price Arrived
M391A2K43BB1-CRC - Samsung 1x 16GB DDR4-2400 ECC UDIMM PC4-19200T-E Dual Rank x8 Module Lifetime $208.39 2 $416.78
Freight $69.41
Total $486.19

eBay

Component Note Price Quantity Total Arrived
SSD M.2 250GB WD Blue 2280 SATA Solid State Drive $75.95 2 $151.90
M.2 Nvme SSD NGFF TO PCI-E X4 3.0 PCB Adapter M Key B Key Dual Interface Ri C1G1 low form factor $27.46 1 $27.46 2019-03-18
WD 4TB Red PRO 3.5” 7200RPM SATA3 NAS Hard Drive airgap $268.51 2 $537.02 2019-03-18
Samsung 16GB 1Rx4 DDR4-2400MHZ PC4-2400T ECC Registered 288Pin Server Memory Ram $199.00 2 $398.00 2019-03-18
Delock 89588 interface cards/adapter Internal SATA PCI Express Card 4x $167.00 1 $167.00 2019-03-20
SSD M.2 500GB WD Blue 2280 SATA Solid State Drive $120.00 2 $240.00 2019-03-11
2x Compressed Air Duster Can Cleaner 400ml for Notebook Laptop PC Keyboard AU $18.65 1 $18.65 2019-03-18
Netgear GS108 ProSafe 8 Port Gigabit Home Desktop Ethernet Unmanaged Switch $53.00 1 $53.00 2019-03-11
Network Cable CAT6 1000Mbps 5m Green, 5m Grey $9.95 2 $19.90 2019-03-18
Network Cable CAT6 1000Mbps 3m Black, 3m Blue $6.95 2 $13.90 2019-03-18
2 Port HDMI KVM Switch with Cables EL-21UHC E6I2 $33.37 1 $33.37
KVM Switch 2 Port ATEN CS22D USB DVI with cable remote $64.90 1 $64.90
Power Shield Defender LCD 650VA UPS $115.00 1 $115.00 2019-03-19
Total: $1,840.1

Scorptec Computers

Gonna put Ubuntu LTS on the SSD and put the four mag disks in RAID10 (software raid) for 8TB usable for ZFS /data.

Component Warranty Unit Price Quantity Price Arrived
HP Enterprise ProLiant X3421 Gen10 Microserver 1 year $899.00 1 $899.00 2019-03-11
Western Digital WD Ultrastar DC HC310 4TB HDD 5 years $329.00 4 $1,316.00 2019-03-11
Western Digital WD Black 500GB PCIe NVMe M.2 (PCIe) SSD 5 years $229.00 1 $229.00 2019-03-11
Aten CS22U 2 Port KVM 2 years $32.00 1 $32.00 2019-03-11
Kingston DTSE9G2 16GB USB 3.0 DataTraveler 5 years $12.00 3 $36.00 2019-03-11
Sub-total $2,512.00
Freight $68.00
Surcharge $25.80
Total $2,605.80
Component Warranty Unit Price Quantity Price Arrived
Silverstone M.2 NVMe SSD NGFF to PCIeX4 Adapter Card 1 year $35 1 $35 2019-03-11
Tax and shipping $12.47
Total $47.47

Stone Computers

Gonna put Ubuntu LTS on the EVO and put the four WD Golds in RAID10 (software raid) for 8TB usable for ZFS /data.

Product Code Product Name Unit Price Quantity Price
PC-HP-873830-375 HPE ProLiant MicroServer Gen10 (873830-375) Server $729 1 $729
HD-WD4000FYYZ-WD4002FYYZ 4TB WD4002FYYZ WD Gold SATA3-6gb/s HDD (128MB, 7200rpm) $311 4 $1,244
SSD-SAMSUNG-MZ-V7E250BW 250GB Samsung 970 EVO M.2 SSD (MZ-V7E250BW) $119 1 $119
NET-ATEN-CS-22U Aten Petite CS-22U 2 Port USB KVM Switch $47 1 $47
Total Price: $2,139

Need to replace SATA drive with NVME drive:

Old SSD-WD-WDS120G2G0B 120GB WD GREEN 3D NAND SSD, M.2 SATA $53 1 $53
New SSD-SAMSUNG-MZ-V7E250BW 250GB Samsung 970 EVO M.2 SSD (MZ-V7E250BW) $119 1 $119

Amazon.com.au

Gonna put Ubuntu LTS on the WD Black and put the four WD Golds in RAID10 (software raid) for 8TB usable.

Component Price Units Total
HP ProLiant MicroServer Gen10 AMD Opteron X3216, 8GB ECC UDIMM $837 x1 $837
WD 4TB Gold Enterprise-Class Hard Drive - WD4002FYYZ $316 x4 $1,267
WD Black SN750 500GB NVMe Internal Gaming SSD - Gen3 PCIe, M.2 2280, 3D NAND - WDS500G3X0C $188 x1 $188
QNINE M.2 NVME to PCIe Adapter $15 x1 $15
Aten 2-Port USB VGA Cable KVM Switch $45 x1 $45
$2,355

staticICE

Component Vendor (price) Vendor (price) Vendor (price) Vendor (price)
HPE ProLiant MicroServer Gen10 TechBuy ($714) Stone Computers ($729) i.store ($729)
WD 4TB Gold ARC ($291) Stone Computers ($311) i.store ($311) RAM City ($319)
WD Black 500GB NVMe PC Case Gear ($179) CPLOnline ($199) BudgetPC ($199) PLE Computers ($199)
M.2 NVME to PCIe Adapter Scorptec ($35)
2-Port USB VGA Cable KVM Switch BudgetPC ($32) JW Computers ($33) i-Tech ($34) Dick Smith ($34)

Research

Prices

Component Unit Price Qty Total Price
Server $566 1 $566
SSD disk $130 1 $130
Mag disk $232 4 $928
KVM $23 1 $23
UPS $94 1 $94
$1,741

Server

CPU

SSD disk

Mag disk

KVM

UPS

RAM

Mag disk

12 TB disk

Newegg 2-item combo: Total: $133.50/ea $267

4 TB disk

Note: we almost certainly don't want 5400RPM drives.