I’ve been doing things the wrong way for a long time and now it’s time to pass my incompetence onto others.

  • 0 Posts
  • 27 Comments
Joined 1 year ago
cake
Cake day: June 7th, 2023

help-circle






  • It won’t work well. Other instances make assumptions about paths as it’s standardized. For example they don’t store the full pictrs path, just the ID. It’s assumed the path is /pictrs/image. You would have to set 301 redirects. Other instances may link directly to your root uri when making links back to you. The api requests are also likely going to your root uri.

    The front end can do whatever you want. Just change the paths typescript files before you compile as those also use root urls.

    I have my instance setup to use a CDN and separate URL for all static content including pictrs. I have a script that modifies the front end while compiling to accomplish this. I use 301 redirects to fix all the assumed paths and requests from other instances.


  • Proxmox and shell scripts. I have everything automated from base install to updates.

    All the VMs are Debian which install with a custom seed file. Each VM has a config script that will completely setup all users, ip tables, software, mounts, etc. SSL certs are updated on one machine with acme.sh and then pushed out as necessary.

    One of these days I’ll get into docker but half the fun is making it all work. I need some time to properly set it up and learn how to configure it securely.




  • At home I have a Proxmox cluster consisting of two Dell R340 and one Intel NUC. There are 25 VM’s across all three machines. They do various development duties along with home assistant, plex, and Blue Iris. The rack lives in a closet under the stairs and I have fiber ran to my office. We did a massive renovation when we purchased so I wired the entire house since the walls were all opened. Average power draw is around 480 watts.

    Here is a picture of the rack back when it was all R330. Those have since been sold and upgraded to R340. I added vents during the renovation. Inlet temps stay around the house ambient, and exhaust is about 20 degrees F hotter. I cover the front with additional sound baffles to better route fresh air and control noise. Its pretty much silent outside the closet.

    This is the patch panel. I have perimeter cameras all the way around the house plus more than enough wifi access points. Each room also got 2x ethernet on each side, 4x total. My office has got 6x ethernet plus 4x fiber and a 2-inch conduit to pull whatever else I can think of later.

    I use Grafana and custom-made scripts for monitoring and alerting. Most of the infrastructure is automated with scripts. One of these days Ill learn Ansible but I really enjoying just figuring it all out. This isn’t my job I just do it for fun. Here is the dashboard I run on one of my desk monitors.

    I run my hobby websites, and my Lemmy instance, in the colo but its primary purpose is to be an offsite backup. Proxmox backup server performs best on SSD hence the large array. I also do a lot of travel for work so that’s my remote dev machine too. I run my own mail servers with some small VPSs acting as SMTP and IMAP bouncers to internal servers at home and in the colo working in parallel. HA proxy does the bouncing for high availability and dovecot and postfix do the heavy lifting with solr providing lighting fast search. I do use a third party for outbound mail for better deliverability.

    Dell R350 - Colo Proxmox

    • Intel E-2388G processor
    • 128gb 3200 ECC ram
    • Dell H755p raid
    • 8x Crucial MX500 4TB in raid 6
    • Samsung 990 2TB NVMe

    Dell R340 - Proxmox Node 0

    • Intel E-2278G processor
    • 128gb 3200 ECC ram. Despite the spec sheets and irdrac saying these only support 64gb they run 128gb just fine.
    • Ultrastar DC SN640 7.68TB NVMe
    • Dell H810 Flash with LSI firmware. HBA for SC200 disk shelves.
    • Mellanox CX354A @ 40GbE

    Dell R340 - Proxmox Node 1

    • Intel i3-9100T processor
    • 64gb 2400T ECC ram
    • Ultrastar DC SN640 7.68TB NVMe
    • Intel X520-DA2 @ 2x 10gbe

    Intel 7th Gen Nuc - Proxmox Node 2

    • Intel i5-7260U processor
    • 32gb 2400 ram
    • Ultrastar DC SN640 7.68TB NVMe

    This is mounted to the wall under my desk in a silent case. I use Verizon Wireless home internet as a backup and this server is the router. My entire closet rack can go offline and Ill still have internet access.

    Dell R730 - GPU

    • 2x Intel E5-2696 V4 processor
    • 512gb 2400T ECC ram
    • SanDisk Skyhawk 3.84TB NVMe
    • 2x Nvidia P100 16gb GPU
    • Intel X520-DA2 @ 2x 10gbe
    • This one stays powered off when not in use. I built it to play around with tensorflow and AI but haven’t had much time.

    Dell SC200 Disk Shelf 1

    • 12x WD 8TB 5400 rpm shucked drives
    • Single z2 pool
    • Roughly 74 usable TB
    • Cold backups of the primary array. Only powered on once a month to sync.

    Dell SC200 Disk Shelf 2

    • 12x HGST 10tb SAS drives
    • 2x z2 pools
    • Roughly 72 usable TB
    • Primary storage array. Sitting at about 70% utilized so its time to upgrade.







  • I run opnsense, which has a long a storied history with pfsense and in my opinion is better, on a VM in proxmox.

    I have a cluster of three servers and I can live migrate the VMs around to do maintenance. It gets backed up to proxmox backup server so restoring from a bad upgrade, which I’ve never had happen, or severe experimentation, which happens frequently, is simple.

    It’s also one less device to power on, and pay for. My cluster is running regardless and every watt less helps keep my wife happy.

    I’ve never had any issues that I could attribute to it being run in a VM. It does my 1gbe fiber and a dozen vlans with no issues.


  • Do you plan on allowing other users? If so every image they upload anywhere will be hosted from your instance. You would need a long term plan for continued storage if you do.

    If it’s only yourself then not much. You would need space for your uploaded images and the database. Worst case you have to purge communities to free up DB space and re add them. It only tracks communities from the moment you add them. It doesn’t pull the entire history, and associated db size, into your instance.

    I can pull the real numbers when I’m not on my phone but my database and maybe 150mb now. You can see all the communities I follow here so get a relative idea: https://lemmy.cablepick.net/communities My instance has been up for 4 days. I should start tracking db size growth to give others an idea of what to expect.