• Saik0A
    link
    fedilink
    English
    arrow-up
    63
    arrow-down
    2
    ·
    1 year ago

    I host my own. Specifically for myself and those who are friends or friends of friends.

    I have a cluster of servers operating in my garage. Free real-estate for tons of stuff I want to host. I have to “pay” for electricity… the rest was already paid for long ago. My electricity cost for my whole cluster… is an estimated $1750 a year. But that cluster is 160 CPU cores, 750 GB of RAM, and ~400TB of storage. You ain’t getting that on a cloud hosted provider for $145 a month. About $110 of that is subsidized by my business operations. I host email, websites, nextcloud, plex, etc… boatloads of stuff.

    • datendefekt@lemmy.ml
      link
      fedilink
      English
      arrow-up
      25
      ·
      1 year ago

      You offer paid hosting with hardware in your garage? I’m getting dot-com flashbacks!

      • Saik0A
        link
        fedilink
        English
        arrow-up
        29
        arrow-down
        3
        ·
        edit-2
        1 year ago

        I actually do… Yes… I have dual internet connections, dedicated power off both phases of my electricity with ~5 hours of battery backup, run redundant internal infrastructure (power, network, and server hardware)… I also have a massive backup library and am currently working on obtaining offsite backups solution. The whole site/house is also monitored with cameras.

        I have better uptime than some datacenters in my area have which I can truthfully quantify that as I also hold a CIO/CISO position in another company that operates out of a major Datacenter in my area.

        Edit: Should clarify, an offsite backup solution that ISN’T “just peer with backblaze or some other provider” to store your backups. I intend to do a rotating tape library. With one set of tapes always being off site.

        Also to mention… I have better uptime than AWS-east at this point of the year… Although there will be some outages of my infrastructure here soon for a network hardware update. I suspect something in the order of about 1 minute of downtime total.

          • Saik0A
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            Sadly, Just lost it :( I knew the switch upgrades would do it… Eh. One more set of switch upgrades (40gbps trunk) here soon and they’ll be fully redundant too.

      • Saik0A
        link
        fedilink
        English
        arrow-up
        14
        arrow-down
        1
        ·
        edit-2
        1 year ago

        What kind of load balancing are you thinking about/looking for?

        If we look at Lemmy in specific…

        Jobs that a server performs for users on it’s platform:
        Login services(are you even you to begin with?)
        Session Management (being logged in at multiple places, not mixing you up with other users…)
        Subscriptions (what content to even show you… organizing them on the page… etc.)
        Ban lists (and applying them to what you’re looking at)
        Peering with the other platforms

        Jobs that a server performs for off platform users: Sending ActivityPub messages to the other user’s platform.(effectively the same as “peering with other platforms” from above)

        The idea is that the jobs that the server has to do for it’s local users is actually a lot more both numerically and in taxing tasks than the jobs that one particular instance has to do to send updates to the federated instances… A lot of the list in the first case is a boatload of SQL queries. The activitypub notification is effectively a broadcast of a relatively simple text that doesn’t take much bandwidth.

        So let’s look at actual cases now that we have some ideas of what instances are doing.

        I run a small instance, I have a few users (because I purposefully keep it small). If my users only ever interact with the fediverse through my instance, then other instances don’t have to do most of the jobs in the top list. I’ve taken that load from lemmy.ml or other bigger instances that my users would have ultimately migrated to if they didn’t have my server. The tradeoff is that my few users now cost the 1 connection for activitypub notifications.

        Lets look at a theoretical “perfect” setup… 2,000,000 users across let’s say 1,000 instances. Evenly distributed. So each instance has 2,000 users. In that case, any given server would have to serve 2,000 people from the upper list and 999 activitypub broadcasts. This is significantly easier to do than have a single server try to handle all 2,000,000 users on a single instance serving nothing to the activitypub broadcasts.

        There’s is one caveat with this… activitypub broadcasts will send everything that is subscribed… even if users don’t actually view it. So there could be some waste. This then leads to the discussion of well how many users until there’s a breakeven in cost of the first list vs the potential waste of the activitypub bandwidth. That answer is debatable… But most people agree that take the discussion seriously that the return on investment could be as low as 2-5 users. I think that the bandwidth cost outweighs the SQL (CPU, RAM, etc…) costs quite handily personally.

        Other useful functions that occurs with this setup… As long as I got the ActivityPub message on my server… my users can see all of Lemmy.ml’s content. Regardless of if Lemmy.ml is actually running or not. We saw that with the downtime that some of the bigger servers are seeing, all the other instances could still consume the content… Just no updates were being sent.

        This is relatively simplistic… I’ve skipped some nitty gritty stuff… I’ve simplified some other stuff. But this is the gist.

        Edit: Other functions that you need to think about that lemmy is accomplishing for local instance users… Notifications, saved posts, language filtering, settings, profiles, avatars, reporting, slur filters, moderation, NSFW filters… The top list is actually considerably larger… like I said we simplified.

        • d0m@lemmy.world
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          1 year ago

          Great explanation, thanks for the write up! I have a couple of questions, if I may.

          • So if I setup my own instance tomorrow, it’d be sending notifications of activity to all the other instances? Maybe I’d have to have a list of instances I’d like to notify? How does it discover the other instances?

          • If my instance is listening to all other instances? Won’t saving all posts take up a lot of disk space? I guess there is only text and link data and no media, but wouldn’t it add up fast as the number of users/instances grow ?

          • Saik0A
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            it’d be sending notifications of activity to all the other instances?

            No, you’d only be sending to instances that subscribed to your instance… Or when you conduct and action. So for instance you’d need to subscribe to my community for my instance to send the notifications to you. Also when you upvote,downvote, etc… your instance will send a message out to the origin server to tally that action.

            Maybe I’d have to have a list of instances I’d like to notify?

            This is handled kind of automatically. You will automatically talk to the relevant instances when you subscribe to communities.

            How does it discover the other instances?

            When you subscribe to [email protected]… your instance will reach out to the lemmy.ml instance.

            Won’t saving all posts take up a lot of disk space?

            You’d think… But I’m probably many thousands of posts saved at this point and 15GB of used storage. It’s all text… text is really small. It’s image uploads that will end up eating your disk space. Don’t upload many images and you’ll be fine! There are already scripts and stuff that people have put together to clean out older materials from an instance.

    • Tak@lemmy.ml
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 year ago

      Now you just need solar or something feeding a battery to cover most of your electricity costs.