I host my own. Specifically for myself and those who are friends or friends of friends.
I have a cluster of servers operating in my garage. Free real-estate for tons of stuff I want to host. I have to “pay” for electricity… the rest was already paid for long ago. My electricity cost for my whole cluster… is an estimated $1750 a year. But that cluster is 160 CPU cores, 750 GB of RAM, and ~400TB of storage. You ain’t getting that on a cloud hosted provider for $145 a month. About $110 of that is subsidized by my business operations. I host email, websites, nextcloud, plex, etc… boatloads of stuff.
I actually do… Yes… I have dual internet connections, dedicated power off both phases of my electricity with ~5 hours of battery backup, run redundant internal infrastructure (power, network, and server hardware)… I also have a massive backup library and am currently working on obtaining offsite backups solution. The whole site/house is also monitored with cameras.
I have better uptime than some datacenters in my area have which I can truthfully quantify that as I also hold a CIO/CISO position in another company that operates out of a major Datacenter in my area.
Edit: Should clarify, an offsite backup solution that ISN’T “just peer with backblaze or some other provider” to store your backups. I intend to do a rotating tape library. With one set of tapes always being off site.
Also to mention… I have better uptime than AWS-east at this point of the year… Although there will be some outages of my infrastructure here soon for a network hardware update. I suspect something in the order of about 1 minute of downtime total.
Sadly, Just lost it :( I knew the switch upgrades would do it… Eh. One more set of switch upgrades (40gbps trunk) here soon and they’ll be fully redundant too.
What kind of load balancing are you thinking about/looking for?
If we look at Lemmy in specific…
Jobs that a server performs for users on it’s platform:
Login services(are you even you to begin with?)
Session Management (being logged in at multiple places, not mixing you up with other users…)
Subscriptions (what content to even show you… organizing them on the page… etc.)
Ban lists (and applying them to what you’re looking at)
Peering with the other platforms
Jobs that a server performs for off platform users:
Sending ActivityPub messages to the other user’s platform.(effectively the same as “peering with other platforms” from above)
The idea is that the jobs that the server has to do for it’s local users is actually a lot more both numerically and in taxing tasks than the jobs that one particular instance has to do to send updates to the federated instances… A lot of the list in the first case is a boatload of SQL queries. The activitypub notification is effectively a broadcast of a relatively simple text that doesn’t take much bandwidth.
So let’s look at actual cases now that we have some ideas of what instances are doing.
I run a small instance, I have a few users (because I purposefully keep it small). If my users only ever interact with the fediverse through my instance, then other instances don’t have to do most of the jobs in the top list. I’ve taken that load from lemmy.ml or other bigger instances that my users would have ultimately migrated to if they didn’t have my server. The tradeoff is that my few users now cost the 1 connection for activitypub notifications.
Lets look at a theoretical “perfect” setup… 2,000,000 users across let’s say 1,000 instances. Evenly distributed. So each instance has 2,000 users. In that case, any given server would have to serve 2,000 people from the upper list and 999 activitypub broadcasts. This is significantly easier to do than have a single server try to handle all 2,000,000 users on a single instance serving nothing to the activitypub broadcasts.
There’s is one caveat with this… activitypub broadcasts will send everything that is subscribed… even if users don’t actually view it. So there could be some waste. This then leads to the discussion of well how many users until there’s a breakeven in cost of the first list vs the potential waste of the activitypub bandwidth. That answer is debatable… But most people agree that take the discussion seriously that the return on investment could be as low as 2-5 users. I think that the bandwidth cost outweighs the SQL (CPU, RAM, etc…) costs quite handily personally.
Other useful functions that occurs with this setup… As long as I got the ActivityPub message on my server… my users can see all of Lemmy.ml’s content. Regardless of if Lemmy.ml is actually running or not. We saw that with the downtime that some of the bigger servers are seeing, all the other instances could still consume the content… Just no updates were being sent.
This is relatively simplistic… I’ve skipped some nitty gritty stuff… I’ve simplified some other stuff. But this is the gist.
Edit: Other functions that you need to think about that lemmy is accomplishing for local instance users… Notifications, saved posts, language filtering, settings, profiles, avatars, reporting, slur filters, moderation, NSFW filters… The top list is actually considerably larger… like I said we simplified.
Great explanation, thanks for the write up! I have a couple of questions, if I may.
So if I setup my own instance tomorrow, it’d be sending notifications of activity to all the other instances? Maybe I’d have to have a list of instances I’d like to notify? How does it discover the other instances?
If my instance is listening to all other instances? Won’t saving all posts take up a lot of disk space? I guess there is only text and link data and no media, but wouldn’t it add up fast as the number of users/instances grow ?
it’d be sending notifications of activity to all the other instances?
No, you’d only be sending to instances that subscribed to your instance… Or when you conduct and action. So for instance you’d need to subscribe to my community for my instance to send the notifications to you. Also when you upvote,downvote, etc… your instance will send a message out to the origin server to tally that action.
Maybe I’d have to have a list of instances I’d like to notify?
This is handled kind of automatically. You will automatically talk to the relevant instances when you subscribe to communities.
Won’t saving all posts take up a lot of disk space?
You’d think… But I’m probably many thousands of posts saved at this point and 15GB of used storage. It’s all text… text is really small. It’s image uploads that will end up eating your disk space. Don’t upload many images and you’ll be fine! There are already scripts and stuff that people have put together to clean out older materials from an instance.
I host my own. Specifically for myself and those who are friends or friends of friends.
I have a cluster of servers operating in my garage. Free real-estate for tons of stuff I want to host. I have to “pay” for electricity… the rest was already paid for long ago. My electricity cost for my whole cluster… is an estimated $1750 a year. But that cluster is 160 CPU cores, 750 GB of RAM, and ~400TB of storage. You ain’t getting that on a cloud hosted provider for $145 a month. About $110 of that is subsidized by my business operations. I host email, websites, nextcloud, plex, etc… boatloads of stuff.
You offer paid hosting with hardware in your garage? I’m getting dot-com flashbacks!
I actually do… Yes… I have dual internet connections, dedicated power off both phases of my electricity with ~5 hours of battery backup, run redundant internal infrastructure (power, network, and server hardware)… I also have a massive backup library and am currently working on obtaining offsite backups solution. The whole site/house is also monitored with cameras.
I have better uptime than some datacenters in my area have which I can truthfully quantify that as I also hold a CIO/CISO position in another company that operates out of a major Datacenter in my area.
Edit: Should clarify, an offsite backup solution that ISN’T “just peer with backblaze or some other provider” to store your backups. I intend to do a rotating tape library. With one set of tapes always being off site.
Also to mention… I have better uptime than AWS-east at this point of the year… Although there will be some outages of my infrastructure here soon for a network hardware update. I suspect something in the order of about 1 minute of downtime total.
That would be why I added my edit :)
deleted by creator
Mom’s spare bedroom closet 70 miles away, but yeah.
Gigachad is hitting five nines in their garage.
Sadly, Just lost it :( I knew the switch upgrades would do it… Eh. One more set of switch upgrades (40gbps trunk) here soon and they’ll be fully redundant too.
I was wondering what Gilfoyle was up to these days
forget Gilfoyle I wanna know what Anton is doing to Chat GPT
How does load balancing work across this fediverse… if at all?
What kind of load balancing are you thinking about/looking for?
If we look at Lemmy in specific…
Jobs that a server performs for users on it’s platform:
Login services(are you even you to begin with?)
Session Management (being logged in at multiple places, not mixing you up with other users…)
Subscriptions (what content to even show you… organizing them on the page… etc.)
Ban lists (and applying them to what you’re looking at)
Peering with the other platforms
Jobs that a server performs for off platform users: Sending ActivityPub messages to the other user’s platform.(effectively the same as “peering with other platforms” from above)
The idea is that the jobs that the server has to do for it’s local users is actually a lot more both numerically and in taxing tasks than the jobs that one particular instance has to do to send updates to the federated instances… A lot of the list in the first case is a boatload of SQL queries. The activitypub notification is effectively a broadcast of a relatively simple text that doesn’t take much bandwidth.
So let’s look at actual cases now that we have some ideas of what instances are doing.
I run a small instance, I have a few users (because I purposefully keep it small). If my users only ever interact with the fediverse through my instance, then other instances don’t have to do most of the jobs in the top list. I’ve taken that load from lemmy.ml or other bigger instances that my users would have ultimately migrated to if they didn’t have my server. The tradeoff is that my few users now cost the 1 connection for activitypub notifications.
Lets look at a theoretical “perfect” setup… 2,000,000 users across let’s say 1,000 instances. Evenly distributed. So each instance has 2,000 users. In that case, any given server would have to serve 2,000 people from the upper list and 999 activitypub broadcasts. This is significantly easier to do than have a single server try to handle all 2,000,000 users on a single instance serving nothing to the activitypub broadcasts.
There’s is one caveat with this… activitypub broadcasts will send everything that is subscribed… even if users don’t actually view it. So there could be some waste. This then leads to the discussion of well how many users until there’s a breakeven in cost of the first list vs the potential waste of the activitypub bandwidth. That answer is debatable… But most people agree that take the discussion seriously that the return on investment could be as low as 2-5 users. I think that the bandwidth cost outweighs the SQL (CPU, RAM, etc…) costs quite handily personally.
Other useful functions that occurs with this setup… As long as I got the ActivityPub message on my server… my users can see all of Lemmy.ml’s content. Regardless of if Lemmy.ml is actually running or not. We saw that with the downtime that some of the bigger servers are seeing, all the other instances could still consume the content… Just no updates were being sent.
This is relatively simplistic… I’ve skipped some nitty gritty stuff… I’ve simplified some other stuff. But this is the gist.
Edit: Other functions that you need to think about that lemmy is accomplishing for local instance users… Notifications, saved posts, language filtering, settings, profiles, avatars, reporting, slur filters, moderation, NSFW filters… The top list is actually considerably larger… like I said we simplified.
Great explanation, thanks for the write up! I have a couple of questions, if I may.
So if I setup my own instance tomorrow, it’d be sending notifications of activity to all the other instances? Maybe I’d have to have a list of instances I’d like to notify? How does it discover the other instances?
If my instance is listening to all other instances? Won’t saving all posts take up a lot of disk space? I guess there is only text and link data and no media, but wouldn’t it add up fast as the number of users/instances grow ?
No, you’d only be sending to instances that subscribed to your instance… Or when you conduct and action. So for instance you’d need to subscribe to my community for my instance to send the notifications to you. Also when you upvote,downvote, etc… your instance will send a message out to the origin server to tally that action.
This is handled kind of automatically. You will automatically talk to the relevant instances when you subscribe to communities.
When you subscribe to [email protected]… your instance will reach out to the lemmy.ml instance.
You’d think… But I’m probably many thousands of posts saved at this point and 15GB of used storage. It’s all text… text is really small. It’s image uploads that will end up eating your disk space. Don’t upload many images and you’ll be fine! There are already scripts and stuff that people have put together to clean out older materials from an instance.
Now you just need solar or something feeding a battery to cover most of your electricity costs.