The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.
I’ve done a lot of SysAdmin and DCOps stuff in the past so, thought I’d give you some plausible suggestions (haven’t dug deep into Lemmy DB stuff and DNS/Federation of the stack, so not sure all is practical).
Scenario 1 - Preserve and merge when access is restored
Setup
Spin up two VMs/VPS (or one that has enough grunt for two Lemmy servers). Call them robak.slrpnk.net and slrpnk.net and point DNS appropriately.
Pull federated content from other instances and place it on robak, set as read-only.
Sync important comms to (new) slrpnk.net without content.
Allow users to sign up, vetting as possible (all mods). Keep a list of those that are vetted (call it vetted.list). Inform all users that any non-vetted users will have their content dropped when access is restored.
Merge!
Once access is restored, ensure that (old) slrpnk.net is set to read-only.
Schedule a maintenance window (announce more time than you are likely to need).
During the maintenance window, put (new) slrpnk.net into R/O, or just block external access.
Off topic, but in Portugal was the first time me, a Central European, ever ate a ripe Papaya. Yum! Thanks to the Azores. The shit you get in the supermarkets round here is useless. Same for mangoes, usually.
You are in luck then, because both are climacteric fruits, meaning they continue to ripe after being harvested, due to producing ethylene gas that triggers the ripening
Throw them in a paper bag and let them rest at room temp. If you want to speed up the process, add a ripe banana or apple (or other ripe climacteric fruits)
You might have eaten a different cultivar than those exported. In that case, I agree, there are so many different varieties that we never get to taste, supermarkets usually only have 1 or 2 different types
It’s baffling to me how customisable some fruit features can be. I know someone who works in plant genomics and cultivation and it’s really cool how they can use genomic analysis alongside selective breeding to select for traits. I imagine that imported papayas are cultivars that are easier to transport internationally.
Wondering if a volunteer could go to the physical location necessary to restore service. If it’s in Athens Greece, I can make the trip.
Edit: The server is located in Azores, the flight would cost 415eur, I cannot afford that, maybe someone in Portugal could help?
Is it in a data centre or someone’s house? If the latter, would they let a stranger in?
Surely they would need a backup and replicate db to so in case of hardware failure they switch over.
Sounds like they could improve their setup.
Too much of a single point of failure.
Slrpnk.net admin here.
The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.
Appreciate the answer and the detail. Good luck getting it all resolved.
I’ve done a lot of SysAdmin and DCOps stuff in the past so, thought I’d give you some plausible suggestions (haven’t dug deep into Lemmy DB stuff and DNS/Federation of the stack, so not sure all is practical).
Scenario 1 - Preserve and merge when access is restored
Setup
robak.slrpnk.netandslrpnk.netand point DNS appropriately.Merge!
Scenario 2 - Server is in DC or Admin able to facilitate access
Probably quite expensive, and when doing something as a hobby it’s often hard to get the funds.
Off topic, but in Portugal was the first time me, a Central European, ever ate a ripe Papaya. Yum! Thanks to the Azores. The shit you get in the supermarkets round here is useless. Same for mangoes, usually.
You are in luck then, because both are climacteric fruits, meaning they continue to ripe after being harvested, due to producing ethylene gas that triggers the ripening
Throw them in a paper bag and let them rest at room temp. If you want to speed up the process, add a ripe banana or apple (or other ripe climacteric fruits)
The taste just isn’t the same.
You might have eaten a different cultivar than those exported. In that case, I agree, there are so many different varieties that we never get to taste, supermarkets usually only have 1 or 2 different types
It’s baffling to me how customisable some fruit features can be. I know someone who works in plant genomics and cultivation and it’s really cool how they can use genomic analysis alongside selective breeding to select for traits. I imagine that imported papayas are cultivars that are easier to transport internationally.