One thing I would add to a modern backup strategy: a deferred offline copy
Sure it's bothersome to use the sneakernet once a month (manually copy data to USB stick and stash it somewhere), but when that ransomware hits (can happen on personal devices also), it literally can't access the offline copy, and try to encrypt that one also.
Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view. Offline requires physical access, and maybe a big enough wrench to get the USB stick's disk encryption PIN out of you.
Tape + Iron Mountain is difficult to beat for offline copies.
Surrendering just a little bit of absolute control over the physical media can dramatically increase the likelihood that it survives whatever apocalyptic event. I don't know that I've ever heard of tapes being stolen from Iron Mountain. Even if this happened, what are the odds that they would get all the tapes. Your tapes? It probably looks like the warehouse from Raiders of the Lost Ark in there.
Anything that requires manual work will inevitably fail as a backup strategy in my opinion. You might not need that backup now, while you actively think about it, but in 4 years, when you had other problems to deal with and haven't done a backup in a long time.
For small datasets, a cloud backup (think s3 or azure blob) with credentials that can't be harvested automatically by a malware (eg custom backup script with encrypted credentials - claude will happily write one for you in seconds) is as good as offline. For small datasets (code base, important documents, even photos if you don't go crazy - or perhaps backup a lower resolution as a dooms day last resort thing), this is nearly free.
For larger datasets, you can buy some cheap X11SSH-LN4F or X11SSL-F motherboards on ebay with RAM and CPU for ~$100. These can be remotely switched on and off programmatically with IPMI (same thing, custom scripts with encrypted credentials - claude is your friend). Have your NAS perform an incremental backup once a week or once a month and keep it off the rest of the time (or trigger it from a raspberry pi with own credentials if you don't want to connect IPMI to your LAN or have encrypted IPMI credentials on your NAS). And unless a malware hits right at the time of the backup, it is as good as an offline backup while also being automated. Doesn't protect from a power surge though, which may or may not be a problem depending on where you live.
Also have your backup pull data from your NAS rather than the other way round, and run with different credentials than your NAS, so a malware can't jump from the NAS to the backup (or encrypt the backup). I have seen first hand that if you reuse admin credentials between machines, one machine compromised means all machines compromised within minutes.
My main backup server uses ZFS and I replicate (using my own https://github.com/fazalmajid/zfssnap but there are plenty of alternatives) to a 14TB USB drive using ZFS snapshot and incremental zfs send which makes it very fast, using ZFS encryption in case they get stolen.
I have two of these and rotate them weekly and keep one at my work (and I have a self-hosted healthchecks.io instance to remind me if I forget).
I have a locker at my place of work where I store a few HDDs and USBs. These are the most up to date, but I also have others at my parents place and my in-laws. Gets troublesome keeping track of which ones are up to date as of what date. Good challenge for staying organised though, I've got a whole naming system and numbered hierarchy and scripts that run ordered by priority.
I use a similar system. I have paper console tape on each drive and write the date last used on it and also keep a text file log of which drive and when. The backup batch file also writes a timestamp.txt to the root of the drive.
I also have a set I curate and put in with the family & household papers. That doesn't have any of my personal crap, just documents, photos, and family history etc.
I think there is a system to do this with git annex that will just keep the hash link in a repo but have the file in a offline drive, so you can easily keep track of files that are in cold storage. It may need you to have your own encryption system though.
I do this too, but with ZFS. So, since the snapshots have the creation timestamp in their name, it's obvious which drive has the latest data. But I also tend to remember if I went to the office or to my parents' house last.
I only have a small collection of bits in true regularly updated offline backup (my authentication/other DB, some important documents).
An extra protection against the possibility of malicious access such as ransom attacks is a two-step “soft-offline” backup.
The source machines backup to a central place, and the backup machines pull copies from there. The important part is that the source machines can not connect (or at least can not authenticate against) the final backups and vice-versa, so a malicious process/person getting into one can not affect the other and vice versa. Obviously if I don't notice the damage immediately then the most recent backups may be corrupted because damaged data was pulled, but past snapshots (taken after each pull at the backup side) will still be clean.
You have to be very careful about storing credentials to make sure source credentials don't ever end up on the backup machines and backup credentials don't ever end up on the source machines, even well out of the way of normal places like ~/.ssh, because a targetted attack might find them, but it gives almost the assurance of an offline backup while still being fully automatable. My backup site credentials are in my true offline backups, I need to refer to them for maintenance access, and then they get cycled after that use.
Restores can be mediated the same way. Verifying backups can be done by both sides running hashes on the files and posting the list back to the central machine for comparison - anything that differs without having a timestamp after the previous snapshot is likely corruption.
As a side note about corruption: if doing snapshots in the filesystem (“cp -al after rsync” or one of the many similar options) make sure you have more than one snapshot chains (on separate storage if you have resource for that). If you have a file that hasn't changed in years so every one of your snapshots points to that one version, it could only take one random filesystem error to completely lose the file.
Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view.
Yeah, but an attacker is unlikely to compromise both you and your storage provider at the same point. Many S3-compatible storage providers (e.g. Backblaze and Hetzner) support object lock, where you can lock objects for a certain number of days (and refresh locks if the objects are still used in recent backups). Typically object locks are implemented such that not even you yourself can remove the data from the account settings.
E.g. when I cancelled my Backblaze B2 account, I had to wait until the object locks expired before I could remove the remaining data and delete the account.
Arq on macOS has great support for object locks BTW.
But a malware will scan for anything that looks like some credentials on your machine. So if you have a script with clear credentials backing up to a ftp server, the malware will go after the backup.
At the end of the day it depends on what is your threat model. Mine is 1) automated malwares and 2) my own fuckups. I am not trying to prevent the NSA from hacking me. Against an automated malware, custom scripts with encrypted credentials that don't show in clear in command lines or environment variables are probably good enough.
Ideally an offline copy that can be made read-only with a physical switch. So that when you're trying to restore, you know nothing is going to mess with it. Not sure what the right solution is for something like that.
I get that a lot of immutable backup solutions are code, but they don’t have to be. You can do stuff like block writes in firmware or even blow fuses on the TX traces.
I play modded Starfield. There's been a clear increase in mods lately. Some of them are from self-proclaimed non-programmers who are using the LLM's to reverse engineer the game or other abandoned mods, and they've started to create new cool mods or they've fixed various engine limitations. Can confirm that these actually work and I can finally have my 1000+ modlist.
There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.
There is a point beyond which the probability of finding meaningful content is too low to justify the work required to find it. Beyond this point, limiting yourself means never trying.
Do you mean that they should [stop (absorbing everything)] and therefore absorb only _some_ things, or that they should [(stop absorbing) everything] and therefore absorb nothing?
You have one copy of data (the data that needs to be backed up), and you have two additional copies of that data (the "3" in the strategy), where one of the backup copies is stored offsite (the "1" part), which naturally follows that it satisfies the middle "2" also.
For over a decade we used 4-3-2 strategy, but it got expensive fast and due to ever evolving budget cuts, our (local gov) customers weren't ready to pay the extra.
There's also 3-2-1-1-0, but that's just a fancy way of saying you _really_ should be doing those restore tests you never seem to have the time, because ransomware can and _will_ corrupt your backup copies.
edit: the absolute minimal setup that still satisfies the 3-2-1 rule: 1) you have your data in production directory on local NVMe stick inside your computer, 2) you copy the original production data to another directory inside the very same NVMe, 3) you occasionally manually copy the production directory data to USB and store that USB inside the shed in your backyard. - That's it. Now your home data is covered by the coveted "3-2-1" enterprise ready backup strategy.
Closed-garden mobile-first tech feudalism lobotimized the Internet. I remember trying out the Freenets, I2Ps, and whatnot in the 90's. Too cumbersome nerdy tools to use for anything actually worthwhile. It was born in the idealized world village of the 90's Internet. Maybe real life doesn't work like that, and the lobotimized centralized tech will eventually win.
[Internet facing router with up to date firmware] --> HTTPS --> [separate VLAN DMZ] --> [my hardcore IndieWeb VM/k8s/bare-metal whatever] --> [x No outbound access / paranoid local firewall inside the VM x]
[My home computer] --> SSH --> [my hardcore IndieWeb local cloud]
It's sad that we need this new concept of "IndieWeb", as the whole Internet evolved into a monstrosity hosted and guardrailed by a handful of megacorporations. Hosting files became a privilege, when it should've been a (human) right all along.
edit: The tech to host yourself is obviously still there, but the _mindset_ changed to cloud only.
Sure it's bothersome to use the sneakernet once a month (manually copy data to USB stick and stash it somewhere), but when that ransomware hits (can happen on personal devices also), it literally can't access the offline copy, and try to encrypt that one also.
Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view. Offline requires physical access, and maybe a big enough wrench to get the USB stick's disk encryption PIN out of you.
reply