April 18, 2025

Remote Backups Should Be Append-Only, or You Risk Losing Everything

April 18, 2025

Backups are often seen as a last line of defense — a safety net if everything else fails. But that safety net is only as reliable as its configuration. One subtle but critical aspect of remote backups is often overlooked: whether the storage system allows deletion or modification of existing backup data.

If backups are stored in a location where they can be overwritten or erased — even accidentally — they are vulnerable. Whether the cause is a bug, a misconfigured retention policy, or a compromised account, the result is the same: the backups vanish. And when you reach for them in a crisis, they’re simply not there.

This is why remote backups should be configured as append-only whenever possible. It’s not about convenience. It’s about survivability.

The Fragility of Mutable Backups

A typical backup system writes data to a remote object store, file server, or dedicated backup appliance. Depending on the tool or protocol, it may also clean up old snapshots or manage retention windows automatically. These deletion operations are often taken for granted — after all, nobody wants to store infinite backups forever.

But the moment deletion is allowed, the backup system becomes a potential source of data loss. If the client is compromised, an attacker can simply delete all backups. If the backup software has a bug, it can clean up the wrong set. If the retention configuration is wrong, you may keep far less history than you thought.

There’s a fundamental assumption in most backup architectures: the backup system won’t fail in a destructive way. But history shows otherwise.

Attack Scenarios

One of the most common patterns in ransomware attacks is this: encrypt the production data, then delete the backups. If the backup system is reachable with the same credentials or network access as the main system, it becomes an easy target. Even sophisticated backup software isn’t immune. If it can delete backups by design, that ability can be abused.

Worse, deletions often don’t raise alarms. If a backup job finishes successfully — having deleted old snapshots per its schedule — nothing looks suspicious. Until the day you need a file, and the backup from that time is already gone.

Append-only storage neutralizes this risk. If no process can remove or modify existing backup data, then the worst a compromised client can do is stop making new backups. That’s bad — but it’s recoverable. Data from yesterday still exists. An attacker can’t erase your history.

Bugs and Misconfigurations

Not all failures are malicious. A misconfigured backup job that prunes too aggressively can quietly shrink your available history to a few days. Retention periods might be applied globally instead of per-client. Time calculations might go wrong in edge cases. Backup metadata might become inconsistent, resulting in deletions during cleanup routines.

These are subtle issues. They don’t trigger alerts or crash the system. They don’t make logs scream. They just erase your safety net without anyone noticing.

Append-only backups sidestep this entire class of problems. Once data is written, it stays. If you want to expire old data, you do so deliberately, from a different system with higher privilege and greater scrutiny.

Implementing Append-Only in Practice

Many modern object stores and backup platforms support append-only or write-once-read-many (WORM) configurations.

  • AWS S3 supports object lock policies that prevent deletion or overwriting of files for a defined retention period. Once enabled, even account administrators cannot remove protected data until the policy expires.
  • Immutable snapshots in systems like ZFS or EBS provide read-only restore points that cannot be altered once created.
  • Backup appliances like Veeam or Rubrik offer hardened repositories that enforce immutability at the storage layer, independent of the backup client.

If you build your own system, the same principles apply: backups should go to storage that cannot be modified or deleted by the client writing them. Ideally, the client does not even have the ability to list or delete previous snapshots. It just writes — and that's it.

Local vs. Remote Considerations

Append-only configurations are particularly important for remote backups — backups that reside in a different region, availability zone, or cloud account. These are the backups that survive physical disasters, ransomware, and large-scale outages. But they are also often network-accessible from the same systems being backed up.

This creates a paradox: your most important backups are stored where they are most at risk. Unless access is carefully isolated, or deletion is explicitly blocked, remote backups can be destroyed from the very systems they’re meant to protect against.

Append-only policies create a boundary. Even if your main systems fail catastrophically — whether from attack, mistake, or outage — your backup history remains intact.

Backups Are Only Useful If They Exist

A backup system that can delete its own history is a liability. It works until the day it doesn’t — and when it fails, it tends to fail silently and completely. You might not know the backups are gone until you’re depending on them.

Append-only backups aren't a nice-to-have. They're a requirement for any system that takes resilience seriously. They protect not just against hardware failure, but against human error, malicious access, and software bugs. They turn a backup from a hopeful assumption into a guaranteed asset.

When you’re designing your backup strategy, ask one question: what’s the worst thing that can happen to this data? If the answer includes “it could be deleted,” then your backups aren’t finished.

Enhance Your Business with Scalar Dynamic Consulting Services

Unlock the potential of your business with Scalar Dynamic's consulting services. Our specialized offerings, Scalar Compass and Scalar Exceed, revolutionize the way businesses handle systems analysis, technology project governance, infrastructure, DevOps, and cloud services. We are dedicated to boosting your business with customized solutions that emphasize efficiency and quality.

Interested in DevOps, Infrastructure, and Cloud Services?
Explore Scalar Exceed
Interested in Systems Analysis and Project Governance?
Explore Scalar Compass

Here's why our services stand out:

01

Extensive Hands-On Experience

With decades of hands-on experience, we are more than just another consultancy. Our team has been in the trenches, actively developing software as part of our cloud software offering. This real-world experience ensures we bring practical, effective solutions to your business.

02

High Attention to Detail

We prioritize your business and your product with meticulous attention to detail. Our commitment goes beyond a single project; we aim to build long-term relationships. Your project is never just a task for us — it's an opportunity to partner with you for sustained success.

03

Continuous Improvement and Support

Our commitment to you doesn't end with project completion. We provide ongoing support and continuous improvement for all our services and software. We ensure your business remains at the cutting edge, adapting and thriving in a constantly evolving landscape.