How Cloud PBS Saved My Home Lab: A Disaster Recovery Story

People often treat the cloud as something evil. They don't want their data leaving their premises. I used to think the same way. But recently, the cloud saved my entire home lab from complete disaster.

This is a story about mistakes, failed hardware, counterfeit drives, and how a cloud backup service I set up "just to test" became the only thing standing between me and losing years of work.

The VMware to Proxmox Migration

About 1.5 years ago, I wrote about my journey from VMware to Proxmox. For years at previous employers and in my own lab, I was a VMware guy. But with recent changes to the VMUG program, getting licenses became harder. Proxmox was the logical choice.

With VMware, I always installed the hypervisor on a USB drive. The VMs lived on separate datastores on proper disks. This setup worked flawlessly for years in my lab. Never had a single issue.

So naturally, I thought I'd do the same with Proxmox.

That was my first mistake.

The First Failure

Everything ran fine for about six months. Then came a power outage.

When power returned, disaster struck. Kernel panic. Rewrite errors on the USB drive. The system wouldn't boot.

I thought to myself: "Alright, I probably bought a cheap USB stick. Should've gone with something more reliable."

Since my VMs were on separate disks, recovery wasn't terrible. But I had to pull the Intel NUCs out of my rack, connect them to a monitor, and reinstall everything. You know the hassle. My wife was not happy because certain home services stopped working.

New OS. New USB drives. This time I bought better ones, more expensive, supposedly more reliable. Rebuilt the cluster from scratch.

The Second Failure (Same Mistake)

Six months later. Another outage. Same result.

All three nodes dead. USB drives corrupted. Kernel panics everywhere.

I was furious. At myself, mostly. The assumption that I couldn't install VMs on the same disk as the OS (something true for VMware ESXi) didn't apply to Proxmox at all. Proxmox handles this differently, and I should have known better.

Time to get rid of the USB sticks permanently.

Then My NAS Decided to Join the Party

While dealing with the USB disaster, my storage situation got worse.

My NAS had been running RAID 6 for years with Western Digital Red 3TB drives. Two drives started showing warnings simultaneously. RAID 6 can handle two drive failures, but I was now at high risk.

I checked the drive stats: 3,600 days of uptime. That's almost 10 years of continuous operation. These drives had earned their retirement.

I found replacement WD Red 3TB drives online, checked the prices, and ordered two. They arrived about three weeks later.

I didn't notice they shipped from China.

The Counterfeit Drive Disaster

The drives looked identical to my originals. Same labels, same form factor, same everything. I installed them in the NAS, initiated the RAID rebuild, and felt relieved.

Two days later, both new drives failed.

At this point, I started questioning my NAS itself. If drives that ran for 10 years died, and brand new drives died after two days, maybe the NAS was the problem?

I decided to buy a new NAS. But that's a story for another blog post.

The immediate problem: my backup storage was dead. And my Proxmox Backup Server VM was running on the same cluster that just failed.

I was in deep trouble.

The Cloud PBS Lifesaver

Here's where things could have gone really bad. My backup strategy included syncing to Google Drive, but those backups were only accessible through my PBS appliance, which was now gone.

But months earlier, I had tested Cloud PBS. It's a managed Proxmox Backup Server hosted in the cloud. I set it up, configured it to run alongside my local backups, and honestly forgot about it.

That "testing" saved everything.

What is Cloud PBS?

Cloud PBS is a managed backup solution specifically designed for Proxmox Virtual Environment. Instead of running your own Proxmox Backup Server on local hardware (which, as I learned, can fail along with everything else), Cloud PBS hosts your backups in a secure, redundant cloud infrastructure.

Key features that matter:

• Native Proxmox Integration: Works exactly like a local PBS. You add it as a storage target in your Proxmox cluster, and backups flow automatically.

• Deduplication: Just like local PBS, Cloud PBS deduplicates data. You're not storing full copies every time, only changed blocks.

• Encryption: Your data is encrypted in transit and at rest. Important when sending backups over the internet.

• Redundant Storage: Your backups exist on infrastructure designed for high availability. No single point of failure like my USB sticks or aging NAS.

• Bandwidth Efficient: Incremental backups mean you're only sending changes, not full VM images every time.

Setting Up Cloud PBS

Setting up Cloud PBS is straightforward. After signing up at cloud-pbs.com, you receive connection credentials for your cloud-hosted PBS instance.

In your Proxmox cluster, you add Cloud PBS as a storage target:

1. Navigate to Datacenter → Storage → Add → Proxmox Backup Server
2. Enter the Cloud PBS server address, datastore name, and credentials
3. Configure fingerprint verification for secure connection
4. Set up your backup schedule

The configuration looks nearly identical to setting up a local PBS. The only difference is the server address points to the cloud instead of your local network.

Server: your-instance.cloud-pbs.com
Datastore: your-datastore
Username: your-user@pbs
Fingerprint: XX:XX:XX:XX:...

Once configured, you create backup jobs just like you would for local PBS. The backups run on schedule, deduplicate data, and sync to the cloud.

The Recovery Process

When my cluster died, I had two options for recovery:

Option 1: Restore VM by VM

After reinstalling Proxmox on my nodes (this time on proper SSDs, no more USB drives), I added the Cloud PBS as a storage target. From there, I could browse available backups and restore individual VMs directly.

Option 2: Sync Cloud PBS to Local PBS

Cloud PBS can sync its contents to a local Proxmox Backup Server. This is useful if you want to restore from local storage for better performance, or if you're rebuilding your entire backup infrastructure.

I chose Option 1 for critical VMs and was surprised by the performance.

Performance That Impressed Me

I expected the restore to take forever. Downloading entire VMs over the internet? I figured I'd be waiting days.

Instead, Cloud PBS completely saturated my internet bandwidth. The service pulled data as fast as my connection allowed. Within a few hours, my critical VMs were back online.

The deduplication helped here too. Since only unique data blocks needed to transfer, and many of my VMs shared common base images, the actual data transferred was significantly less than the total VM sizes.

Lessons Learned

1. Don't run Proxmox on USB drives.

This works for VMware ESXi. It does not work reliably for Proxmox. Install on a proper SSD or NVMe drive.

2. Your backup server shouldn't live on the infrastructure it's backing up.

My PBS VM running on the same cluster it backed up was a single point of failure. When the cluster died, so did my backup access.

3. Off-site backups are not optional.

Local backups are great for quick restores. But when your local storage fails (NAS, drives, whatever), you need something completely separate.

4. "Testing" a backup solution means nothing if you don't keep it running.

I set up Cloud PBS to test it. Then I left it running. That accidental decision saved me.

5. Verify your hardware sources.

Those counterfeit drives from China cost me time and nearly cost me data. Buy from reputable sources, even if it costs more.

6. Test your restores.

I knew Cloud PBS worked because I had tested restoring a VM when I first set it up. That confidence made the real disaster recovery much less stressful.

The Cost Question

People always ask about cost. Cloud PBS pricing is based on storage used. For a home lab, the cost is reasonable, especially compared to:

• Buying and maintaining backup hardware
• Electricity for running that hardware
• Replacing failed drives
• The value of your time when disasters happen

My entire restore took a few hours. If I had to rebuild from scratch without backups? Days, maybe weeks. Some configurations I'd never fully remember.

Current Setup

After this disaster, my backup strategy looks different:

+------------------+     +------------------+     +------------------+
|  Proxmox Nodes   |     |  Proxmox Nodes   |     |  Proxmox Nodes   |
|   (SSD boot)     |     |   (SSD boot)     |     |   (SSD boot)     |
+--------+---------+     +--------+---------+     +--------+---------+
         |                        |                        |
         +------------------------+------------------------+
                                  |
                    +-------------+-------------+
                    |                           |
           +--------v--------+         +--------v--------+
           |   Local PBS     |         |   Cloud PBS     |
           | (New NAS/SSD)   |         | (cloud-pbs.com) |
           +-----------------+         +-----------------+

Local PBS handles daily backups for fast restores. Cloud PBS runs in parallel for disaster recovery. If my local infrastructure fails again (and eventually, all hardware fails), I have a completely independent copy in the cloud.

Happy Lab, Happy Life

My wife doesn't care about Proxmox, deduplication, or backup strategies. She cares that the home automation works, the media server streams, and the network stays up.

Cloud PBS kept those services down for hours instead of days. That's the real value.

Shout out to Emmanuel Le Nohaïc who gave me access to Cloud PBS for testing. That "test" turned into a production lifesaver.

If you're running a Proxmox home lab without off-site backups, learn from my mistakes. Set up Cloud PBS, configure it to run alongside your local backups, and hope you never need it.

But if you do need it, you'll be very glad it's there.

Resources:

• Cloud PBS - Managed Proxmox Backup Server
• Cloud PBS Documentation - Setup and configuration guides
• Proxmox Backup Server Docs - Official PBS documentation
• My Previous Post: My Proxmox Backup Strategy - Local backup setup details