Go back up your HA installation right now.
I, like many others, was running HA OS on a raspberry pi with whatever sd card I happened to have at hand at the time when I started to play with the whole thing.
It has since became a part of my household with light automations and other stuff and with everything going on in life it’s been on a back burner to impove automations more, include more data, manage proper backups and so on. And specially the backups are the one I’ve been slacking on and now I have no one else to blame than myself.
Sd card eventually died yesterday evening and today when I started to study why the server went down but still responded to ping I was greeted with a docker error that exec format is wrong (or something along those lines). At first I thought that maybe some upgrade went sideways and decided to grab a image of the card before messing around it so I have something to come back to which failed almost immediately with IO errors on dmesg.
Gladly I could get a backup off from the card from october, so at worst I lost some automation tweaks on HA side but what I’m still afraid is that I might’ve lost Z-wave keys. Ddrescue is running on the card right now so I don’t have the final results yet but I’m not too hopeful that it’ll create a working image.
So, I did what I should’ve done already ages ago and installed HA on my proxmox server and added that to backup cycle so that’s been taken care of properly and now I’m not limited to Rpi3 performance with addons either. I still need to get a new installation for the pi with Z-wave JS due to RaZberry 7 hat but that should be pretty straightforward task, assuming I can retrieve the keys from broken card.
So: Don’t be like me, verify that you have proper backups of your stuff.
You must log in or register to comment.
It was such a breath of fresh air when I finally put HA onto proxmox.
“Oh, I actually have resource now? Sweet!”Best of luck with the Drescue mate. Update with the results?
IMHO RPi is still a good choice for HA. SD cards are cheap enough now that you can have a spare handy with Home Assistant OS already flashed on it, then if/when your current SD card dies, just swap it out and restore HA from last backup. Only takes a few minutes and happens about as often as a hard drive dies.
All depends on how much you value separation of concerns with a proxmox setup.
I recommend just using an USB 3.x external SSD.
The SD card dying all the time is not what you want and is easily avoided with a real drive
Beyond always back up everything, Pi’s and SD cards have always been a bummer. Even booting from an old memory stick is faster and more reliable.
Pi’s have supported foregoing the SD card and booting from USB since forever. I run a file server on a Pi5 with a USB SSD disk as main drive, never had any issues. Even bad blocks from wear are compensated for through the SSD’s built in system.
You can even chain load from SD card to usb on an older pi this doesn’t support USB boot.
You can run the entire thing off an nvme or ssd connected via usb3 on a pi. You don’t actually need to use the sd card. It’s just super convenient for testing major changes quickly. If it needs to run more than a week, don’t use the SD card slot.
Interesting that you have a proxmox server running but chose to run homeassistant on a raspberry instead. What was the goal here?
RPi came before Proxmox and I just kicked the migration further down a road a bit too long. Also, specially at the start, I didn’t want to have anything ‘quality of life’ critical stuff running on proxmox since it is (was) kind of my own sandbox to mess around with.
My HA runs on a Pi for various reasons, including GPIO devices, but I’ve moved its database to the ‘big’ server. That means a lot less load on the SD card, no loss of data if the card fails, and just generally feels like a better way to do things. As I’ve accumulated containers, I don’t like that each of them runs its own database when I could just have one database to manage & backup, and not have a lot of replicated overhead.
I have a Proxmox cluster and still went with a separate machine for HA. I figure the thing controlling my house should be on it’s own, since the cluster is more a playground for me than anything else.
That is very interesting to me, as I am currently setting up a cluster for all the important stuff (including home assistant), to have failover and high availability.
I haven’t figured out the whole high availability thing yet; I just move VMs/containers to different nodes if I need to bring a node down, or shift resources around, or whatever.
Ive been running Pis (full sized ones and zeros) from USB drives for a few years now and so far none of those have failed.
It’s the SD cards that fail, usually. The RPi runs and runs and runs, at least the older models. The newer ones might die when the fan dies, or at least shut down.
Yeah thats why getting rid of the SD card as the weakpoint is worth it. I had an SD card fail twice and havent used one in a SBC that does anything important since.
@Treczoks @unexposedhazard Your Pi has fans?
Pi5 with load - is good with small fan, yes.
Anything you have on SD cards should not be your only copy of that thing.
Anything you have on any storage really. At least that you don’t want to lose.
SD cards are mostly designed for use cases that do very little writing. There’s high endurance SD cards, but those are designed for long continuous writes (mainly for dashcams and security cameras). Home Assistant does a lot of small writes, which is the worst case scenario for an SD card.
Back when I used a Pi for Home Assistant, I had a SATA SSD attached to it using a cable like this: https://a.co/d/2tlYZW2.
These days I’d probably try a USB NVMe drive, like a SanDisk Extreme Portable SSD or similar product. NVMe drives were a bit iffy with the older Pis since some of them pull way more power than SATA SSDs and the Pi couldn’t always handle them, but it should be fine with the newer ones.
If you don’t have any offsite backups yet, I’d get a storage VPS (look for good deals on LowEndTalk on Black Friday!) or Hetzner storage box and back up to it using Borgbackup and Borgmatic.
I didn’t realise that the Pi 5 has an NVMe hat! Is that what you did? It’s a great solution.
The TLC NAND chips used in most commodity memory cards these days are only good for something like 1500 write cycles per cell before they are prone to wearing out and coming back with errors. The difference between a dedicated SSD and a dinky memory card is partially the amount of extra space reserved for relocating data as the memory cells wear out, of which a consumer MicroSD card typically has little (on expensive ones) or none (on cheap ones).
I’ve heard it said, or rather seen it written, that some TLC NAND can endure “up to” as many as 3000 write cycles, but everyone is cagey about the true number and most consumer grade card vendors are tight-lipped about what kind of chips are actually in their products. So in other words, if you’re just scarfing a cheap card off of Amazon or from Microcenter or whatever, don’t expect ironclad longevity.
The one thing with flash storage writes that’ll bite you and it’ll bite you fast is logging. Unix-like systems love to incessantly write little one line additions to oodles and oodles of log files all the time, and if you want to extend your poor overworked little SD card’s lifespan you can dabble in turning some of that stuff off, once you’re positive you don’t need it for troubleshooting.
There also exist high reliability cards sold for industrial embedded applications, which will use lower capacity SLC but be able to endure upwards of 100,000 write cycles (per the marketing literature, at least). Expect capacities to realistically top out at about 64 gigs and for a single unit at that capacity expect to pay north of $100 for the privilege. It may be more appealing to use an NVME SSD at that rate and connect it with USB adapter or a hat.
Also: run your HA on something that makes more sense than Pi. For a little more you can run a much more powerful N100/150 based mini PC with very similar power draw.
Pi makes sense, just don’t use an SD for anything you want to run more than a day.
A 7th-gen i7-powered tiny/mini/micro is perfect for HA. Plenty of grunt for lots of HA addons and integrations, lots of USB ports for dongles (zigbee, z-wave, etc), often with 2x M.2 slots (usually one B/M key and one A/E key) and SATA interface, very low power draw, and cheap due to businesses offloading them all the time.
I don’t think it has the power draw of an N100/N150-based machine (10-15W), but it’s not a huge power hungry server either.
I mean, 35W maximum is still incredibly low. At that point, you’re looking at a cost difference in the single-digits over the course of an entire year.
My little lab has 5 machines, 3 of which are tiny/mini/micro PCs. Total draw from my entire setup, including the t/m/m machines, is right around 100W. And since I started measuring it back in February, it’s used a total of 635 kWh. And most of that is from the
spinning rusthard drives. For reference, my whole household’s monthly usage averages around 1200 kWh.
