Backups, recovery in a HA environment

[Swallowtail23]

I have HA running as a VM on KVM/virtio. I have a solid and automated backup process for all of my IT environment - the VM gets backed up locally and to AWS S3 regularly/automatically, and HA is set to a full daily/weekly/monthly backup routine using one of the great add-ons available. I've tested recovery from both full VM file and add-on.

I've now added ZWave, with an Aeotec Z-Stick Gen5+, about 35 devices and ZWave 500 entities in HA, and am looking to see how to automate backups for simple recovery (if needed). I'm using ZWaveJS2MQTT as control panel only, MQTT gateway disabled.

Starting again with exclusion/reinclusion/renaming/automation-testing and -fixing/etc takes me about 2 days to do from scratch (having just done it to migrate from Vera).

I have no access to where ZWaveJS2MQTT keeps its "Store" except through the Control Panel, where I can manually download json files but not obviously recover them. "Backup" in advanced creates a useless (for recovery) nodes.json. I can (again manually) run an NVM backup (thanks @AlCalzone for your support on that in last 24h) but I am unclear on what that will recover if restored to bare metal / new install.

There is some very basic instructions on backup

Backing Up Settings

Settings, scenes and the Z-Wave configuration are stored in JSON files under the project store folder. It is a good idea to backup those files, which can be done by backing up the store or using the import/export buttons.

...but beyond reminding that it is a "good idea" that doesn't actually provide much useful info/details.

Suggestion:

• a Backup page on the wiki / help page from the add-on that tells what is critical for backup/recovery. I understand that my network is built by components in the Store and on the stick, so what is needed to backup/recover?
• scenarios for backup/restore - e.g. bare metal/new HA/new stick etc.
• instructions on how to backup Store, etc.
• ideally a Service Call available for HA to call which backs up the Store for simple offline/off-site backup - after all HA can kinda handle automation, right? :)
• a way to rebuild on a clean install - restore the Store/ restore NVM?

Restore should be easy, backups should be automatic.

I can get involved with testing, I'm technical but not a developer, I'm happy to get involved.

Thank you all for the amazing piece of work. I LOVE this add-on, and my ZWave is now MILES better and easier to manage than it was with Vera.

[robertsLando]

@Swallowtail23 So essentially you would like a new Backup/Restore section on docs to explain in details all the things you listed there?

BTW the important things to backup are settings.json scenes.json users.json nodes.json then you may want to do an NVM backup of the stick but that's only needed when and if you need to change your stick. Otherwise by backup those files you can install another z2m version on another machine, place those files in store foldler, start application with the same stick plugged in and everything would be the same

[AlCalzone]

Restore should be easy, backups should be automatic.

@robertsLando I think a good start would be to offer an easy way (button press) to download the z2m backup with everything that's needed. And a similarly simple way to restore the backup, e.g. after migrating to a different host.
Not documentation, but actual funcationality in the UI.

A smarter (automatic) way to handle NVM backups would also not be too terrible. Apparently some commercial gateways do that before each network management operation (adding/removing nodes, maybe some others too). So if something goes wrong here, you can always revert the stick back to the last working state. Sure, it always takes a couple of seconds to do that, but that's less pain than having to start over when you fuck up.

[Swallowtail23]

I've learned quite a bit over the last 24h on this subject from a thread on the HA forums, but it has reinforced that it is not intuitively clear (UI, UX, documentation) how to backup / restore zwjs2mqtt's state, and that at the moment to achieve what I need is a combination of HA functionality (add-on backup to capture the store) and zwjs2mqtt functionality (NVM). I know now that zwavejs2mqtt can run in other environments to HA, but for this discussion I'm working with HA so will talk in that context.

You make a comment @robertsLando "place those files in store folder" - I cannot directly do that in my HA environment, I don't have access to the folder. I can use HA capability to restore it yes... but: A key question: is zwavejs2mqtt wanting to 'stand alone' with regard to its ability to backup/restore the ZWave environment, regardless of its 'host'?

The end goal is restoration when required, yes? Working back from the end goal there are four scenarios I can see which need to be catered for:

1. New system, same stick (e.g. system migration, system failure, messed up system, etc.)
2. Same system, new stick (e.g. stick failure, replacement stick)
3. New system, new stick (complete start again - perhaps your box caught fire and destroyed the system AND the stick :-) )
4. Same system, same stick (you messed up, and want to revert to a working configuration)

After reading, learning and a lot of thought... what would make sense to me :) is if zwjs2mqtt included inbuilt restorability (i.e. manual backup/restore) for Store and NVM (as between them those cover the 4 scenarios) and (ideally/optionally) exposed that capability via api or to built-in host tools to whatever 'host' system is running to be able to automate backups.

[robertsLando]

IMO this could be handled easily by simply providing a backup/restore button on store UI, this will allow to download/upload a zip file. The NVM backup restore function is already present on advanced menu in control panel. Then some docs about this (explaining when you need to backup/restore one thing or the other like in your 4 examples).

Do you think this would be enought?

[Swallowtail23]

Yes I think a Store backup/restore button would be a great thing, whether on Store page or under Control Panel / Advanced where you do NVM and all the other advanced functions could be debated.
I agree NVM is covered. Agree on need for better doco for what/when/how.
That then means zwjs2mqtt has self-contained and intuitive recoverability.
Separate examples of how to automate Store backup dependent on your host (e.g. HA) could be provided as examples in docs.
Thank you @robertsLando.

[LordMike]

@robertsLando I on the other hand don't think the current state of the NVM backup is enough. As I wrote in my issue #2084, I was requesting automated backups of the NVM memory. I couldn't find an API call to begin the NVM backup, the UI seems to do it via. websockets.. I haven't checked if there is an MQTT topic I can publish to.

It turns out, that my current system is entirely vulnerable to loss/disaster, because I had entirely forgotten about the controller and its memory. If I had lost my stick, I would have been screwed.

As @AlCalzone suggests, some devices apparently do automatic backups on actions that modify the controllers config. This is a great suggestion - in my issue, I suggested something similar, but "later" - as I think the ~10-15s it takes for the backup to complete would be hindering to users.. I could also imagine something would break if the Z-wave network is non-functional (from a home automation point of view), immediately upon adding or removing devices... They may expect to be able to do a lot of querying or such.

So I strongly advocate for some form of automated backup, be it:

• Scheduled (daily, weekly, ..)
• Automatic, on changes (immediately)
• Automatic, after changes (f.ex. every night, if any changes have occurred)

If the binary blob is placed in the store directory, my regular backup can then pick it up and send it out.

[robertsLando]

Ok, based on this we could list some of the addition needed:

• Add backup/restore buttons in store UI. This will download the files in store needed for backup
• Add a setting to automatically backup NVM after each node operation (add/remove/replace)

About the first one, should a backup also trigger an NVM backup and download also that file in the zipped buckup?
And should a restore also do a nvm restore?

I would keep the NVM and store backup/restore two separete things but I accept suggestions

[Swallowtail23]

Ok, based on this we could list some of the addition needed:

• Add backup/restore buttons in store UI. This will download the files in store needed for backup
• Add a setting to automatically backup NVM after each node operation (add/remove/replace)

About the first one, should a backup also trigger an NVM backup and download also that file in the zipped buckup? And should a restore also do a nvm restore?

I would keep the NVM and store backup/restore two separete things but I accept suggestions

Personally I agree that NVM should be kept separate. Given that it should only be needed after a change it is not unreasonable to have it manual.

[christianreiss]

So,

Long time :)
How would I backup nvm manually in the meantime? :-)

[AlCalzone]

In zwavejs2mqtt, on the nodes table under Actions -> Advanced:

[nickwilkie]

Not sure if this is still being considered, but wanted to give my input supporting the ideas in this thread. I do weekly manual NVM backups, and today my Zooz Z-wave 700 stick got corrupted after a problematic SmartStart inclusion. Initially I tried to restore my last ProxMox VM backup, but it didn't help, only my last NVM download saved my ass. My whole house is Z-wave, and without the manual backups I would have basically needed to start from scratch in a house with 50+ Z-wave devices.

Having a way to automatically backup NVM with inclusion/exclusion/firmware updates, with HA backups, and best of all using an api, would be amazing.

[robertsLando]

@nickwilkie It's on my TODOs, will do it as soon as possible.

[nickwilkie]

Thanks so much! Appreciate the work you guys do on this project. My house is so crazy smart because of you guys.

[robertsLando]

Working on it: #2469

[robertsLando]

cc @nickwilkie @christianreiss @LordMike @Swallowtail23

[GrizzlyAK]

I came here to ask about this, because I just saw the Store Backup function in zw2m CP. Up to this point, I've assumed that doing a Snapshot in HA would back up everything, including integrations and their configuration data. Is this not the case? Do I need to be backing up zw2m configs regularly too, along with the NVM?

BTW, my backup settings are both turned off (default? These have been added since last I looked at options).

Also, do you guys know if the latest Aeotec Z-Stick 5+ can be soft-reset safely, or should I check with Aeotec?

... and ... it looks like you've already added the Update/Restore functionality! ;-)

[robertsLando]

I've assumed that doing a Snapshot in HA would back up everything, including integrations and their configuration data. Is this not the case?

It just backups data present on the container, so it covers the backup of the store dir (everything) but not the NVM, nvm backup must be triggered from z2m, it backups the controller non-volatile-memory (NVM) and stores it to store. In poor words you can keep the snapshot backup provided by hass on and just backup nvm :) Or keep also store backup on if you want to keep some old backup of it.

Also, do you guys know if the latest Aeotec Z-Stick 5+ can be soft-reset safely, or should I check with Aeotec?

Yes AFAIK, you must use the serial-by/id path BTW

[LordMike]

Slight thing. Doing a snapshot on the host has the (very slight) risk that you get a corrupt file which is being written to.

It’s very, very slight. But it’s there.

[GrizzlyAK]

Thanks guys!