multi: add migrate-db command

[guggero]

This is work in progress and NOT recommended to be used with production/mainnet data!

[kiwiidb]

Using this code I was getting an error:
Runtime error: db connection set not initialized
I started looking around and found that you were probably missing this: https://github.com/lightningnetwork/lnd/blob/master/lncfg/db.go#L140

So I added a quick patch: getAlby@0ffcae3
(should be a config param probably).

And then I was able to do the migration. The last logs were:

2022-07-14 10:41:42.534 LNDINIT: Opened destination DB
2022-07-14 10:41:42.535 LNDINIT: Checking tombstone marker on source DB
2022-07-14 10:41:42.536 LNDINIT: Checking DB version of source DB
2022-07-14 10:41:42.537 LNDINIT: Checking if migration was already applied to target DB
2022-07-14 10:41:42.540 LNDINIT: Starting the migration to the target backend
2022-07-14 10:41:42.540 LNDINIT: Copying top-level bucket 'waddrmgr'
2022-07-14 10:41:48.981 LNDINIT: Committing bucket 'waddrmgr'
2022-07-14 10:41:48.983 LNDINIT: Copying top-level bucket 'wtxmgr'
2022-07-14 10:41:49.016 LNDINIT: Committing bucket 'wtxmgr'
2022-07-14 10:41:49.017 LNDINIT: Creating 'wallet created' marker
2022-07-14 10:41:49.024 LNDINIT: Committing 'wallet created' marker

It seems everything has migrated succesfully, so I would say that a log line indicating this would also be useful.

[twofaktor]

Hello, what happened with this PR? Have been it abandoned? I think this is very useful to migrate an existing bbolt to postgres and following this guide

Thanks!

[ziggie1984]

This is an important PR we need to focus on to allow nodes to finally migrate from kv to native sql. cc @saubyk

[Roasbeef]

sqlite should be added here: https://github.com/lightninglabs/lndinit/pull/21/files#diff-00eb92ba2060dddcdbce2dba1dc551065557b5ebcf98e47e70946e1b50ffd243R453

The top-level bucket structure might also have changed somewhat, so we should update that aspect.

One other thing is we'll need to figure out a way to ensure that all the data has truly been migrated (eg: we don't forget some other top level bucket recently added).

[saubyk]

Still testing, but wanted to note some comments

[saubyk]

shouldn't the link be: https://github.com/lightningnetwork/lnd/blob/master/docs/postgres.md?

[saubyk]

Should be a WARNING/CAUTION instead of NOTE

[saubyk]

Should end with a section advising user to run the migrated node with the updated db config settings

[ziggie1984]

One other thing is we'll need to figure out a way to ensure that all the data has truly been migrated (eg: we don't forget some other top level bucket recently added).

so you mean having the db files constant here (channeldb, towerdb etc) is not enough but we need to also make sure every new bucket is in the new db ? Hmm not sure I understand this one? Because as long as we iterate through each bucket in the db file, there shouldn't be a problem that we miss a bucket ?

[ziggie1984]

Migration now supports sqlite as well. Keep in mind this is still an experimental feature because we need to implement some consistency checks.

This PR still needs some work:

1. Need to do the migration in batches, because memory issues can be a problem
2. Add consistency checks: Base level: Compare the two dbs and make sure every key-value pair was successfully copied
   Do some structural data tests for example compare the Channels from the old db and the new migrated db.
3. This only works if you run LND 18.3 otherwise we do not allow the migration.
4. Add a wtclient.db check so that we only migrate if the specific version is met.

Example for the sqlite migration:

 lndinit -v migrate-db  --network regtest --source.bolt.data-dir /LND-DIR/data   --dest.backend sqlite  --dest.sqlite.data-dir /LND-DIR/data

make sure the data-dir for the sqlite directory is the same as the boltdb otherwise you need to copy the new files into the other directory when starting lnd.

for postgress:

 lndinit -v migrate-db  --network regtest --source.bolt.data-dir /LND-DIR/data  --dest.backend postgres  --dest.postgres.dsn="postgres://lnd_migration:lnd_migration@localhost:5432/lnd_migration_db?sslmode=disable"

make sure you create a db in postgres before otherwise the opening of the db will fail.

Also make sure you choose the correct network, otherwise the bolt.db will not be found in the LND dir.