Skip to content

Latest commit

 

History

History
519 lines (403 loc) · 15.9 KB

README.md

File metadata and controls

519 lines (403 loc) · 15.9 KB
Raw

brudi

When it comes to backup-creation there are several solutions to use.
In general everybody's doing some sort of dump or tar and backing up the results incremental with restic or similar programs.

This is why brudi was born. brudi supports several backup-methods and is configurable by a simple yaml file. The advantage of brudi is, that you can create a backup of a source of your choice and save it with restic afterwards in one step. Under the hood, brudi uses the given binaries like mysqldump, mongodump, pg_dump, tar or restic.

Using brudi will save you from finding yourself writing bash-scripts to create your backups.

Besides creating backups, brudi can also be used to restore your data from backup in an emergency.

Table of contents

Usage

CLI

In order to use the brudi-binary on your local machine or a remote server of your choice, ensure you have the required tools installed.

  • mongodump (required when running brudi mongodump)
  • mysqldump (required when running brudi mysqldump)
  • tar (required when running brudi tar)
  • redis-cli (required when running brudi redisdump)
  • restic (required when running brudi --restic)
$ brudi --help

Easy, incremental and encrypted backup creation for different backends (file, mongoDB, mysql, etc.)
After creating your desired tar- or dump-file, brudi backs up the result with restic - if you want to

Usage:
  brudi [command]

Available Commands:
  help           Help about any command
  mongodump      Creates a mongodump of your desired server
  mongorestore   Restores a server from a mongodump
  mysqldump      Creates a mysqldump of your desired server
  mysqlrestore   Restores a database from an sqldump
  pgdump         Creates a pg_dump of your desired postgresql-server
  pgrestore      Restores a database from a pgdump using pg_restore
  psql           Restores a database from a plain-text pgdump using psql
  redisdump      Creates an rdb dump of your desired server
  tar            Creates a tar archive of your desired 
  tarrestore     Restores files from a tar archive
  version        Print the version number of brudi

Flags:
      --cleanup         cleanup backup files afterwards
  -c, --config string   config file (default is ${HOME}/.brudi.yaml)
  -h, --help            help for brudi
      --restic          backup result with 'restic backup'
      --restic-forget   executes 'restic forget' after backing up things with restic
      --version         version for brudi

Use "brudi [command] --help" for more information about a command.

Docker

In case you don't want to install additional tools, you can also use brudi inside docker:

docker run --rm -v ${HOME}/.brudi.yml:/home/brudi/.brudi.yml quay.io/mittwald/brudi mongodump --restic --cleanup

The docker-image comes with all required binaries.

Configuration

As already mentioned, brudi is configured via .yaml. The default path for this file is ${HOME}/.brudi.yaml, but it's adjustable via -c or --config. The config file itself can include environment-variables via go-template:

restic:
  global:
    flags:
      repo: "{{ .Env.RESTIC_REPOSITORY }}"

Since the configuration provided by the .yaml-file is mapped to the corresponding CLI-flags, you can adjust literally every parameter of your source backup.
Therefore you can simply refer to the official documentation for explanations on the available flags:

Every source has a an additionalArgs-key which's value is an array of strings. The value of this key is appended to the command, generated by brudi. Even though brudi should support all cli-flags to be configured via the .yaml-file, there may be flags which are not.
In this case, use the additionalArgs-key.

It is also possible to provide more than one configuration file, for example -c mongodump.yaml -c restic.yaml. These configs get merged at runtime. If available, the default config will always be laoded first and then overwritten with any values from user-specified files. In case the same config file has been provided more than once, only the first instance will be taken into account.

Sources

Tar
tar:
  options:
    flags:
      create: true
      gzip: true
      file: /tmp/test.tar.gz
    additionalArgs: []
    paths:
      - /tmp/testfile
  hostName: autoGeneratedIfEmpty

Running: brudi tar -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
tar -c -z -f /tmp/test.tar.gz /tmp/testfile

All available flags to be set in the .yaml-configuration can be found here.

MySQLDump
mysqldump:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      opt: true
      allDatabases: true
      resultFile: /tmp/test.sqldump
    additionalArgs: []

Running: brudi mysqldump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
mysqldump --all-databases --host=127.0.0.1 --opt --password=mysqlroot --port=3306 --result-file=/tmp/test.sqldump --user=root

All available flags to be set in the .yaml-configuration can be found here.

MongoDump
mongodump:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []

Running: brudi mongodump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
mongodump --host=127.0.0.1 --port=27017 --username=root --password=mongodbroot --gzip --archive=/tmp/dump.tar.gz

All available flags to be set in the .yaml-configuration can be found here.

PgDump
pgdump:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      password: postgresroot
      username: postgresuser
      dbName: postgres
      file: /tmp/postgres.dump
    additionalArgs: []

Running: brudi pgdump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
pg_dump --file=/tmp/postgres.dump --dbname=postgres --host=127.0.0.1 --port=5432 --username=postgresuser

All available flags to be set in the .yaml-configuration can be found here.

Limitations

Unfortunately PostgreSQL is very strict when it comes to version-compatibility.
Therefore your pg_dump-binary requires the exact same version your PostgreSQL-server is running.

The Docker-image of brudi always has the latest version available for the corresponding alpine-version installed.

Redis
redisdump:
  options:
    flags:
      host: 127.0.0.1
      password: redisdb
      rdb: /tmp/redisdump.rdb
    additionalArgs: []

Running: brudi redisdump -c ${HOME}/.brudi.yml

Becomes the following command: redis-cli -h 127.0.0.1 -a redisdb --rdb /tmp/redisdump.rdb bgsave

As redis-cli is not a dedicated backup tool but a client for redis, only a limited number of flags are available by default, as you can see here.

Restic

In case you're running your backup with the --restic-flag, you need to provide a valid configuration for restic.
You can either configure restic via brudis .yaml-configuration, or via the environment variables used by restic.

If you're already using restic in your environment, you should have everything set up perfectly to use brudi with --restic.

Forget

It's also possible to run restic forget-cmd after executing restic backup with brudi by using --restic-forget.
The forget-policy is defined in the configuration .yaml for brudi.

Example .yaml-configuration:

restic:
    global:
      flags:
        # you can provide the repository also via RESTIC_REPOSITORY
        repo: "s3:s3.eu-central-1.amazonaws.com/your.s3.bucket/myResticRepo"
    backup:
      flags:
        # in case there is no hostname given, the hostname from source backup is used
        hostname: "MyHost"
      # these paths are backuped additionally to your given source backup
      paths: []
  forget:
    flags:
      keepLast: 48
      keepHourly: 24
      keepDaily: 7
      keepWeekly: 2
      keepMonthly: 6
      keepYearly: 2
    ids: []

Sensitive data: Environment variables

In case you don't want to provide data directly in the .yaml-file, e.g. sensitive data like passwords, you can use environment-variables. Each key of the configuration is overwritable via environment-variables. Your variable must specify the whole path to a key, seperated by _.
For example, given this .yaml:

mongodump:
  options:
    flags:
      username: "" # we will override this by env
      password: "" # we will override this by env
      host: 127.0.0.1
      port: 27017
      gzip: true
      archive: /tmp/dump.tar.gz

Set your env's:

export MONGODUMP_OPTIONS_FLAGS_USERNAME="root"
export MONGODUMP_OPTIONS_FLAGS_PASSWORD="mongodbroot"

As soon as a variable for a key exists in your environment, the value of this environment-variable is used in favour of your .yaml-config.

Gzip support for binaries without native gzip support

The tools mysqldump, pg_dump and redis-cli don't natively support gzip. However, if the desired path for the backup file is suffixed with .gz, brudi will automatically gzip the backup after creation and delete the uncompressed backup file. For restoration, the file will be automatically uncompressed. Example for mysql:

mysqldump:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      opt: true
      allDatabases: true
      resultFile: /tmp/test.sqldump.gz
    additionalArgs: []
mysqlrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      Database: test
    additionalArgs: []
    sourceFile: /tmp/test.sqldump.gz

Restoring from backup

TarRestore
tarrestore:
  options:
    flags:
      extract: true
      gzip: true
      file: /tmp/test.tar.gz
      target: "/"
    additionalArgs: []
  hostName: autoGeneratedIfEmpty

Running: brudi tarrestore -c ${HOME}/.brudi.yml

Becomes the following command: tar -x -z -f /tmp/test.tar.gz -C /

MongoRestore
mongorestore:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []

Running: brudi mongorestore -c ${HOME}/.brudi.yml

Becomes the following command:
mongorestore --host=127.0.0.1 --port=27017 --username=root --password=mongodbroot --gzip --archive=/tmp/dump.tar.gz

All available flags to be set in the .yaml-configuration can be found here.

MySQLRestore
mysqlrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      Database: test
    additionalArgs: []
    sourceFile: /tmp/test.sqldump

Running: brudi mysqlrestore -c ${HOME}/.brudi.yml

Becomes the following command:
mysql --database=test --host=127.0.0.1 --password=mysqlroot --port=3306 --user=root < /tmp/test.sqldump

All available flags to be set in the .yaml-configuration can be found here.

PgRestore

Restoration for PostgreSQL databases is split into two commands, psql and pgrestore. Which one to use depends on the format of the dump created with pg_dump:

psql can be used to restore plain-text dumps, which is the default format.

pgrestore can be used if the format option of pg_dump was set to tar, directory or custom.

Restore using pg_restore
pgrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      username: postgresuser
      password: postgresroot
      dbname: postgres
    additionalArgs: []
    sourcefile: /tmp/postgres.dump

Running: brudi pgrestore -c ${HOME}/.brudi.yml

Becomes the following command:
pg_restore --host=127.0.0.1 --port=5432 --username=postgresuser --db-name=postgres /tmp/postgress.dump

This command has to be used if the format option was set to tar, directory or custom in pg_dump.

All available flags to be set in the .yaml-configuration can be found here.

Restore using psql
psql:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      user: postgresuser
      password: postgresroot
      dbname: postgres
    additionalArgs: []
    sourcefile: /tmp/postgres.dump

Running: brudi pgrestore -c ${HOME}/.brudi.yml

Becomes the following command: psql --host=127.0.0.1 --port=5432 --user=postgresuser --db-name=postgres < /tmp/postgress.dump

This command has to be used if the format option was set to plain in pg_dump, which is the default.

All available flags to be set in the .yaml-configuration can be found here.

Restoring using restic

Backups can be pulled from a restic repository and applied to your server by using the --restic flag in your brudi command. Example configuration for mongorestore:

mongorestore:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []
restic:
  global:
    flags:
      repo: "s3:s3.eu-central-1.amazonaws.com/your.s3.bucket/myResticRepo"
  restore:
    flags:
      target: "/"
    id: "latest"

This will pull the latest snapshot of /tmp/dump.tar.gz from the repository, which mongorestore then uses to restore the server. It is also possible to specify concrete snapshot-ids instead of latest.

Featurestate

Source backup methods

  • mysqldump
  • mongodump
  • tar
  • pg_dump
  • redisdump

Restore backup methods

  • mysqlrestore
  • mongorestore
  • tarrestore
  • pgrestore
  • redisrestore

Incremental backup of the source backups

  • restic
    • commands
      • restic backup
      • restic forget
      • restic restore
    • storage
      • s3