Backy

Backy is a tool for automating data backup and remote command execution. It can work over SSH, and provides completion and failure notifications, error reporting, and more.

Why the name Backy? Because I wanted an app for backups.

View the changelog here.

Tip

Feel free to open a PR, raise an issue(s), or request new feature(s).

Features

Allows easy configuration of executable commands
Allows for running package operations
Allows configuring failure, success, and final hooks
Allows for commands to be run on many hosts over SSH
Commands can be grouped in list to run in specific order
Notifications on completion and failure
Run in cron mode
For any command, especially backup commands

Getting started

If you have not installed Backy, see the install documentation.

If you need to configure it, see the config page.

Install Backy

Binaries are available from the release page. Make sure to get the correct version for your system, which supports x86_64, ARM64, and i386.

Source Install

You can install from source. You will need Go installed.

Then run:

go install git.andrewnw.xyz/CyberShell/backy@master

Once set, jump over to the config docs and start configuring your file.

Config File Definitions

Commands

The commands section is for defining commands. These can be run with or without a shell and on a host or locally.

See the commands documentation for further information.

commands:
  stop-docker-container:
    output: true # Optional and only when run in list and notifications are sent
    cmd: docker
    args:
      - compose
      - -f /some/path/to/docker-compose.yaml
      - down
    # if host is not defined, cmd will be run locally
    host: some-host 
  backup-docker-container-script:
    cmd: /path/to/script
    # The host has to be defined in the config file
    host: some-host
    environment:
      - FOO=BAR
      - APP=$VAR # defined in .env file in config directory
  shell-cmd:
    cmd: rsync
    shell: bash
    args:
      - -av
      - some-host:/path/to/data 
      - ~/Docker/Backups/docker-data
  script:
    type: scriptFile # run a local script on a remote host
    cmd: path/to/your/script.sh
    host: some-host  
  hostname:
    cmd: hostname

Lists

To execute groups of commands in sequence, use a list configuration.

cmdLists:
  cmds-to-run: # this can be any name you want
    # all commands have to be defined in the commands section
    order:
      - stop-docker-container
      - backup-docker-container-script
      - shell-cmd
      - hostname
    getOutput: true # Optional and only for when notifications are sent
    notifications:
      - matrix
    name: backup-some-server
  hostname:
    name: hostname
    order:
      - hostname
    notifications:
      - prod-email

Hosts

The hosts object may or may not be defined.

Info

If any host from a commands object does not match any host object, the needed values will be checked in the default SSH config files.

hosts:
  # any needed ssh_config(5) keys/values not listed here will be looked up in the config file or the default config file
  some-host:
    hostname: some-hostname
    config: ~/.ssh/config
    user: user
    privatekeypath: /path/to/private/key
    port: 22
    # can also be env:VAR or the password itself
    password: file:/path/to/file
    # can also be env:VAR or the password itself
    privatekeypassword: file:/path/to/file
    # only one is supported for now
    proxyjump: some-proxy-host

Notifications

The notifications object can have two forms.

For more, see the notification object documentation. The top-level map key is id that has to be referenced by the cmdLists key notifications.

notifications:
  mail:
    prod-email:
      id: prod-email
      type: mail
      host: yourhost.tld
      port: 587
      senderAddress: email@domain.tld
      to:
        - admin@domain.tld
      username: smtp-username@domain.tld
      password: your-password-here
  matrix:
    matrix:
      id: matrix
      type: matrix
      home-server: your-home-server.tld
      room-id: room-id
      access-token: your-access-token
      user-id: your-user-id

Logging

cmd-std-out controls whether commands output is echoed to StdOut.

If logfile is not defined, the log file will be written to the config directory in the file backy.log.

console-disabled controls whether the logging messages are echoed to StdOut. Default is false.

verbose basically does nothing as all necessary info is already output.

logging:
  verbose: false
  file: path/to/log/file.log
  console-disabled: false
  cmd-std-out: false

Vault

Vault can be used to get some configuration values and ENV variables securely.

vault:
  token: hvs.tXqcASvTP8wg92f7riyvGyuf
  address: http://127.0.0.1:8200
  enabled: false
  keys:
    - name: mongourl
      mountpath: secret
      path: mongo/url
      type:  # KVv1 or KVv2
    - name:
      path:
      type:
      mountpath:

Configuring Backy

This is the section on the config file.

To use a specific file: backy [command] -f /path/to/file

You can also use a remote file:

backy [command] -f `s3/http source`

See remote resources docs for specific info.

If you leave the config path blank, the following paths will be searched in order:

./backy.yml
./backy.yaml
The same two files above contained in a backy subdirectory under in what is returned by Go’s os package function UserConfigDir().

UserConfigDir() documentation:

Up-to date documentation for this function may be found on GoDoc.

UserConfigDir returns the default root directory to use for user-specific configuration data. Users should create their own application-specific subdirectory within this one and use that.

On Unix systems, it returns $XDG_CONFIG_HOME as specified by https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html if non-empty, else $HOME/.config. On Darwin, it returns $HOME/Library/Application Support. On Windows, it returns %AppData%. On Plan 9, it returns $home/lib.

If the location cannot be determined (for example, $HOME is not defined), then it will return an error.

See the rest of the documentation, titles included below, in this section to configure it.

Commands
Commands are just that, commands
Command Lists
This page tells you how to use command lists.
Hosts
This page tells you how to use hosts.
Remote resources
This is dedicated to configuring remote resources.
Notifications
This page tells you how to get set up Backy notifications.
Vault
Set up and configure vault.

Commands

Example Config

commands:
  stop-docker-container:
    cmd: docker
    Args:
      - compose
      - -f /some/path/to/docker-compose.yaml
      - down
    # if host is not defined, command will be run locally
    # The host has to be defined in either the config file or the SSH Config files
    host: some-host
    hooks
      error:
        - some-other-command-when-failing
      success:
        - success-command
      final:
        - final-command
  backup-docker-container-script:
    cmd: /path/to/local/script
    # script file is input as stdin to SSH
    type: scriptFile # also can be script
    environment:
      - FOO=BAR
      - APP=$VAR

Values available for this section (case-sensitive):

name	notes	type	required
`cmd`	Defines the command to execute	`string`	yes
`Args`	Defines the arguments to the command	`[]string`	no
`environment`	Defines environment variables for the command	`[]string`	no
`type`	See documentation further down the page. Additional fields may be required.	`string`	no
`getOutput`	Command(s) output is in the notification(s)	`bool`	no
`host`	If not specified, the command will execute locally.	`string`	no
`scriptEnvFile`	When type is `scriptFile` or `script`, this file is prepended to the input.	`string`	no
`shell`	Run the command in the shell	`string`	no
`hooks`	Hooks are used at the end of the individual command. Must have at least `error`, `success`, or `final`.	`map[string][]string`	no

cmd

cmd must be a valid command or script to execute.

Args

args must be arguments to cmd as they would be passed on the command-line:

cmd [arg1 arg2 ...]

Define them in an array:

Args:
  - arg1
  - arg2
  - arg3

getOutput

Get command output when a notification is sent.

Is not required. Can be true or false.

host

Info

If any host is not defined or left blank, the command will run on the local machine.

Host may or may not be defined in the hosts section.

Info

If any host from the commands section does not match any object in the hosts section, the Host is assumed to be this value. This value will be used to search in the default SSH config files.

For example, say that I have a host defined in my SSH config with the Host defined as web-prod. If I assign a value to host as host: web-prod and don’t specify this value in the hosts object, web-prod will be used as the Host in searching the SSH config files.

shell

If shell is defined, the command will run in the specified shell. Make sure to escape any shell input.

scriptEnvFile

Path to a file.

When type is script or scriptFile , the script is appended to this file.

This is useful for specifying environment variables or other things so they don’t have to be included in the script.

type

The following options are available:

name	description
script	`cmd` is used as the script
scriptFile	Can only be run on a host. `cmd` is read and used as the script, and `scriptEnvFile` can be used to add env variables
package	Run package operations. See dedicated page for configuring package commands
user	Run user operations. See dedicated page for configuring package commands

environment

The environment variables support expansion:

using escaped values $VAR or ${VAR}

For now, the variables have to be defined in an .env file in the same directory that the program is run from.

If using it with host specified, the SSH server has to be configured to accept those env variables.

If the command is run locally, the OS’s environment is added.

hooks

Hooks are run after the command is run.

Errors are run if the command errors, success if it returns no error. Final hooks are run regardless of error condition.

Values for hooks are as follows:

command:
  hook:
    # these commands are defined elsewhere in the file
    error:
      - errcommand
    success:
      - successcommand
    final:
      - donecommand

packages

See the dedicated page for package configuration.

Packages

This is dedicated to package commands. The command type field must be package. Package is a type that allows one to perform package operations. There are several additional options available when type is package:

name	notes	type	required
`packageName`	The name of a package to be modified.	`string`	yes
`packageManager`	The name of the package manger to be used.	`string`	yes
`packageOperation`	The type of operation to perform.	`string`	yes
`packageVersion`	The version of a package.	`string`	no

example

The following is an example of a package command:

 update-docker:
    type: package
    shell: zsh
    packageName: docker-ce
    packageManager: apt
    packageOperation: install
    host: debian-based-host

packageOperation

The following package operations are supported:

install
remove
upgrade
checkVersion

packageManager

The following package managers are recognized:

apt
yum
dnf

package command args

You can add additional arguments using the standard Args key. This is useful for adding more packages, yet it does not work with checkVersion.

Development

The PackageManager interface provides an easy way to enforce functions and options. There are two interfaces, PackageManager and ConfigurablePackageManager in the directory pkg/pkgman. Go’s import-cycle “feature” caused me to implement functional options using a third interface. PackageManagerOptionis a function that takes an interface.

PackageManager

// PackageManager is an interface used to define common package commands. This shall be implemented by every package.
type PackageManager interface {
	Install(pkg, version string, args []string) (string, []string)
	Remove(pkg string, args []string) (string, []string)
	Upgrade(pkg, version string) (string, []string) // Upgrade a specific package
	UpgradeAll() (string, []string)

	// Configure applies functional options to customize the package manager.
	Configure(options ...pkgcommon.PackageManagerOption)
}

There are a few functional options that should be implemented using the ConfigurablePackageManager interface:

// ConfigurablePackageManager defines methods for setting configuration options.
type ConfigurablePackageManager interface {
	SetUseAuth(useAuth bool)
	SetAuthCommand(authCommand string)
}

User commands

This is dedicated to user commands. The command type field must be user. User is a type that allows one to perform user operations. There are several additional options available when type is user:

name	notes	type	required
`userName`	The name of a user to be configured.	`string`	yes
`userOperation`	The type of operation to perform.	`string`	yes
`userID`	The user ID to use.	`string`	yes
`userGroups`	The groups the user should be added to.	`[]string`	yes
`userShell`	The shell for the user.	`string`	yes
`userHome`	The user’s home directory.	`string`	no

example

The following is an example of a package command:

  addUser:
	name: add user backy with custom home dir
    type: user
    userName: backy
    userHome: /opt/backy
    userOperation: add
    host: some-host

userOperation

The following package operations are supported:

add
remove
modify
password
checkIfExists

Development

The UserManager interface provides an way easy to add new commands. There is one interface Usermanager in directory pkg/usermanager.

UserManager

// UserManager defines the interface for user management operations.
// All functions but one return a string for the command and any args.
type UserManager interface {
	AddUser(username, homeDir, shell string, isSystem bool, groups, args []string) (string, []string)
	RemoveUser(username string) (string, []string)
	ModifyUser(username, homeDir, shell string, groups []string) (string, []string)
	// Modify password uses chpasswd for Linux systems to build the command to change the password
	// Should return a password as the last argument
	// TODO: refactor when adding more systems instead of Linux
	ModifyPassword(username, password string) (string, *strings.Reader, string)
	UserExists(username string) (string, []string)
}

Command Lists

Command lists are for executing commands in sequence and getting notifications from them.

The top-level object key can be anything you want but not the same as another.

Lists can go in a separate file. Command lists should be in a separate file if:

key ‘cmdLists.file’ is specified
lists.yml or lists.yaml is found in the same directory as the backy config file (this includes remote config files as of v0.7.0)

Info

The lists file is also checked in remote resources.

The lists file is ignored under the following condition:

If a remote config file is specified (on the command-line using -f) and the lists file is not found in the same directory, the lists file is assumed to not exist.

1
2
3
4
5
6
7
8
9


  test2:
    name: test2
    order:
      - test
      - test2
    notifications:
      - mail.prod-email
      - matrix.sysadmin
    cron: "0 * * * * *"

key	description	type	required
`order`	Defines the sequence of commands to execute	`[]string`	yes
`getOutput`	Command(s) output is in the notification(s)	`bool`	no
`notifications`	The notification service(s) and ID(s) to use on success and failure. Must be `service.id`. See the notifications documentation page for more	`[]string`	no
`name`	Optional name of the list	`string`	no
`cron`	Time at which to schedule the list. Only has affect when cron subcommand is run.	`string`	no

Order

The order is an array of commands to execute in order. Each command must be defined.

order:
  - cmd-1
  - cmd-2

getOutput

Get command output when a notification is sent.

Is not required. Can be true or false. Default is false.

Notifications

An array of notification IDs to use on success and failure. Must match any of the notifications object map keys.

Name

Name is optional. If name is not defined, name will be the object’s map key.

Cron mode

Backy also has a cron mode, so one can run backy cron and start a process that schedules jobs to run at times defined in the configuration file.

Adding cron: 0 0 1 * * * to a cmdLists object will schedule the list at 1 in the morning. See https://crontab.guru/ for reference.

Tip

Note: Backy uses the second field of cron, so add anything except * to the beginning of a regular cron expression.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


cmdLists:
  docker-container-backup: # this can be any name you want
    # all commands have to be defined
    order:
      - stop-docker-container
      - backup-docker-container-script
      - shell-cmd
      - hostname
      - start-docker-container
    notifications:
      - matrix.id
    name: backup-some-container
    cron: "0 0 1 * * *"
  hostname:
    name: hostname
    order:
      - hostname
    notifications:
      - mail.prod-email

Hosts

Key	Description	Type	Required
`OS`	Operating system of the host (used for package commands)	`string`	no
`config`	Path to the SSH config file	`string`	no
`host`	Specifies the `Host` ssh_config(5) directive	`string`	yes
`hostname`	Hostname of the host	`string`	no
`knownhostsfile`	Path to the known hosts file	`string`	no
`port`	Port number to connect to	`uint16`	no
`proxyjump`	Proxy jump hosts, comma-separated	`string`	no
`password`	Password for SSH authentication	`string`	no
`privatekeypath`	Path to the private key file	`string`	no
`privatekeypassword`	Password for the private key file	`string`	no
`user`	Username for SSH authentication	`string`	no

exec host subcommand

Backy has a subcommand exec host. This subcommand takes the flags of -m host1 -m host2. For now these hosts need to be defined in the config file.

Remote resources

Remote resources can be used for a lot of things, including config files and scripts.

Config file

For the main config file to be fetched remotely, pass the URL using -f [url].

If using S3, you should use the s3 protocol URI: s3://bucketName/key/path. You will also need to set the env variable S3_ENDPOINT to the appropriate value. The flag --s3-endpoint can be used to override this value or to set this value, if not already set.

Scripts

Scripts will be coming later.

Notifications

Notifications are only configurable for command lists, as of right now.

Notifications can be sent on command list completion and failure.

The supported platforms for notifications are email (SMTP) and Matrix.

Notifications are defined by service, with the current form following below. Ids must come after the service.

notifications:
  mail:
    prod-email:
      host: yourhost.tld
      port: 587
      senderaddress: email@domain.tld
      to:
        - admin@domain.tld
      username: smtp-username@domain.tld
      password: your-password-here

  matrix:
    matrix:
      home-server: your-home-server.tld
      room-id: room-id
      access-token: your-access-token
      user-id: your-user-id

Sections recognized are mail and matrix

There must be a section with an id (eg. mail.test-svr) following one of these sections.

mail

key	description	type
`host`	Specifies the SMTP host to connect to	`string`
`port`	Specifies the SMTP port	`uint16`
`senderaddress`	Address from which to send mail	`string`
`to`	Recipients to send emails to	`[]string`
`username`	SMTP username	`string`
`password`	SMTP password	`string`

matrix

key	description	type
`home-server`	Specifies the Matrix server connect to	`string`
`room-id`	Specifies the room ID of the room to send messages to	`string`
`access-token`	Matrix access token	`string`
`user-id`	Matrix user ID	`string`

To get your access token (assumes you are using Element) :

Log in to the account you want to get the access token for. Click on the name in the top left corner, then “Settings”.
Click the “Help & About” tab (left side of the dialog).
Scroll to the bottom and click on <click to reveal> part of Access Token.
Copy your access token to a safe place.

To get the room ID:

On Element or a similar client, navigate to the room.
Navigate to the settings from the top menu.
Click on Advanced, the room ID is there.

Info

Make sure to quote the room ID, as YAML spec defines tags using !.

Vault

Vault is a tool for storing secrets and other data securely.

Vault config can be used by prefixing vault: in front of a password or ENV var.

This is the object in the config file:

vault:
  token: hvs.tXqcASvTP8wg92f7riyvGyuf
  address: http://127.0.0.1:8200
  enabled: false
  keys:
    - name: mongourl
      mountpath: secret
      path: mongo/url
      type:  # KVv1 or KVv2
    - name:
      path:
      type:
      mountpath:

CLI

This page lists documentation for the CLI.

Backy

Backy is a command-line application useful for configuring backups, or any commands run in sequence.

Usage:
  backy [command]

Available Commands:
  backup      Runs commands defined in config file. Use -l flag multiple times to run multiple lists.
  completion  Generate the autocompletion script for the specified shell
  cron        Starts a scheduler that runs lists defined in config file.
  exec        Runs commands defined in config file in order given.
  help        Help about any command
  list        Lists commands, lists, or hosts defined in config file.
  version     Prints the version and exits

Flags:
  -f, --config string        config file to read from
  -h, --help                 help for backy
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

Use "backy [command] --help" for more information about a command.

Subcommands

backup

Backup executes commands defined in config file.
Use the --lists or -l flag to execute the specified lists. If not flag is not given, all lists will be executed.

Usage:
  backy backup [--lists=list1 --lists list2 ... | -l list1 -l list2 ...] [flags]

Flags:
  -h, --help                help for backup
  -l, --lists stringArray   Accepts comma-separated names of command lists to execute.

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

cron

Cron starts a scheduler that executes command lists at the time defined in config file.

Usage:
  backy cron [flags]

Flags:
  -h, --help   help for cron

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

exec

Exec executes commands defined in config file in order given.

Usage:
  backy exec command ... [flags]
  backy exec [command]

Available Commands:
  host        Runs command defined in config file on the hosts in order specified.

Flags:
  -h, --help   help for exec

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

Use "backy exec [command] --help" for more information about a command.

exec host

Host executes specified commands on the hosts defined in config file.
Use the --commands or -c flag to choose the commands.

Usage:
  backy exec host [--command=command1 --command=command2 ... | -c command1 -c command2 ...] [--hosts=host1 --hosts=hosts2 ... | -m host1 -m host2 ...]  [flags]

Flags:
  -c, --command stringArray   Accepts space-separated names of commands. Specify multiple times for multiple commands.
  -h, --help                  help for host
  -m, --hosts stringArray     Accepts space-separated names of hosts. Specify multiple times for multiple hosts.

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

version

Prints the version and exits. No arguments just prints the version number only.

Usage:
  backy version [flags]

Flags:
  -h, --help   help for version
  -n, --num    Output the version number only.
  -V, --vpre   Output the version with v prefixed.

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

list

Backup lists commands or groups defined in config file.
Use the --lists or -l flag to list the specified lists. If not flag is not given, all lists will be executed.

Usage:
  backy list [--list=list1,list2,... | -l list1, list2,...] [ -cmd cmd1 cmd2 cmd3...] [flags]

Flags:
  -c, --cmds strings    Accepts comma-separated names of commands to list.
  -h, --help            help for list
  -l, --lists strings   Accepts comma-separated names of command lists to list.

Global Flags:
  -f, --config string        config file to read from
      --log-file string      log file to write to
      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
  -v, --verbose              Sets verbose level

Exec

The exec subcommand can do some things that the configuration file can’t do yet. The command exec host can execute commands on many hosts.

exec host takes the following arguments:

  -c, --commands strings   Accepts comma-separated names of commands.
  -h, --help               help for host
  -m, --hosts strings      Accepts comma-separated names of hosts.

The commands have to be defined in the config file. The hosts need to at least be in the ssh_config(5) file.

backy exec host [--commands=command1,command2, ... | -c command1,command2, ...] [--hosts=host1,hosts2, ... | -m host1,host2, ...]  [flags]

Repositories

The repo mirrors are:

Backy

Features

Subsections of Backy

Getting started

Subsections of Getting started

Install Backy

Source Install

Config File Definitions

Commands

Lists

Hosts

Notifications

Logging

Vault

Configuring Backy

Subsections of Configuring Backy

Commands

Example Config

cmd

Args

getOutput

host

shell

scriptEnvFile

type

environment

hooks

packages

Subsections of Commands

Packages

example

packageOperation

packageManager

package command args

Development

PackageManager

User commands

example

userOperation

Development

UserManager

Command Lists

Order

getOutput

Notifications

Name

Cron mode

Hosts

exec host subcommand

Remote resources

Config file

Scripts

Notifications

mail

matrix

Vault

CLI

Backy

Subcommands

backup

cron

exec

exec host

version

list

Subsections of CLI

Exec

Repositories