MariaDB continuous integration infrastructure (MDBCI)

MDBCI is a set of tools for testing MariaDB components on the wide set of configurations. The main features of MDBCI are:

  • automatic creation of virtual machines according to the configuration template,
  • automatic and reliable deploy of MariaDB, Galera, MaxScale and other packages to the created virtual machines,
  • creation and management of virtual machine state snapshots,
  • reliable destruction of created virtual machines.

Read More information about MDBCI

MDBCI documentation is also available in the HTML format. Read more about building docs.

Team

  • Project leader: Sergey Balandin

  • Developers:

    • Timofey Turenko
    • Andrey Vasilyev
    • Danil Nazarov

  • Former Developers:

    • Alexander Kaluzhniy
    • Kirill Krinkin
    • Ilfat Kinyaev
    • Mark Zaslavskiy
    • Tatyana Berlenko
    • Kirill Yudenok
    • Maksim Kosterin
    • Roman Vlasov
    • Evgeny Vlasov
    • Dmitriy Karpovskiy
    • Grigoriy Kostryukov

Architecture and workflow

This section describes MDBCI architecture, workflow and other technical details.

Terminology

  • Box is a description of virtual machine image template. For vagrant provider the box have the same meaning; for AWS EC2 box is similar to image. Boxes described in file. Read more about providers and boxes.

  • MDBCI is a standard set of tools for testing MariaDB components on the wide set of configurations.

  • MariaDB is an enhanced, drop-in replacement for MySQL. It contains several set of components which can be used in standalone configurations and in cluster based heterogenous systems.

  • Node is a particular instance of virtual machine of its description.

  • Product is a description of the particular version of software which is being under control of MDBCI. Current version supports next products:

    • mariadb -- MariaDb server and client,
    • maxscale -- Maxscale server and client,
    • mysql -- Mysql server and client,
    • galera -- Galera server and clients,
    • xpand -- Xpand server. Read more,
    • mariadb_plugins -- Plugins for MariaDb. Read more,
    • mdbe_build -- Dependencies for MariaDb build,
    • connetors_build -- Dependencies for MariaDb connectors build,
    • kerberos -- Kerberos packages. Read more,
    • docker -- Docker packages.

    Full list of products.

  • Repo is a description of package repository with particular product version. Repositories are described in JSON format and stored in repo.d directory (see. repo.d files).

  • Template is a set of node definitions in JSON format. Templates are used for setup a set of virtual machines. Read more about template creation.

  • Configuration is a directory that contains a set of files that MDBCI uses to state of manage virtual machines.

Workflow

MDBCI Workflow includes the following steps:

  • Repository preparation.
  • Creating a template.
  • Configuration creation using a template and repository data.
  • Spinning up virtual machine(s) using configuration.
  • Using created virtual machine(s).
  • Destroying virtual machine(s).

Environmental variables

MDBCI_VM_PATH variable points to the directory for virtual machines definitions.

Creating configuration

MDBCI generates service files from template and stores them inside the configuration directory. Template example is available as instance.json. You can copy this file with another name and tailor configuration for your needs. It's possible to create multi-VM configurations.

Since new template is created you can generate configuration.

  ./mdbci --template vm_template.json generate NAME

In this example MDBCI will generate new configuration from vm_template.json template. It will be placed in NAME subdirectory.

NB Many configurations could be configured by MDBCI in subdirectories. Each configuration is autonomous.

Box, products, versions

MDBCI makes matching between boxes, target platforms, products and versions by lexicographical base. If we have a look at the output of next command

./mdbci show repos

we can see something like this:

galera@5.1+debian^squeeze => [http://mirror.netinch.com/pub/mariadb/repo/5.1/debian squeeze main]
galera@5.1+debian^jessie => [http://mirror.netinch.com/pub/mariadb/repo/5.1/debian jessie main]
galera@10.0.16+rhel^5 => [http://yum.mariadb.org/10.0.16/rhel5-amd64]

It means that each exact product/platform version combination is encoded.

product@version+platform^platform_version

In cases, when we need to use default product version on particular platform this encoding will be

mdbe@?+opensuse^13 => [http://downloads.mariadb.com/enterprise/WY99-BC52/mariadb-enterprise/5.5.42-pgo/opensuse/13]

where mdbe@? means default MariaDB community version on openSuse 13 target platform.

See also:

Installation

These instructions install the bare minimum that is required to run the MaxScale system test setup. This configuration requires about 10GB of memory to run.

Install MDBCI dependencies

MDBCI is a tool written in Ruby programming language. In order to ease the deployment of the tool the AppImage distribution is provided. It allows to use MDBCI as a standalone executable. FUSE should be installed on all linux distributions as it's required to execute AppImage file.

  • glibc >= 2.14
  • fuse
  • fuse-libs - additional fuse libraries for CentOS
  • libfuse2 - additional fuse libraries for Ubuntu and Debian

Debian/Ubuntu:

sudo apt-get install -y libfuse2 fuse

CentOS/RHEL:

sudo yum install -y fuse-libs fuse

You also may need to add current user to the fuse user group in case you are getting fuse: failed to open /dev/fuse: Permission denied error.

sudo addgroup fuse
sudo usermod -a -G fuse $(whoami)

Check Toubleshooting section for additional help.

Install MDBCI and configure MDBCI

  1. Download MDBCI:

    sudo wget https://mdbe-ci-repo.mariadb.net/MDBCI/mdbci -O /usr/local/bin/mdbci && sudo chmod +x /usr/local/bin/mdbci

  2. Run installing MDBCI dependencies: mdbci setup-dependencies

  3. Fill in the MDBCI configuration settings (for example, credentials of cloud platforms, private repositories, and etc.): mdbci configure

    You can follow the MDBCI configuration to read more about MDBCI configuration.

  4. Log out and back in again. This needs to be done in order for the new groups to become active.

Generate Configuration and Start VMs

Most products to be installed on the machines require additional information to be stored on the MDBCI host. Run the following command to generate product repositories configuration files.

./mdbci generate-product-repositories

See generate-product-repositories command documentation for details.

To get configuration examples, use command:

mdbci deploy-examples

The following command will place confs and scripts directories into the current working directory.

In order to generate the configuration out of the sample template and create VMs run:

mdbci generate -t confs/libvirt.json my-setup
mdbci up my-setup

Once the last command finishes, you should have a working set of VMs in the my-setup subfolder.

See also:

Configuration

Configuration files are located in ~/.config/mdbci/ or in config directory of the project.

Full list of configuration files:

config.yaml

Main MDBCI configuration file. Includes cloud platforms credentials, some repository and subscription parameters. More about config.yaml

boxes

boxes directory contains descriptions of the virtual machine images and necessary requirements for their creation. More about boxes

clustrix_license

clustrix_license file describes the Xpand configuration. File format:

set global license=
'{"expiration":"TIME",
"maxnodes":"NUMBER",
"company":"COMPANY NAME",
"maxcores":"NUMBER",
"email":"EMAIL",
"person":"NAME",
"signature":"KEY"}'

Xpand product in MDBCI.

hidden-instances.yaml

hidden-instances.yaml file describes the information about hidden instances that will not be shown as a result of the list_cloud_instances command. File format:

---
gcp:
  - gcp-name
aws:
  - aws-name

Read more about list_cloud_instances.

repo.d

Repositories for products described in json files. More about repo.d

required-network-resources.yaml

required-network-resources.yaml file describes the list of network resources which MDBCI must check before configure a virtual machine. File format:

---
- https://github.com

windows.pem

windows.pem file describes the RSA key for ssh connection to the Windows machine. Read more about Windows machine creation.

config.yaml

Configuration is required for some MDBCI commands.

Interactive step-by-step configuration

./mdbci configure

You can use the ./mdbci configure command to configure all products, or you can use the ./mdbci configure --product <product-name> command to configure a specific product.

Manual configuration

To configure MDBCI manually, create a config.yaml file in the ~/.config/mdbci folder and fill in the necessary settings based on the file template below:

---
aws:
  authorization_type: web-identity # AWS authorization type ('web-identity' or 'standard')
  access_key_id: # only for 'standard' authorization type
  secret_access_key: # only for 'standard' authorization type
  role_arn: # AWS role ARN (only for 'web-identity' authorization type)
  region: eu-west-1
  availability_zone: eu-west-1a
  use_existing_vpc: true
  vpc_id: # id of the existing vpc
  subnet_id: # id of the existing vpc subnet
docker:
  username: # username
  password: # password
  ci-server: https://maxscale-docker-registry.mariadb.net
rhel:
  username: # username
  password: # password
mdbe:
  key: # key
gcp:
  credentials_file: # path to credentials gcp file. Can be downloaded from https://console.cloud.google.com/apis/credentials
  project: # project id
  default_region: # the region used when the CPU quota is met
  regions: # list of supported regions
  - us-central1
  - europe-west4
  use_existing_network: true
  network: # name of the existing vpc
  tags:
  - allow-all-traffic
  use_only_private_ip: false
digitalocean:
  region: FRA1
  token: # token
suse:
  email: # Customer's email
  key: # Subscription registration code
  registration_proxy: # SUSE Registration Proxy Server address
mdbe_ci:
  mdbe_ci_repo:
    username: # username
    password: # password
  es_repo:
    username: # username
    password: # password
force: # true or false

AWS authorization types

Currently MDBCI supports 2 authorization types:

  • standard is based on AWS credentials specified in the config.yaml file (access_key_id and secret_access_key).
  • web-identity is based on a Web Identity Token retrieved from GCloud Auth (via gcloud auth print-identity-token command) and the AWS Role ARN, that should be specified in the config.yaml.

Standard configuration example

aws:
  authorization_type: standard
  access_key_id: AKIAIOSFODNN7EXAMPLE
  secret_access_key: wJalrXUtnFEMI/K7MDENG/qwertyuiopEXAMPLE
  ...

Web Identity AWS configuration example

aws:
  authorization_type: web-identity
  role_arn: arn:aws:iam::012345678910:role/buildbot_aws
  ...

See also

How to add another region support for GCP?

Add the region name to regions list in GCP section. Read more about GCP regions management in MDBCI

How to use existing VPC for AWS and GCP?

Read more about how to create a new VPC for AWS or GCP in the corresponding README files - AWS VPC and GCP VPC.

Set the use_existing_vpc setting to true and fill vpc_id and subnet_id fields in the MDBCI configuration for AWS, and Set the use_existing_network setting to true and fill network and tags fields for GCP.

Boxes

A box describes the image and necessary requirements for VM creation. They are described in json files. Each file can contain one or more box descriptions. Descriptions are usually grouped by the provider and the processor architecture. Box description files are located in ~/.config/mdbci/boxes folder or in config/boxes directory of the project.

File format:

{
  "debian_bullseye_gcp": {
    "provider": "gcp",
    "architecture": "amd64",
    "image": "debian-cloud/debian-11",
    "platform": "debian",
    "platform_version": "bullseye",
    "default_machine_type": "g1-small",
    "default_cpu_count": "1",
    "default_memory_size": "1024",
    "supported_instance_types": ["a2-highgpu-1g", "a2-highgpu-2g"]
  },

  "debian_bullseye_aws" : {
    "provider": "aws",
    "architecture": "amd64",
    "ami": "ami-05b99bc50bd882a41",
    "user": "admin",
    "default_machine_type": "t3.medium",
    "default_cpu_count": "2",
    "default_memory_size": "4096",
    "platform": "debian",
    "platform_version": "bullseye",
    "vpc": "true",
    "supported_instance_types": ["t2.nano","t2.micro"]
  },

  "debian_bullseye_libvirt": {
    "provider": "libvirt",
    "architecture": "amd64",
    "box": "generic/debian11",
    "platform": "debian",
    "platform_version": "bullseye"
  }
}

Each box name usually consists of distribution name, version and box provider

Common parameters

  • provider - box provider (aws, gcp, libvirt, digitalocean, docker)
  • architecture - processor arcitecture { amd64 | aarch64 }
  • platform - distribution name
  • platfrom_version - distribution version
  • skip_configuration - boolean flag. If set, the machine will not be configured using Chef. None of products will be installed.

Common cloud boxes parameters:

  • default_memory_size - node RAM size
  • default_cpu_count - node number of processors
  • default_machine_type - node machine type
  • supported_instance_types - list of machine types that can de run with this image

Provider-specific parameters:

  • AWS:
    • ami - Amazon Machine Image ID
    • vpc - boolean flag, indicates whether the machine will be lauched in an Amazon VPC
  • GCP:
    • image - image or image family name
  • Libvirt:
    • box - Vagrant box name

Distribution-specific parameters:

  • configure_subscription_manager - boolean flag for RedHat system registration. When the flag is set MDBCI registers the system via RHSM and attaches it to an available subscription on the machine creation and unsubscribes and de-registers on the destruction.
  • configure_suse_connect - boolean flag for SLES system registration. When the flag is set MDBCI activates the system via SUSEConnect on the machine creation and deactivates it on the destruction.

repo.d

Repositories for products are described in json files. Each file could contain one or more repo definitions (fields are commented below). During the start mdbci scans repo.d directory and builds full set of available product versions.

File format:

[
  {
    "repo": "http://yum.mariadb.org/10.5/centos/7/x86_64/",
    "repo_key": "https://yum.mariadb.org/RPM-GPG-KEY-MariaDB",
    "platform": "centos",
    "platform_version": "7",
    "product": "mariadb",
    "version": "10.5"
  },
  {
    "repo": "http://yum.mariadb.org/10.5/centos/8/x86_64/",
    "repo_key": "https://yum.mariadb.org/RPM-GPG-KEY-MariaDB",
    "platform": "centos",
    "platform_version": "8",
    "product": "mariadb",
    "version": "10.5"
  }
]

Available options

  • product - product name,
  • version - product version,
  • repo - link to the repo,
  • repo_key - link to repo key,
  • platform - name of target platform,
  • platform_version - name of version of platform.

Virtual machine usage

Before the machine creation

  1. Make sure that required provider's settings are specified in the config.yaml.
  2. Generate (or update) the repository configuration for the products hat will be installed on the machines: 
    ./mdbci generate-product-repositories --product mariadb
    See generate-product-repositories command documentation for details.

Steps

The core steps required to create virtual machines using MDBCI are:

  1. Create or copy the configuration template that describes the VMs you want to create.
  2. Generate concrete configuration based on the template.
  3. Issue VMs creation command and wait for it's completion.
  4. Use the created VMs for required purposes.
  5. When done, call the destroy command that will terminate VMs and clear all the artifacts: configuration, template (may be kept) and network configuration file.

1. Template creation

Template is a JSON document that describes a set of virtual machines. For example, call it template.json.

{
  "mariadb_host": {
    "hostname": "mariadbhost",
    "box": "ubuntu_jammy_gcp",
    "labels": [
      "alpha",
      "bravo"
    ],
    "product": {
      "name": "mariadb",
      "version": "10.5",
      "cnf_template": "server1.cnf",
      "cnf_template_path": "../cnf"
    }
  },
  "several_products_host": {
    "hostname": "severalproductshost",
    "box": "ubuntu_jammy_gcp",
    "products": [
      {
        "name": "maxscale",
        "version": "2.5"
      },
      {
        "name": "mariadb",
        "version": "10.5"
      }
    ]
  }
}

Each host description contains the hostname and box fields. The first one is set to the created virtual machine. The box field describes the image that is being used for VM creation and the provider. In the example we will create two Ubuntu Jammy machines using ubuntu_jammy_gcp box of the GCP provider.

You can get the list of boxes using the ./mdbci show platforms command.

Then each host is setup with the product. The products will be installed on the machines. You can install several products on one host, use the products field for it and describe the products list as array of json-objects (see several_products_host for reference).

When installing a database you must also specify the name of the configuration file and the path to the folder where the file is stored. It is advised to use absolute path in cnf_template_path as the relative path is calculated from within the configuration directory.

labels names groups of machines that could be brought up independently of other machines in the configuration file. A set of machines with the same labels will be created when calling mdbci up with --labels option.

Read more about template creation.

2. Configuration generation

In order to create configuration you should issue the generate command. Let's assume you have called the template file in the previous step template.json. Then the generation command might look like this:

./mdbci generate --template template.json config

After that the config directory containing the MDBCI configuration will be created.

During the generation procedure MDBCI will look through the repositories to find the required image and product information. Please look through the warnings to determine the issues in the template.

On this step you can safely remove the configuration directory, modify the template and regenerate the configuration once again.

3. Virtual machine creation

MDBCI tries to reliably bring up the virtual machines. In order to achieve it the creation and configuration steps may be repeated several times if they fail. By default the procedure will be repeated 5 times.

In order to run the configuration issue the following command:

./mdbci up config

The option --attempts specifies the number of attempts that MDBCI will make to bring up and configure each of the virtual machines. It is advised to reduce this number to one as it is sufficient to catch most issues.

The option --recreate specifies that existing VMs must be destroyed before the configuration of all target VMs. The destruction will be done with the help of reliable destroy command.

The option --labels specifies the list of desired labels. It allows to filter VMs based on the label presence. If any of the labels passed to the command match any label in the machine description, then this machine will be brought up and configured according to it's configuration.

Labels specified in the --labels option should be separated with commas, do not contain any whitespace. Examples:

  • one tag: mdbci up config --labels alpha,
  • several tags: mdbci up config --labels alpha,beta,gamma.

If no machines matches the required set of labels, then no machine will be brought up.

4. Using the virtual machines

After the successful startup the virtual machine is available for operation. The file config_network_config will be created. This file contains information about the network information of the created entities. You can either use this information or issue commands directly using special MDBCI commands. You can also connect to the VM via ssh using config_ssh_config:

ssh -F config_ssh_config several_products_host

Read more about ssh connection.

Interaction examples

Run a command on the machine using sudo

./mdbci sudo --command "tail /var/log/anaconda.syslog" config/node0 --silent

Where config is the path to the configuration directory and node0 is the node name.

Run a command on the machine via ssh

./mdbci ssh --command "cat anaconda.syslog" config/node0 --silent

Install a repository to the given configuration node

./mdbci setup_repo --product mariadb --product-version 10.0 config/node0

Install the product to the given configuration node

./mdbci install_product --product 'maxscale' config/node0

5. Shutting down the virtual machines

When finished and virtual machines are no longer needed you can issue destroy command:

./mdbci destroy config

This will:

  • stop the virtual machines reliably;
  • remove configuration directory;
  • remove network information file;
  • remove ssh config file;
  • remove template that was used to generate the configuration.

If you do not want to delete the template file, add the --keep-template option.

See also:

Providers and supported boxes

Providers

MDBCI uses the Vagrant and a set of low-level tools to create virtual machines and reliably destroy them when the need for them is over. Currently the following Vagrant back ends are supported:

  • Libvirt to manage kvm virtual machines.

MDBCI uses the Terraform to create cloud virtual machines and reliably destroy them when the need for them is over. Currently the following Terraform back ends are supported:

MDBCI also supports:

Boxes

MDBCI currently provides support for the following distributions:

  • CentOS 7, 8;
  • Debian Buster, Bullseye;
  • RHEL 7, 8, 9;
  • Rocky Linux 8, 9;
  • SLES 12, 15;
  • Ubuntu 18.04, 20.04, 22.04;
  • Windows Server 2019 via Google Cloud Platform. Read more.

Machine template

The first step in creating the virtual machine is to create a template. The template is described in json format.

The main section describes the set of nodes to be generated. Example for creating two nodes:

{
    "node1": {
    ...
    },
    "node2": {
    ...
    }
}

Nodes description

Required attributes

Required attributes are the box name and the host name. Boxes must be of the same category for each node. Example:

{
    "node1": {
        "box": "debian_buster_libvirt",
        "hostname": "host1"
    },
    "node2": {
        "box": "centos_8_libvirt",
        "hostname": "host2"
    }
}

See the list of available providers and boxes

Optional attributes

Optional parameters are product, products, labels, cnf_template_path, box_parameters:

  • product is a description of a single product. Full list of available products.
  • products is a description of a list of several products. Full list of available products.
  • labels is a set of labels. Names groups of machines that could be brought up independently of other machines in the configuration file. A set of machines with the same labels will be created when calling mdbci up with --labels option.
  • cnf_template_path is the path to the configuration files to be passed to the machine. When installing a database you must also specify the name of the configuration file and the path to the folder where the file is stored. It is advised to use absolute path in cnf_template_path as the relative path is calculated from within the configuration directory.
  • box_parameters is a description of the selected box parameters that are being overridden for a single node (e.g. disable RHEL system registration setting configure_subscription_manager flag to false). See boxes configuration for more information.

Cloud node attributes

You can specify some special parameters when creating a cloud node template (via AWS, Digitalocean or GCP):

  • memory_size - node RAM size
  • cpu_count - node number of processors
  • machine_type - node machine type family

Example:

{
  "node_000": {
    "hostname": "node000",
    "box": "rhel_7_gcp",
    "memory_size": "2048",
    "cpu_count": "8",
    "machine_type": "g1-small"
  }
}

Product attributes

Also need to specify for the product:

  • name is the product name. Full list of available products.
  • version is the product version. The version is required for some products, see full list of available products.
  • (Optional) cnf_template_path is the path to the configuration files to be passed to the machine.
  • (Optional) cnf_template is the name of the file.
  • (Optional) key is the repository key. The key from repo.d will be ignored.
  • (Optional) force_version is a usage the specific version. Use true to disable smart searching for repo and install specified version.
  • (Optional) include_unsupported with true value allows you to use an unsupported repository for the current product. List of products with unsupported repositories

Example:

{
    "node1": {
        "box": "debian_buster_libvirt",
        "hostname": "host1",
        "product":{
            "name": "maxscale",
            "version": "2.5"
        },
        "labels": [
             "alpha"
        ]
    },
    "node2": {
        "box": "centos_8_libvirt",
        "hostname": "host2",
        "cnf_template_path": "../cnf",
        "products": [
        {
            "name": "maxscale",
            "version": "2.5"
        },
        {
            "name": "mariadb",
            "version": "10.5",
            "cnf_template": "server1.cnf"
        }]
    },
    "node3": {
        "box": "rhel_8_libvirt",
        "hostname": "host3",
        "product": {
            "name": "mariadb",
            "version": "10.6.5"
        },
        "box_parameters": {
            "configure_subscription_manager": "false"
        }
    }
}

Connecting to a virtual machine

MDBCI generates the SSH configuration file that eases interaction with the created virtual machines. The file is located near by the configuration directory with the _ssh_config suffix.

Suppose you have configuration named build_vms and node with the name base_vm, then you can connect to this machine with the following command:

ssh -F build_vms_ssh_config base_vm

Windows Machines

MDBCI supports Windows machines.

Supported function

  • generate
  • up
  • destroy

Create SSH-key

You must to create the SSH-key (windows.pem) in the MDBCI configuration folder (full path: ~/.config/mdbci/windows.pem)

Example

  1. Create template.

    Windows node template:

    {
    "windows" :
    {
            "hostname": "windows",
            "box": "windows_gcp"
    }
}

  2. Use the generate command: mdbci generate --template windows-template.json windows-machine

  3. Use the up command: mdbci up windows-machine

    Now the machine is created and you can use it. You can connect to machine via ssh: ssh -F windows-machine_ssh_config windows

  4. After using machine, you can destroy it: mdbci destroy windows-machine

Using dedicated servers

MDBCI allows to configure dedicated servers too.

Creating a dedicated box

You must add the description of the server as dedicated in the list of boxes. You may use the following template as a starting point:

{
    "debian_dedicated": {
        "provider": "dedicated",
        "platform": "debian",
        "platform_version": "buster",
        "host": "example-host-name",
        "user": "user",
        "ssh_key": "/home/user/.ssh/id_ed25519"
    }
}

You must provide all the properties in this file.

Creating a dedicated machine

The next step is to create a dedicated machine template.

You must specify the machine name, hostname, and a pre-created dedicated box. You can also specify the names and version of products to install on the dedicated machine Sample template:

{
        "centos_dedicated_node" : {
                "hostname" : "mdbcinode",
                "box" : "centos_dedicated",
                "products":[
                  {
                    "name": "docker",
                    "version": "19.03"
                  }
                ]
        },
        "debian_dedicated_node" : {
                "hostname" : "mdbcinode",
                "box" : "debian_dedicated",
                "products":[
                  {
                    "name": "mariadb",
                    "version": "10.4"
                  }
                ]
        }
}

Use the configure command to create a configuration folder.

The next step is to raise a dedicated machine.

Use the up command to raise the machine. The machine was created successfully. Now you can use the list of commands:

  • install_product - command to install a specific product.
  • remove_product - command to remove a specific product.
  • destroy - command to delete the configuration directory.

Commands

SYNOPSIS

mdbci [options] command

OPTIONS

-a, --attempts [number]

How many times to perform certain action.

-b, --boxes [boxes file]

Uses [boxes file] for existing boxes. By default 'boxes.json' will be used as boxes file.

-c, --command [command]

Set command to execute.

--configuration-file [path]

Path to the configuration file used for a command.

-f, --field [box configuration field]

Use [box configuration field] for existing box configuration field.

--force

Use the --force flag to remove interactivity.

-i, --platform-version [version]

Platform version for the show boxes command. Must be used together with --platform option.

--ipv6

If ipv6 must be added to network_config file (also enables ipv6 for libvirt).

--json

Print information in json format.

-k, --key [key file]

Key file to the node for public_keys command.

--keep-template

Do not destroy the template during the destroy command.

--maxscale-ci [release]

Name of the MaxScale release in the CI repository. Used during repository generation command.

-n, --box-name [box name]

Use [box name] for existing box names.

-node-name [node name]

Name of the node.

-o, --platform [name]

Platform name for the show boxes command.

-p, --product [product name]

Product name for setup repository and install product commands.

--path-to-nodes [path]

Path to directory with nodes configuration.

-r, --repo-dir [path]

Set path to the product repository overriding the default locations.

-s, --silent

Keep silence, output only requested info or nothing if not available.

--snapshot-name [Name]

Name of the snapshot.

-t, --template [configuration file]

Uses [configuration file] for running instance. By default instance.json will be used as configuration template.

--user

Name of the user to create.

-v, --product-version [version]

Product version for setup repo and install product commands.

-w, --override

Override previous configuration.

COMMANDS

check_relevance

Check for relevance of 'network_config' file.

clean-unused-resources

Destroy additional cloud resources that are lost or unused. More about clean-unused-resources command

configure

Create a configuration file for MDBCI

create_user

Creates a new user on the VM.

deploy-examples

Deploy examples from AppImage to the current working directory.

destroy

Destroy configuration with all artefacts or a single node.

generate

Generate a configuration based on the template.

generate-product-repositories

Generate product repository configuration for all known products.

help

Show information about MDBCI tool and it commands. More about help command

install_product

Install a product onto the configuration node. More about install_product command

list_cloud_instances

Show list all active instances on Cloud Providers.

list-cloud-resources

Show all cloud resources (VMs, disks, key pairs, security groups) More about list-cloud-resources command

remove_product

Remove the product on selected node.

provide-files

Provide files from the local computer to the Node.

public_keys

Copy ssh keys to configured nodes.

setup-dependencies

Install all dependencies.

setup_repo

Install product repository and update it. More about setup_repo command

show

Get information about MDBCI and configurations. More about show subcommands

snapshot

Manage snapshots of configurations and nodes.

ssh

Execute command on the configuration node.

sudo

Execute command using sudo on the node.

up

Setup environment as specified in the configuration.

update-configuration

Update the service configuration file and restart the service.

SHOW SUB COMMANDS

box, boxes, boxinfo, boxkeys, help, keyfile, network, network_config, platforms, private_ip, provider, repos, versions More about show repos command

EXAMPLES

mdbci show versions --platform ubuntu

mdbci show boxes --platform centos

mdbci show boxes --platform ubuntu --platform-version trusty

mdbci show versions --platform ubuntu

mdbci show box T/node

mdbci show boxinfo --box-name centos_6_vbox --field box

mdbci show boxinfo --box-name suse_13_aws

mdbci show provider suse_13_aws

mdbci sudo --command "tail /var/log/anaconda.syslog" T/node0 --silent

mdbci ssh --command "cat script.sh" T/node1

mdbci --repo-dir /home/testbed/config/repos show repos

mdbci up --attempts 4 T/node0

mdbci setup_repo --product maxscale T/node0

mdbci setup_repo --product mariadb --product-version 10.0 T/node0

mdbci install_product --product maxscale T/node0

mdbci public_keys --key keyfile.pem T/node0

mdbci snapshot list --path-to-nodes T --node-name N

mdbci snapshot [take, revert, delete] --path-to-nodes T [ --node-name N ] --snapshot-name S

generate-product-repositories

Most MDBCI products require additional information that is stored in the repo.d. Use the generate-product-repositories command to generate (or update) the repo.d:

./mdbci generate-product-repositories

If you only need to generate a specific product, use the --product [NAME] option:

./mdbci generate-product-repositories --product mariadb

If you only need to generate a specific product with a specific version, use the --product[NAME] and --product-version [VERSION] options:

./mdbci generate-product-repositories --product mariadb --product-version 10.6

You should run this command regularly to keep up with the latest changes in the repositories of the products, supported by MDBCI. Full list products

help

To display all available MDBCI commands use:

mdbci help

To view additional information about a specific command use:

mdbci [COMMAND] --help

Example:

mdbci up --help

install_product

This command installs a product to the given configuration node.

You must provide the following parameters to the command:

  • Name of the product with --product
  • Version of the product with --product-version
  • Node to which product will be installed

You also may provide the following optional parameters:

  • Hard-set repository key with --repo-key. The key from repo.d will be ignored
  • --force-version to disable smart searching for repo and install specified version
  • --include-unsupported to include an unsupported repository. Full list of products with unsupported repositories

setup_repo

This command installs a repository to the given configuration node.

You must provide the following parameters to the command:

  • Name of the product with --product
  • Version of the product with --product-version
  • Node to which repository will be installed

You also may provide the following optional parameters:

  • Hard-set repository key with --repo-key. The key from repo.d will be ignored
  • --force-version to disable smart searching for repo and install specified version
  • --include-unsupported to include an unsupported repository. Full list of products with unsupported repositories

show repository

This command is meant to extract repository information out of the MDBCI database.

You must provide all the following parameters to the command:

  • Name of the product with --product
  • Version of the product with --product-version
  • Name of the platform with --platform
  • Version of the platform with --platform-version

If you want to use this command inside other applications, please use --silent argument.

Examples

Get the repository for the MaxScale CI product with version maxscale-2.4.9-debug on Ubuntu Bionic:

mdbci show repository --product maxscale_ci --product-version maxscale-2.4.9-debug \
  --platform debian --platform-version bionic --silent

Lost resources management commands

General description

MDBCI can find and destroy unused cloud resources, such as:

  • GCP and AWS instances
  • GCP disks
  • AWS volumes
  • AWS security groups
  • AWS key pairs

List unused resources

mdbci list-cloud-resources

This command lists in a table form the names of the unused resources and their creation dates. The resources are grouped by their type and provider.

Options

  • --output-file FILENAME The name of the JSON report file with the list of resources that will be generated by MDBCI. If not specified, the list will be written to standard output.
  • --hours NUMBER_OF_HOURS The number of hours after which a resource is considered unused. If not specified, all active cloud resources will be listed.
  • --json If specified, the list of instances will be written to the standard output in a JSON format instead of a table.

A resource is considered unused if it was created (or generated) earlier than NUMBER_OF_HOURS hours ago an is not currently being used by any VM. The command ends with an error if any resource is present, no otherwise

Clean unused resources

mdbci clean-unused-resources --resources-list PATH_TO_REPORT_FILE

This command destroys the cloud resources.

Options

  • (Required) --resources-list PATH_TO_REPORT_FILE The name of the JSON report file with the list of resources that will be destroyed.

Report file format

{
  "instances": {
    "aws": [
      {
        "node_name": "somename", // Required to destroy
        "launch_time": "2022-04-21T19:31:45+00:00"
      }
    ],
    "gcp": [
      {
        "type": "c2-standard-4",
        "node_name": "mdbci-3uwpyr1v-1669724674-build", // Required to destroy
        "launch_time": "2022-11-29T12:24:54+00:00",
        "zone": "us-central1-c", // Required to destroy
        "path": "/path/to/config",
        "username": "username"
      }
    ]
  },

  "disks": {
    "aws": [
      {
        "name": "volume-name", // Required to destroy
        "zone": "eu-west-1-a",
        "creation_date": "2022-11-02T08:29:31-07:00"
      }
    ],
    "gcp": [
      {
        "name": "mdbci-ezmlb5cv-1661161923-xpand-002-disk", // Requred to destroy
        "zone": "europe-north1-a", // Required to destroy
        "creation_date": "2022-11-02T08:29:31-07:00"
      }
    ]
  },

  "key_pairs": [
    {
      "name": "mdbci-ezmlb5cv-1661161923-xpand-002-key", // Required to destroy
      "creation_date": "2022-11-02T08:29:31-07:00"
    }
  ],

  "security_groups": [
    {
      "group_id": "sg-0228b331c2a7be875", // Required to destroy
      "configuration_id": "mdbci-ezmlb5cv-1661161923",
      "creation_date": "2022-11-02T08:29:31-07:00"
    }
  ]
}

Fields required to destroy a resource are marked. Other fields are optional and contain additional information about the resources.

Supported products

product name - name of the product to specify in the template or install_product command.

different version support - indicates whether different versions are supported. MDBCI generate-product-repository.

cnf file support - indicates whether cnf files are supported.

deletion is available - indicates whether the product can be deleted. If deletion is available, it shows the name of the product to be deleted using the remove_product command.

unsupported repositories - indicates whether the product has an unsupported repositories

product namedifferent versions supportcnf file supportdeletion is availableunsupported repositories
mariadb++mariadb-
mariadb_ci++mariadb-
mariadb_staging++mariadb-
mdbe++mariadb-
mdbe_ci++mariadb-
mdbe_staging++mariadb-
maxscale++maxscale-
maxscale_ci++maxscale-
mysql++--
galera+ (version for mariadb)+--
galera_config-+--
docker+---
xpand+---
xpand_staging+---
mdbe_build----
connectors_build----
galera_3_enterprise+---
galera_4_enterprise+---
galera_3_community+---
galera_4_community+---
google-authenticator----
kafka+---
kerberos----
kerberos_server----
mariadb_plugins----
sysbench----
core_dump----
connector_odbc+---
connector_odbc_staging+---
connector_odbc_ci+---
caching_tools----
debugging_tools----
extra_package_management----
security_tools----
java+---
python----
binutils----

Xpand

MDBCI supports installing Xpand on virtual machines.

You can specify two product versions:

  • Standard version. Version consists of 3 digits separated by dots. For example 9.1.3
{
  "centos_node" :
  {
    "hostname" : "hostname",
    "box" : "centos_7_libvirt",
    "products": [
    {
            "name": "xpand",
            "version": "9.1.4"
    }]
  }
}
  • Internet version. Version starts with http. For example http://xpand.source/xpand-.el7.tar.bz2
{
  "centos_node" :
  {
    "hostname" : "hostname",
    "box" : "centos_7_libvirt",
    "products": [
    {
            "name": "xpand",
            "version": "http://xpand.source/xpand-.el7.tar.bz2"
    }]
  }
}

The second time Xpand is not installed. To reinstall, you need to recreate the machine.

Galera

MDBCI supports installing different versions of Galera on virtual machines. You need to select the Galera type and Galera version to install Galera.

Supported Galera types

  • galera_3_enterprise
  • galera_4_enterprise
  • galera_3_community
  • galera_4_community

Example

{
   "node":{
      "hostname":"hostname",
      "box":"box_name",
      "products":[
         {
            "name":"galera_4_community",
            "version":"mariadb-4.x"
         }
      ]
   }
}

Kerberos

MDBCI supports installing Kerberos on virtual machines.

The MDBCI allows you to install the kerberos and kerberos_server products. kerberos includes client kerberos packages, kerberos_server includes client, server packages and rng-tools.

You can specify two product versions:

  • Client version. Product name is kerberos
{
  "centos_node" :
  {
    "hostname" : "hostname",
    "box" : "centos_7_libvirt",
    "products": [
    {
            "name": "kerberos"
    }]
  }
}
  • Server version. Product name is kerberos_server
{
  "centos_node" :
  {
    "hostname" : "hostname",
    "box" : "centos_7_libvirt",
    "products": [
    {
            "name": "kerberos_server"
    }]
  }
}

MariaDB plugins

MDBCI supports installation of plugins for MariaDB Enterprise or Community as a stand-alone product or as an additional product to the MariaDB server. In the first case you must specify it's name in the list of products and version of the MariaDB server you want to install. In the latter case you must provide version for the MariaDB product and leave it empty for the plugin.

The list of plugins and their corresponding MDCI product names:

  • backup, mdbe_plugin_backup;
  • cmapi, mdbe_plugin_cmapi;
  • columnstore, mdbe_plugin_columnstore;
  • connect, mdbe_plugin_connect;
  • cracklib password check, mdbe_plugin_cracklib_password_check;
  • gssapi client, mdbe_plugin_gssapi_client;
  • gssapi server, mdbe_plugin_gssapi_server;
  • hashicorp key management, mdbe_plugin_hashicorp_key_management;
  • mariadb test, mdbe_plugin_mariadb_test;
  • mroonga, mdbe_plugin_mroonga;
  • oqgraph, mdbe_plugin_oqgraph;
  • rocksdb, mdbe_plugin_rocksdb;
  • s3, mdbe_plugin_s3;
  • spider, mdbe_plugin_spider;
  • xpand, mdbe_plugin_xpand.

MDBCI template examples

Install standalone plugin

This will install MDBE server 10.5 and columnstore plugin.

{
  "node_product" : {
    "hostname" : "host",
    "box" : "box",
    "products": [
      {
        "name": "mdbe_plugin_columnstore",
        "version": "10.5"
      }
    ]
  }
}

Install plugin along with MDBE server

This will install MDBE server 10.5 and connect plugin.

{
  "node_product" : {
    "hostname" : "host",
    "box" : "box",
    "products": [
      {
        "name": "mdbe_plugin_connect"
      },
      {
        "name": "mdbe",
        "version": "10.5"
      }
    ]
  }
}

Install several plugins

This will install MDBE server 10.5 and two plugins: xpand and gssapi server.

{
  "node_product" : {
    "hostname" : "host",
    "box" : "box",
    "products": [
      {
        "name": "mdbe_plugin_xpand",
        "version": "10.5"
      },
      {
        "name": "mdbe_plugin_gssapi_server"
      }
    ]
  }
}

Auxiliary products

  • caching_tools - in-memory data structure store instruments. Includes memcached and redis packages.
  • debugging_tools - suite of tools for debugging and profiling. Includes gdb and valgrind.
  • extra_package_management - instruments for additional package and repository management. Includes yum-utils and epel-release.
  • security_tools - installs vault package.
  • java - OpenJDK Java Runtime Environment. A certain version can be specified. Otherwise version latest will be installed.
  • python - installs python3 package (python36 for RHEL 8)
  • binutils - installs binutils package

Using the MariaDB product from CI server

MDBCI supports installing MariaDB from CI server on virtual machines.

Use the mariadb_ci product in the configuration if you need MariaDB from CI server. You must also specify the Galera version in the configuration (galera_3_community or galera_4_community). You can specify Galera either before mariadb_ci or after it.

Plugins

The mariadb_ci product also supports the installation of plugins.

You may not specify mariadb_ci if you use plugins. Just indicate the name of this plugin and the version of mariadb_ci

List of supported plugins

  • columnstore, mariadb_plugin_columnstore;
  • connect, mariadb_plugin_connect;
  • cracklib password check, mariadb_plugin_cracklib_password_check;
  • gssapi client, mariadb_plugin_gssapi_client;
  • gssapi server, mariadb_plugin_gssapi_server;
  • mariadb test, mariadb_plugin_mariadb_test;
  • mroonga, mariadb_plugin_mroonga;
  • oqgraph, mariadb_plugin_oqgraph;
  • rocksdb, mariadb_plugin_rocksdb;
  • spider, mariadb_plugin_spider;
  • xpand, mariadb_plugin_xpand.

Examples

{
        "node" : {
                "hostname" : "hostname",
                "box" : "box_name",
		        "products":[
			        {
				         "name": "mariadb_plugin_oqgraph",
				         "version": "10.4-2020-Jun-17-12-00-00"
			        },
			        {
				         "name": "galera_4_community",
				         "version": "mariadb-4.x"
			        }
                ]
        }
}

{
        "node" : {
                "hostname" : "hostname",
                "box" : "box_name",
		        "products":[
			        {
				         "name": "galera_3_community",
				         "version": "mariadb-3.x"
			        },
			        {
				         "name": "mariadb_ci",
				         "version": "10.2-2020-Jun-19-08-00-00"
			        }
                ]
        }
}

Interaction with cloud platforms

MDBCI uses virtual machines from cloud providers such as

Detailed topics

Google Cloud Platform regions management

Configuration file

Currently supported regions are listed in config.yaml file in the ~/.config/mdbci. If the set of machines to be created does not meet the CPU quota in the default region, MDBCI will select another one from the regions list.

gcp:
  ...
  default_region: # the region used when the CPU quota is met
  regions: # list of supported regions
  - us-central1
  - europe-west4
  ...

Add support for a new region

1. Modify configuration file

Add region name to the configuration file described above.

2. Check Cloud Routers

Check if the added zone has a Cloud Router. The Router allows the machines without external IPs to access the external resources. You can list the routers in the current project using this command:

gcloud compute routers list

3. Create the Router

Create and configure the Cloud Router if none exists in the current region using gcloud cli or a Terrafom file.

Create a router instance:

gcloud compute routers create ROUTER_REGION \
    --network default \
    --region ROUTER_REGION

Configure the router for Cloud NAT:

gcloud compute routers nats create nat-config \
    --router-region ROUTER_REGION \
    --router ROUTER_NAME \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips

Read more about building internet connectivity to GCP VMs

Create and configure the Cloud Router using Terraform

1. Create Terraform NAT configuration file

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 3.65.0"
    }
  }
}

provider "google" {
  credentials = file("<path to GCP credentials file>")
  project     = "<project name>"
}

resource "google_compute_router" "<router name>" {
  name    = "<router name>"
  region  = "<router region>"
  network  = "default"
}

resource "google_compute_router_nat" "<NAT name>" {
  name                               = "<NAT name>"
  router                             = "<router name>"
  region                             = "<router region>"
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

2. Apply the Terraform configuration

terraform apply

SUSE registration

SLES systems should be registered via SUSE Customer Center. The corresponding boxes should have confugure_suse_connect flag set to true. Also it is necessary to specify SUSE credentials: email address, subscription registration code (key) and the Registration Proxy Server address in the config.yaml file.

...
suse:
  email: # Customer's email
  key: # Subscription registration code
  registration_proxy: # SUSE Registration Proxy address
...

Registration Proxy

SLES systems can be registered in 2 ways: to the Customer Center directly or via a Registration Proxy. A Registration Proxy is a server authorized by SUSE that can register systems and exchange information daily with the Customer Center in order to reduce the load on the SUSE servers. Currently only SLES GCP machines are able to use proxy if possible.

See also:

Finding AWS virtual machine images

MDBCI stores various information about every provided box - it's provider, distribution, architecture, etc. You can find that information into configuration/boxes/ directory.

Every AWS box requires AMI (Amazon Machine Image) id. There is several ways to find the desired image.

1. Looking at official distributions wiki

Many popular linux distributions have a detailed page with official AMI ids. There is list of these pages:

2. Using AWS-CLI tool

AWS-CLI is a powerfull tool for managing AWS services. The following steps will show you how to find the desired AMI using this tool.

  1. Install the tool with pip install awscli command
  2. Configure the tool with the aws configure command. It will require you to enter amazon access_key_id, secret_access_key and the output format (json / yaml / text / table)
  3. To find AMI id execute
aws ec2 describe-images \
   --owners 309956199498 \
   --query 'Images[*].[CreationDate,Name,ImageId, Architecture]' \
   --filters "Name=name,Values=RHEL-7.?*GA*" \
   --region us-east-1 \
   --output table | sort -r

Where:

  • --owners - image owner id
  • --query - describes output values
  • --filters - AMI filter parameters
  • --region - aws region
  • --output - output format

Refer to aws-cli filter usage page, describe-images command page for detailed command info

List of known owners id:

  • Amazon Marketplace - 679593333241
  • RHEL - 309956199498
  • Ubuntu - 099720109477
  • SUSE - 013907871322
  • Debian - 136693071363
  • Rocky Linux - 792107900819

Also you can get detail information about AMI by executing aws ec2 describe-images --image-ids ami-07a44bb660e25b065 command.