Copy Volumes

See what we're up to: Runway is here! 🚀

Copying volumes #

Copying (or duplicating) Docker volumes may seem like a daunting task, but here are some pointers to help you along the way.

Copying the whole Stack #

**New in quantum-cli 2.5.0**

quantum-cli version 2.5.0 or newer includes a sub-command that can be used to duplicate a whole stack - either on the same endpoint, or from one endpoint to another:

$ quantum-cli migration \
  --source-endpoint sourceenddpoint --stack sourcestack \
  --target-endpoint targetendpoint --target-stack targetstackname

This command copies the complete stack, with all stack volumes (plus all external volumes used by the stack) and config objects - all including contents.

That is often enough, but not always - see the more manual ways of copying volumes and their contents below.

We recommend stopping all traffic to the stack before attempting quantum-cli migration, and also stopping any databases that write into the volumes (since copying the data from a running database usually leads to a corrupted database). See the hint about databases below.

Databases #

Be advised that the external content is aimed towards running databases on a server or virtual machine. The concepts apply to containers as well.

If moving or duplicating a database is the objective, then we suggest to use the build-in replication features to duplicate/copy the contents of the database.

The most common databases among our customers are MySQL (or MariaDB), PostgreSQL and Redis. Use one of the following guides to setup replication (between your primary database server and your replica (or target)):

Using minio #

Minio is an Amazon-S3-compatible object store. Follow our example on how to setup a Docker registry backed by minio and setup a minio service.

On a high-level, we will perform these two steps:

  1. Copy data to minio.
  2. Copy data from minio.
For brevity, we'll do one-off work directly inside a container. This is not recommended. Instead, customize a `Dockerfile` and keep the setup documented and repeatable in there.

Copy to minio #

In this example, we want to copy the assets directory of a web server to another volume. In order to achieve our goal, we create and deploy an auxiliary stack to help us move the data:

version: "3.7"

volumes:
  source-volume:
    external: true

services:
  copy_service:
    image: centos:latest
    entrypoint: /bin/sh -c "while true; do sleep 1; done"
    volumes:
      - source-volume:/from:ro

Deploy the stack with quantum-cli:

$ quantum-cli stack deploy --create my-copy-stack
...

After the successful deployment, we use the web shell in the console to connect to copy_service and execute the following commands to install and configure the minio client:

$ yum make-cache && yum install -y wget
$ wget https://dl.min.io/client/mc/release/linux-amd64/mc
...
$ chmod +x mc
$ mc alias set backup-minio http://MINIO-SERVER-ADDRESS ACCESS_KEY SECRET

Test the connection to backup-minio:

$ mc admin info backup-minio
In case the steps don't work as expected, please follow [the official documentation](https://docs.min.io/minio/baremetal/reference/minio-cli/minio-mc.html).

Once mc is configured, copy the files to a bucket called asset-backup in minio:

$ mc cp --recursive /from/ backup-minio/asset-backup

Copy from minio #

Repeat the steps above, but this time mount the target volume and re-deploy the stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

version: "3.7"

volumes:
  target-volume:
    external: true

services:
  copy_service:
    image: centos:latest
    entrypoint: /bin/sh -c "while true; do echo hello world; sleep 1; done"
    volumes:
      - target-volume:/to

Use the web shell one more time and copy data from minio to the docker volume:

$ mc cp --recursive backup-minio/asset-backup /to

From volume to volume #

It may be tempting to use this method to move databases as well, but keep in mind that without proper locks on the database data might be lost and the database could be corrupted.

Assume the following setup:

  1. An existing volume called volume-source
  2. A new (and empty) volume called volume-target

Let’s adjust the stack to copy from source to target:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

version: "3.7"

volumes:
  volume-source:
    external: true
  volume-target:

services:
  my_service:
    image: project/service:1.0
    volumes:
      - volume-source:/var/www/assets-old:ro
      - volume-target:/var/www/assets
As demonstrated, we ingest `volume-source` as an external volume into the new stack. The volume is mounted read-only (`:ro`).

Deploy the stack with quantum-cli:

$ quantum-cli stack deploy --create my-new-stack
...

Access my_service using the web shell on the console and copy the assets:

$ cd /var/www/assets-old
$ cp -R * /var/www/assets/

Once the operation is done, re-deploy the stack with the volume-source removed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

version: "3.7"

volumes:
  volume-target:

services:
  my_service:
    image: project/service:1.0
    volumes:
      - volume-target:/var/www/assets