:orphan:

.. _ec2-backup-and-restore:

======================
EC2 Backup and Restore
======================

.. default-domain:: mongodb

.. contents:: On this page
   :local:
   :backlinks: none
   :depth: 1
   :class: twocols

This page describes how to backup, verify, and restore a MongoDB running
on EC2 using `EBS Snapshots <http://aws.amazon.com/ebs/>`_.

.. seealso::

   `Cloud Manager
   <https://www.mongodb.com/cloud/cloud-manager?jmp=docs>`_
   orchestrates critical operational tasks you currently perform
   manually across the servers in your MongoDB deployment.

How you backup MongoDB will depend on whether you are using the
``--journal`` option, which is available in versions 1.8 and above.

Backup with ``--journal``
-------------------------

The journal file allows for roll forward recovery. The journal files are
located in the dbpath directory so will be snapshotted at the same time
as the database files.

If the dbpath is mapped to a single EBS volume then proceed to
:ref:`ec2-backup-database-files`.

If the dbpath is mapped to multiple EBS volumes, then in order to guarantee
the stability of the filesystem you will need to :ref:`ec2-flush-and-lock-database`.

.. note:: Snapshotting with the journal is only possible if the journal
   resides on the same volume as the data files, so that one snapshot
   operation captures the journal state and data file state atomically.

Backup without ``--journal``
----------------------------

In order to correctly backup a MongoDB, you need to ensure that writes
are suspended to the filesystem before you backup the filesystem. If
writes are not suspended then the backup may contain partially written
or data which may fail to restore correctly.

Depending on your version of MongoDB, use
:manual:`fsync and lock </reference/command/fsync/>`
or, after MongoDB 2.0, use
:manual:`db.fsyncLock() </reference/method/db.fsyncLock/>`.

If the filesystem is being used only by the database, then you can
use the snapshot facility of EBS volumes to create a backup. If you are
using the volume for any other application then you will need to ensure
that the filesystem is frozen as well (e.g. on XFS filesystem use
``xfs_freeze``) before you initiate the EBS snapshot.

The overall process looks like:

.. image:: /figures/ec2-backup-and-restore-backup.jpg
   :alt: Steps to create a backup using the snapshot facility of EBS volumes
   :width: 700px

.. _ec2-flush-and-lock-database:

Flush and Lock the Database
---------------------------

Writes have to be suspended to the filesystem in order to make a stable
copy of the database files.

Prior to MongoDB version 2.0, this is achieved through the MongoDB
shell using
:manual:`fsync and lock </reference/command/fsync/>`:

.. code-block:: sh

   mongo shell> use admin
   mongo shell> db.runCommand({fsync:1,lock:1});
   {
      "info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock",
      "ok" : 1
   }

MongoDB 2.0 added the
:manual:`db.fsyncLock() </reference/method/db.fsyncLock/>`
method to lock the database and flush writes to disk and added the
:manual:`db.fsyncUnlock() </reference/method/db.fsyncUnlock/>`
method to unlock the database after the snapshot has completed.

During the time the database is locked, any write requests that this
database receives will be rejected. Any application code will need to
deal with these errors appropriately.

.. _ec2-backup-database-files:

Backup the Database Files
-------------------------

There are several ways to create an EBS Snapshot. The following examples use the
:aws-docs:`AWS command line <AWSEC2/latest/CommandLineReference/Welcome.html>`
tool.

Find the EBS Volumes Associated with MongoDB
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the mapping of EBS Block devices to the MongoDB data volumes is
already known, then this step can be skipped. The example below shows
how to determine the mapping for an LVM volume, please confirm with your
System Administrator how the original system was setup if you are
unclear.

Find the EBS Block Devices Associated with the Running Instance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: sh

   shell> ec2-describe-instances
   RESERVATION   r-eb09aa81  289727918005    tokyo,default
   INSTANCE      i-78803e15  ami-4b4ba522    ec2-50-16-30-250.compute-1.amazonaws.com    ip-10-204-215-62.ec2.internal    running    scaleout    0    m1.large    2010-11-04T02:15:34+0000    us-east-1a    aki-0b4aa462    monitoring-disabled    50.16.30.250    10.204.215.62    ebs    paravirtual
   BLOCKDEVICE   /dev/sda1   vol-6ce9f105    2010-11-04T02:15:43.000Z
   BLOCKDEVICE   /dev/sdf    vol-96e8f0ff    2010-11-04T02:15:43.000Z
   BLOCKDEVICE    /dev/sdh   vol-90e8f0f9    2010-11-04T02:15:43.000Z
   BLOCKDEVICE   /dev/sdg    vol-68e9f101    2010-11-04T02:15:43.000Z
   BLOCKDEVICE   /dev/sdi    vol-94e8f0fd    2010-11-04T02:15:43.000Z

As can be seen in this example, there are a number of block devices
associated with this instance. You must determine which volumes make
up the filesystem to snapshot.

Determine how the dbpath is Mapped to the File System
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Log onto the running MongoDB instance in EC2. To determine where the
database file are located, either look at the startup parameters for the
:program:`mongod` process or if :program:`mongod` is running, then you
can examine the running process. In the following example, ``dbpath`` is
set to ``/var/lib/mongodb/tokyo0``.

.. code-block:: sh

   root> ps -ef | grep mongo
   ubuntu 10542 1 0 02:17 ? 00:00:00 /var/opt/mongodb/current/bin/mongod --port 27000 --shardsvr --dbpath /var/lib/mongodb/tokyo0 --fork --logpath /var/opt/mongodb/log/server.log --logappend --rest

Map the dbpath to the Physical Devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using the ``df`` command, determine what the ``--dbpath`` directory is mapped to

.. code-block:: sh

   root> df /var/lib/mongodb/tokyo0
   Filesystem           1K-blocks      Used Available Use% Mounted on
   /dev/mapper/data_vg-data_vol
                        104802308      4320 104797988   1% /var/lib/mongodb

Next determine the logical volume associated with this device. In the
example above, this is ``/dev/mapper/data_vg-data_vol``.

.. code-block:: sh

   root> lvdisplay /dev/mapper/data_vg-data_vol
     --- Logical volume ---
     LV Name                /dev/data_vg/data_vol
     VG Name                data_vg
     LV UUID                fixOyX-6Aiw-PnBA-i2bp-ovUc-u9uu-TGvjxl
     LV Write Access        read/write
     LV Status              available
     # open                 1
     LV Size                100.00 GiB
   ...

This output indicates the volume group associated with this logical
volume, in this example ``data_vg``. Next determine how this maps to the
physical volume.

.. code-block:: sh

   root> pvscan
     PV /dev/md0   VG data_vg   lvm2 [100.00 GiB / 0    free]
     Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0   ]

From the physical volume, determine the associated physical devices, in
this example ``/dev/md0``.

.. code-block:: sh

   root> mdadm --detail /dev/md0
   /dev/md0:
           Version : 00.90
     Creation Time : Thu Nov  4 02:17:11 2010
        Raid Level : raid10
        Array Size : 104857472 (100.00 GiB 107.37 GB)
     Used Dev Size : 52428736 (50.00 GiB 53.69 GB)
      Raid Devices : 4
   ...
              UUID : 07552c4d:6c11c875:e5a1de64:a9c2f2fc (local to host ip-10-204-215-62)
            Events : 0.19

       Number   Major   Minor   RaidDevice State
          0       8       80        0      active sync   /dev/sdf
          1       8       96        1      active sync   /dev/sdg
          2       8      112        2      active sync   /dev/sdh
          3       8      128        3      active sync   /dev/sdi

The block devices ``/dev/sdf`` through ``/dev/sdi`` make up this
physical devices. Each of these volumes will need to be snapped in order
to complete the backup of the filesystem.

.. _create-ebs-snapshot:

Create the EBS Snapshot
~~~~~~~~~~~~~~~~~~~~~~~

Create the snapshot for each device. Using the
:aws-docs:`ec2-create-snapshot <AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateSnapshot.html>`
command, use the Volume Id for the device listed by the
:aws-docs:`ec2-describe-instances <AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html>`
command.

.. code-block:: sh

   shell> ec2-create-snapshot -d backup-20101103 vol-96e8f0ff
   SNAPSHOT     snap-417af82b     vol-96e8f0ff     pending     2010-11-04T05:57:29+0000     289727918005     50     backup-20101103
   shell> ec2-create-snapshot -d backup-20101103 vol-90e8f0f9
   SNAPSHOT     snap-5b7af831     vol-90e8f0f9     pending     2010-11-04T05:57:35+0000     289727918005     50     backup-20101103
   shell> ec2-create-snapshot -d backup-20101103 vol-68e9f101
   SNAPSHOT     snap-577af83d     vol-68e9f101     pending     2010-11-04T05:57:42+0000     289727918005     50     backup-20101103
   shell> ec2-create-snapshot -d backup-20101103 vol-94e8f0fd
   SNAPSHOT     snap-2d7af847     vol-94e8f0fd     pending     2010-11-04T05:57:49+0000     289727918005     50     backup-20101103

Unlock the Database
-------------------

Once the snapshots are in ``pending`` state, the
database can be unlocked. Use the following operation to unlock the database.

.. code-block:: javascript

   mongo shell> db.$cmd.sys.unlock.findOne();
   { "ok" : 1, "info" : "unlock requested" }

When unlocked, the database is available to process write requests.

Verify the Backup
-----------------

In order to verify the backup, you must do the following:

- Check the status of each snapshot to ensure it is "completed."

- Create new volumes based on the snapshots and mount the new volumes.

- Run :program:`mongod` and verify the collections.

.. image:: /figures/ec2-backup-and-restore-verify.jpg
   :width: 700px
   :alt: Steps to verify the backup

Typically, the verification is performed on another machine so that
you do not burden your production systems with the additional CPU and
I/O load of the verification processing.

Describe the Snapshots
~~~~~~~~~~~~~~~~~~~~~~

Using the :aws-docs:`ec2-describe-snapshots <AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeSnapshots.html>`
command, find the snapshots that make up the backup. Using a filter on
the ``description`` field, snapshots associated with the given backup are
easily found. The search text used should match the text used in the ``-d``
flag passed to :aws-docs:`ec2-create-snapshot <AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateSnapshot.html>` command when the backup was made.

.. code-block:: sh

   backup shell> ec2-describe-snapshots --filter "description=backup-20101103"
   SNAPSHOT    snap-2d7af847    vol-94e8f0fd    completed    2010-11-04T05:57:49+0000    100%    289727918005    50    backup-20101103
   SNAPSHOT    snap-417af82b    vol-96e8f0ff    completed    2010-11-04T05:57:29+0000    100%    289727918005    50    backup-20101103
   SNAPSHOT    snap-577af83d    vol-68e9f101    completed    2010-11-04T05:57:42+0000    100%    289727918005    50    backup-20101103
   SNAPSHOT    snap-5b7af831    vol-90e8f0f9    completed    2010-11-04T05:57:35+0000    100%    289727918005    50    backup-20101103

Create New Volumes Based on the Snapshots
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using the :aws-docs:`ec2-create-volume <AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateVolume.html>`
command, create a new volumes based on each of the snapshots that make
up the backup.

.. code-block:: sh

   backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-2d7af847
   VOLUME    vol-06aab26f    50    snap-2d7af847    us-east-1a    creating    2010-11-04T06:44:27+0000
   backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-417af82b
   VOLUME    vol-1caab275    50    snap-417af82b    us-east-1a    creating    2010-11-04T06:44:38+0000
   backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-577af83d
   VOLUME    vol-12aab27b    50    snap-577af83d    us-east-1a    creating    2010-11-04T06:44:52+0000
   backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-5b7af831
   VOLUME    vol-caaab2a3    50    snap-5b7af831    us-east-1a    creating    2010-11-04T06:45:18+0000

Attach the New Volumes to the Instance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using the :aws-docs:`ec2-attach-volume <AWSEC2/latest/CommandLineReference/ApiReference-cmd-AttachVolume.html>`
command, attach each volume to the instance where the backup will be
verified.

.. code-block:: sh

   backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdp vol-06aab26f
   ATTACHMENT     vol-06aab26f     i-cad26ba7     /dev/sdp     attaching     2010-11-04T06:49:32+0000
   backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdq vol-1caab275
   ATTACHMENT     vol-1caab275     i-cad26ba7     /dev/sdq     attaching     2010-11-04T06:49:58+0000
   backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdr vol-12aab27b
   ATTACHMENT     vol-12aab27b     i-cad26ba7     /dev/sdr     attaching     2010-11-04T06:50:13+0000
   backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sds vol-caaab2a3
   ATTACHMENT     vol-caaab2a3     i-cad26ba7     /dev/sds     attaching     2010-11-04T06:50:25+0000

Mount the Volumes
~~~~~~~~~~~~~~~~~

First make the filesystem visible to the host operating system. This
process will vary based on the particular volume management scheme
that you are using.  Check with your system administrator if you are
unsure.

If you are using Logical Volume Manager, assemble the logical
device from the physical devices. The UUID for the device will be the
same as the original UUID from which the backup was made, and can be
obtained using the ``mdadm`` command.

.. code-block:: sh

   backup shell> mdadm --assemble --auto-update-homehost -u
   07552c4d:6c11c875:e5a1de64:a9c2f2fc --no-degraded /dev/md0
   mdadm: /dev/md0 has been started with 4 drives.

You can confirm that the physical volumes and volume groups appear
correctly to the O/S by executing the following:

.. code-block:: sh

   backup shell> pvscan
     PV /dev/md0 VG data_vg lvm2 [100.00 GiB / 0 free]
     Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0 ]

   backup shell> vgscan
     Reading all physical volumes. This may take a while...
     Found volume group "data_vg" using metadata type lvm2

Next, create the mount point and mount the filesystem:

.. code-block:: sh

   backup shell> mkdir -p /var/lib/mongodb

   backup shell> cat >> /etc/fstab << EOF
   /dev/mapper/data_vg-data_vol /var/lib/mongodb xfs noatime,noexec,nodiratime 0 0
   EOF

   backup shell> mount /var/lib/mongodb

Start the Database
~~~~~~~~~~~~~~~~~~

After the filesystem has been mounted, MongoDB can be started. Ensure
that the owner of the files is set to the correct user and group. Since
the backup was made with the database running, the lock file will need
to be removed in order to start the database.

.. code-block:: sh

   backup shell> chown -R mongodb /var/lib/mongodb/toyko0
   backup shell> rm /var/lib/mongodb/tokyo0/mongod.lock
   backup shell> mongod --dbpath /var/lib/mongodb/tokyo0

Verify the Collections
~~~~~~~~~~~~~~~~~~~~~~

:manual:`Validate </reference/command/validate/>`
each collection to ensure it does not contain any invalid BSON objects.

.. code-block:: javascript

   mongo shell> db.blogs.validate({full:true})

Restore
-------

Restore uses the same basic steps as the verification process:

.. NOTE The following terse steps are taken from the screenshot (see note after steps).

1. :manual:`db.shutdownServer() </reference/method/db.shutdownServer/>`

2. :aws-docs:`ec2-create-volume <AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateVolume.html>`
   and
   :aws-docs:`ec2-attach-volume <AWSEC2/latest/CommandLineReference/ApiReference-cmd-AttachVolume.html>`.

3. Mount the file system.

4. Run :program:`mongod`.

   .. image:: /figures/ec2-backup-and-restore-restore.jpg
      :width: 600px
      :alt: Basic steps to restore

After the filesystem is mounted you can decide to:

- Copy the database files from the backup into the current database
  directory

- Start :program:`mongod` from the new mount point, specifying the new
  mount point in the ``--dbpath`` argument.

After the database is started, it will be ready to transact. It will be
at the specific point in time from the backup, so if it is part of a
primary/secondary or replica set relationship, then the instance will need to
synchronize itself to get itself back up to date.

See Also
--------

|mms-home| for seamless automation, backup, and monitoring.

.. include:: /includes/replacement-mms.rst
