Zextras Backup

Table of Contents

Real Time Scan

What is the Real Time Scanner?

The Real Time Scanner is the big innovation in Zextras Backup. Each event on the system is recorded live to Zimbra’s RedoLog and saved by Zextras Backup, which means that it is always possible to rollback an account to a previous state. Thanks to the Real Time Scanner, all the restore modes work with split-second precision.

How Does it Work?

The Real Time Scanner reads all the events of the mail server almost real-time by following the flow of informations provided by the RedoLog. Then it 'replicates' the same operations on its own data structure, creating items or updating their metadata. No information is ever overwritten in the backup, so every item has its own complete history.

Managing the Real Time Scanner

Enabling the Real Time Scanner

Via the Administration Zimlet
  • Select the Zextras Backup Tab.

  • Under Real Time Scanner, press the Enable button.

When the Real Time Scanner is enabled for the first time or re-enabled after a stop, a Live Full Scan is required. A warning will be displayed after enabling the Real Time Scanner, and you will be prompted to start the Full Scan.
Via the CLI

To enable the Real Time Scanner via the CLI, the ZxBackup_RealTimeScanner property of the Zextras Backup module must be set to true:

zxsuite backup setProperty ZxBackup_RealTimeScanner TRUE

Disabling the Real Time Scanner

Via the Administration Zimlet
  • Select the Zextras Backup Tab.

  • Under Real Time Scanner, press the Disable button.

Via the CLI

To disable the Real Time Scanner via the CLI, the ZxBackup_RealTimeScanner property of the Zextras Backup module must be set to false:

zxsuite backup setProperty ZxBackup_RealTimeScanner FALSE
Why Should I Disable the Real Time Scanner?

The only time you should disable the Real Time Scanner is while performing an External Restore of multiple domains. This is a safety measure to avoid high load on your server. After the import, re-enable the Real Time Scanner and perform a SmartScan when prompted.

Limitations and Safety Scan

The main limitation when restoring data acquired via the Real Time Scanner is:

  • Emptied Folder - when a user uses the Empty Folder button in the right-click context menu

In this case, and any time Zextras Backup cannot determine the status of an item by reading the metadata saved by the Real Time Scan, an Account Scan on the given account is triggered BEFORE the restore.

This fixes any misaligned data and sanitizes the backed up metadata for the mailbox.

Blobless Backup Mode

Blobless Backup Mode is a new feature that avoids backing up item blobs while still safeguarding all other item-related information.

This mode is designed to take advantage of advanced storage capabilities of the storage solution such as built-in backup or data replication optimizing both the backup module’s disk space usage and restore speed.

Blobless Backup Requirements

These requirements must be met to enable Blobless Backup Mode:

  • The server is running Zimbra 8.8.15 or higher.

  • No "independent" third-party volumes must exist: Blobless Backup Mode is only compatible with local volumes and centralized third-party volumes.

Blobless Backup Mode is storage-agnostic and can be enabled on any server or infrastructure that meets the requirements above regardless of the specific storage vendor.

How Blobless Backup Mode works

Blobless Backup Mode works exactly as its default counterpart: the RealTime Scanner takes care of backing up item changes while the SmartScan manages domain/cos/account consistency, the only difference between the two is that in Blobless Backup Mode the backup contains no items of kind blob while still saving all metadata and transaction history.

It’s essential to consider that once enabled, Blobless Backup Mode affects the entire server and no blobs get backed up regardless of the target volume and HSM policies.

When the backup is set to Blobless Mode, BLOBs will not be deleted until those are out of the retention period.

Restoring Data from a Blobless Backup dataset

To date, Blobless Backup Mode is only compatible with the "Raw Restore" operation. In the future, additional restore operations will become compatible with blobless datasets.

Enabling Blobless Backup Mode

Blobless Backup Mode can be enabled or disabled through the backupBloblessMode Zextras config attribute at global and server level:

zxsuite config global set attribute backupBloblessMode value true
zxsuite config server set mail.example.com attribute backupBloblessMode value true

SmartScan

What is the SmartScan?

The SmartScan is the main coherency check for the health of your backup system. It’s Smart because it operates only on accounts modified since the last SmartScan, hence improving system performance and decreasing scan time exponentially.

By default, a SmartScan is scheduled to be executed each night (if Scan Operation Scheduling is enabled in the Zextras Backup section of the Administration Zimlet). Once a week, on a day set by the user, a Purge is executed together with the SmartScan to clear Zextras Backup’s datastore from any deleted item that exceeded the retention period.

How Does it Work?

The Zextras Backup engine scans all the items on the Zimbra Datastore, looking for items modified after the last SmartScan. It updates any outdated entry and creates any item not yet present in the backup while flagging as deleted any item found in the backup and not in the Zimbra datastore.

Finally, it updates all configuration metadata in the backup, so that domains, accounts, COSs and server configurations are stored along with a dump of all LDAP data and config.

When is a SmartScan Executed?

  • When the Zextras Backup module is started.

  • Daily, if the Scan Operation Scheduling is enabled in the Administration Zimlet.

  • When the Real Time Scanner is re-enabled via the Administration Zimlet after being previously disabled.

Running a SmartScan

Starting the Scan via the Administration Zimlet

To start a SmartScan via the Administration Zimlet,

  • Open the Administration Zimlet.

  • Click the Zextras Backup tab (be sure to have a valid license).

  • Click Run Smartscan.

Starting the Scan via the CLI

To start a FullScan via the CLI, use the doSmartScan command:

Syntax:
   zxsuite backup doSmartScan [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                TYPE
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dosmartscan notifications user1@example.com,user2@example.com
Performs a smart scan and sends notifications to user1@example.com and user2@example.com

Checking the Status of a Running Scan

To check the status of a running scan via the CLI, use the monitor command:

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

Purge

What is the Backup Purge?

The Backup Purge is a cleanup operation that removes from the Backup Path any deleted item that exceeded the retention time defined by the Data Retention Policy.

How Does it Work?

The Purge engine scans the metadata of all deleted items, and it removes any item whose last update (deletion) timestamp is higher than the retention time.

If an item BLOB is still referenced by one or more valid metadata files, due to Zextras Backup’s built-in deduplication, the BLOB itself will not be deleted.

SPostfix customizations backed up by Zextras Backup also follow the backup path’s purge policies. This can be changed in the `Zextras Backup section of the Administration Zimlet by unchecking the Purge old customizations checkbox.

When is a Backup Purge Executed?

  • Weekly, if the Scan Operation Scheduling is enabled in the Administration Zimlet.

  • When manually started either via the Administration Console or the CLI.

Infinite Retention

Should the Data Retention Policy be set to 0, meaning infinite retention, the Backup Purge will immediately exit since no deleted item will ever exceed the retention time.

Running a Backup Purge

Starting the Backup Purge via the Administration Zimlet

To start a BackupPurge via the Administration Zimlet:

  • Click the Zextras Backup tab (be sure to have a valid license).

  • Click the Run Purge button in the top-right part of the UI.

Starting the Backup Purge via the CLI

To start a BackupPurge via the CLI, use the doPurge command:

Syntax:
   zxsuite backup doPurge [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME              TYPE
purgeDays(O)      String
backup_path(O)    Path

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dopurge purgeDays 30 backup_path /opt/zimbra/backup/backup_name

Checking the Status of a Running Backup Purge

To check the status of a running Purge via the CLI, use the monitor command:

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

External Backup

What is the External Backup?

The External Backup is one of the Backup Methods of Zextras Backup. It creates a snapshot of the mail system, which is ready to be used for a migration or for Disaster Recovery. Exported data is deduplicated and compressed to optimize disk utilization, transfer times and I/O rates.

How Does it Work?

The Zextras Backup engine scans all the data in the Zimbra datastore, saving all the items (deduplicated and compressed) into a folder of your choice.

Folder Permissions

The destination folder must be readable and writable by the zimbra user.

To create a valid export directory, run the following commands:

mkdir /opt/zimbra/backup/yourdestfolder

chown -R zimbra:zimbra /opt/zimbra/backup/yourdestfolder

Preparing the Migration

To minimize the risk of errors, please perform the following maintenance procedures before migrating:

  • Double check Zimbra permissions with the following command (must be ran as root): /opt/zimbra/libexec/zmfixperms --verbose --extended

  • Reindex all mailboxes.

  • Check the BLOB consistency with the zxsuite hsm doCheckBlobs utility.

Running an External Backup

Via the Administration Zimlet

To start an External Backup via the Administration Zimlet:

  • Click the Zextras Backup tab.

  • Click the Export Backup button under Import/Export to open the Export Backup wizard.

  • Enter the Destination Path in the textbox, and press Next. The software will check if the destination folder is empty and whether the 'zimbra' user has R/W permissions.

  • Select the domains you want to export, and press Next.

  • Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the Admin account and the user who started the restore procedure are notified by default.

Via the CLI

To start an External Backup via the CLI, use doExport command:

Syntax:
   zxsuite backup doExport {destination_path} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                   TYPE                  DEFAULT
destination_path(M)    Path
domains(O)             Domain Name[,..]      all
notifications(O)       Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doexport /opt/zimbra/backup/ domains example.com notifications john@example.com
Exports a backup of example.com to /opt/zimbra/backup/ and notifies john@example.com

Scheduling Script

The Zextras CLI can be used to schedule External Backup operations. This is handy when you need to keep a daily/weekly/monthly backup for corporate or legal reasons.

Restore on New Account

What is the Restore on New Account?

The Restore on New Account procedure allows you to restore the contents and preferences of a mailbox as it was in a moment in time, into a completely new account. The source account is not changed in any way, so it is possible to recover one or more deleted items in a user’s account without actually rolling back the whole mailbox. When you run this kind of restore, you can choose to hide the newly created account from the GAL as a security measure.

How Does it Work?

When a Restore on New Account starts, a new account is created (the destination account). All the items existing in the source account at the moment selected are recreated in the destination account, including the folder structure and all the user’s data. All restored items will be created in the current primary store unless the Obey HSM Policy box is checked.

When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account’s ID, not to the restored account.

Running a Restore on New Account via the Administration Zimlet

A Restore on New Account can be run in two ways.

From the Account List

Running Restore from the Accounts tab in the Zimbra Administration Console allows you to operate on users currently existing on the server.
If you need to restore a deleted user, please proceed to Restore via the Administration Zimlet.

  • Select Accounts in the left pane of the Administration Console to show the Accounts List.

  • Browse the list and click the account to be restored (Source).

  • On the top bar, press the wheel and then the `Restore ` button.

  • Select Restore on New Account as the Restore Mode and enter the name of the new account (Destination) into the text box. You can then choose whether to Hide in GAL the new account or not. When you’re done choosing, press Next.

  • Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and minute and second via two text boxes. Click Next.

  • Verify all your choice in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Pleas notice that the admin account and the user who started the restore procedure are notified by default.

Click Finish to start the restore.

Running a Restore on New Account via the CLI

To start a Restore on New Account via the CLI, use the doRestoreOnNewAccount command:

Syntax:
   zxsuite backup doRestoreOnNewAccount {source_account} {destination_account} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                       TYPE                  EXPECTED VALUES
source_account(M)          Account Name
destination_account(M)     Account Name/ID
date(M)                    Date                  `dd/MM/yyyy HH:mm:ss`|last
restore_chat_buddies(O)    Boolean               true|false
notifications(O)           Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup dorestoreonnewaccount John NewJohn `28/09/2012 10:15:10`
Restores John's account in a new account named NewJohn

Undelete Restore

What is Undelete Restore?

Undelete Restore is one of the Restore Modes available in Zextras Backup. It allows an administrator to restore all items deleted from a mailbox in a period of time and put them into a dedicated Zimbra folder inside the mailbox itself.

How Does it Work?

During an Undelete Restore, the Zextras Backup engine searches the backup datastore for items flagged as DELETED and restores them in a dedicated folder in the selected mailbox.

To deal with IMAP-deleted emails in a more comfortable way for the user, the deleted IMAP flag will now be stripped from any restored item so that the item itself is visible in the Zimbra Web Client.

Running an Undelete Restore

Via the Administration Console

  • Select `Accounts`in the left pane of the Administration Console to show the Accounts List.

  • Browse the list and click the account to be restored (Source).

  • On the top bar, press the wheel and then the `Restore ` button".

  • Select Undelete as the Restore Mode and press Next.

  • Choose the restore date-time slot. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and the minute and second via two text boxes. Click Next.

  • Verify your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the restore procedure are notified by default.

  • Click Finish to start the Restore.

Via the CLI

To start an Undelete Restore operation, use the doUndelete command:

Syntax:
   zxsuite backup doUndelete {account} {"dd/MM/yyyy HH:mm:ss"|first} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                TYPE                  EXPECTED VALUES
account(M)          Account Name
start_date(M)       Date                  `dd/MM/yyyy HH:mm:ss`|first
end_date(M)         Date                  `dd/MM/yyyy HH:mm:ss`|last
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doundelete John `08/10/2012 10:15:00` last
Performs an undelete on John's account of all items created between 08/10/2012 10:15:00 and the latest data available

External Restore

What is the External Restore?

The External Restore is one of the Restore Modes of Zextras Backup.

How Does it Work?

The External Restore adds to the current Zimbra server all the data, metadata and configuration data stored on an external backup.

The workflow of the import procedure is as follows:

PHASE1

  • ''Operation Started'' notification

  • Read Server Backup Data

  • Create empty Domains

  • Create needed COS (only those effectively used by the imported accounts)

  • Create empty DLs

  • Create empty Accounts

  • Restore all Accounts' attributes

  • Restore all Domains' attributes

  • Restore all DLs' attributes and share informations

  • ''PHASE1 Feedback'' Notification

PHASE2

  • Restore all Items

PHASE3

  • Restore all Mountpoints and Datasources

  • ''Operation Ended'' notification with complete feedback

Before You Start

If Zextras Backup is already initialized on the destination server, disable the RealTime Scanner to improve both memory usage and I/O performance.

To reduce the I/O overhead and the amount of disk space used for the migration, advanced users may tweak or disable Zimbra’s RedoLog for the duration of the import.

To further reduce the amount of disk space used, it’s possible to enable compression on your current primary volume before starting the import. If you do not wish to use a compressed primary volume after migration, it’s possible to create a new and uncompressed primary volume, set it to Current and switch the old one to Secondary. All of this can be done using the Zextras Powerstore module.

Running an External Restore

Via the Administration Zimlet

  • Click the Zextras Backup tab.

  • Click the Import Backup button under Import/Export to open the Import Backup wizard.

  • Enter the Destination Path into the text box and press Forward. The software will check if the destination folder contains a valid backup and whether the 'zimbra' user has Read permissions.

  • Select the domains you want to import and press Forward.

  • Select the accounts you want to import and press Forward.

  • Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the restore procedure are notified by default.

Via the CLI

To start an External Restore operation, use the doExternalRestore command:

Syntax:
   zxsuite backup doExternalRestore {source_path} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                          TYPE                 EXPECTED VALUES    DEFAULT
source_path(M)                Path
accounts(O)                   Account Name[,..]                       all
domains(O)                    Domain Name[,..]                        all
filter_deleted(O)             Boolean              true|false         true
skip_system_accounts(O)       Boolean              true|false         true
skip_aliases(O)               Boolean              true|false         false
skip_distribution_lists(O)    Boolean              true|false         false
provisioning_only(O)          Boolean              true|false         false
skip_coses(O)                 Boolean              true|false         false
notifications(O)              Email Address

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doexternalrestore /opt/zimbra/backup/restorePath/ accounts john@example.com,jack@example.com domains example.com filter_deleted false skip_system_accounts false
Restores the example.com domain, including all system accounts, and the john@example.com and jack@example.com accounts from a backup located in /opt/zimbra/backup/restorePath/

Speeding up the Restore through Multithreading

The concurrent_accounts parameter allows you to restore multiple accounts at the same time, thus greatly speeding up the restore process. This feature is not available via the Administration Console.

Albeit resource consumption does not grow linearly with the number of accounts restored at the same time, it can easily become taxing. Start from a low number of concurrent accounts, and raise it according to your server’s performance.
Usage example:

zxsuite backup doExternalRestore /tmp/external1 domains example0.com,example1.com concurrent_accounts 5

Restores the example0.com and example1.com domain, excluding system accounts, restoring 5 accounts at same time from a backup located in /tmp/external1

After the Restore: Message Deduplication

Running a volume-wide deduplication with the Zextras Powerstore module is highly recommended after an External Restore, since the native deduplication system might be ineffective when sequentially importing accounts.

Restore Deleted Account

What is the Restore Deleted Account?

The Restore Deleted Account procedure allows you to restore the contents and preferences of a mailbox, as it was when said mailbox was deleted, into a completely new account.

How Does it Work?

When a Restore Deleted Account starts, a new account is created (the Destination Account), and all the items existing in the source account at the moment of the deletion are recreated in the destination account, including the folder structure and all the user’s data. All restored items will be created in the current primary store unless the Obey HSM Policy box is checked.

When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account’s ID, not to the restored account.

From the Zextras Backup tab

  • Select `Zextras Backup in the left pane of the Administration Console to show the Zextras Backup tab.

  • On the top bar, push the Restore Deleted Account button.

  • Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and the minute and second via two text boxes. Click Next.

  • Browse the list and click the account to be restored (Source).

  • Enter the name of the new account (Destination) in the text box. You can then choose whether to Hide in GAL the new account or not. When you’re done choosing, press Next.

  • Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the Restore procedure are notified by default.

  • Click Finish to start the Restore.

Item Restore

What is the Item Restore?

The Item Restore is one of the Restore Modes of Zextras Backup.

How Does it Work?

A single item is restored from the backup to the owner’s account. Any type of item can be restored this way.

Running an Item Restore

Via the Administration Zimlet

Item Restore is only available through the CLI.

Via the CLI

To start an Item Restore operation, use the doItemRestore command:

Syntax:
   zxsuite backup doItemRestore {account_name} {item_id} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                 TYPE
account_name(M)      Account Name
item_id(M)           Integer
restore_folder(O)    String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doitemrestore john@example.com 4784
Restores item 4784 in the `john@example.com` mailbox
How to Obtain the itemID

The itemID is one of the metadata of an item consisting in an univoque code that identifies an item in a mailbox.

Along with all other metadata, it is stored in a file inside the items directory of the proper account in

[backup path]/accounts/[accountID]/items/[last 2 digits of itemID]/[itemID]

e.g.:

Item 2057 of account 4a217bb3-6861-4c9f-80f8-f345ae2897b5, default backup path
/opt/zimbra/backup/ng/accounts/4a217bb3-6861-4c9f-80f8-f345ae2897b5/items/57/2057

Metadata are stored in a plain text file, so tools like grep and find can be used to search for contents. To see the metadata contained in a file in a more readable format, you can use the zxsuite backup getItem command:

Syntax:
   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1
Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636
zimbra@simone:~$ zxsuite backup getitem

command getItem requires more parameters

Syntax:
   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1
Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636

''Real Life'' Example

Let’s say a user moves one item to the trash…​

2013-07-18 15:22:01,495 INFO  [btpool0-4361://localhost/service/soap/MsgActionRequest [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailop - moving Message (id=339) to Folder Trash (id=3)

…​and empties the trash.

2013-07-18 15:25:08,962 INFO  [btpool0-4364://localhost/service/soap/FolderActionRequest] [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailbox - Emptying 9 items from /Trash, removeSubfolders=true.

She then calls the Administrator to restore the deleted item. Knowing the itemID and the email address, the Administrator runs the following as the zimbra user to restore the missing item:

zxsuite backup doItemRestore user@domain.com 339

Raw Restore

The "Raw Restore" operation is a new DR-type restore operation that is compatible with both standard and blobless backup. In contrast to similar restore modes such as the External Restore, the Raw Restore operates at a lower level restoring all item metadata maintaining the original IDs for all objects restored.

The Raw Restore restores the source server’s Centralized Storage configuration. This step ensures that any data stored inside of a Centralized Storage is immediately available. If you are using local or independent third-party volumes, it is easy to move the item BLOBs from the primary storage or to restore those from a backup using the Blob Restore operation.

Differences between External Restore and Raw Restore

External Restore Raw Restore

Can be used on any Zimbra version regardless of the source

Must match the very same Zimbra version and patch level as those on the source server

Does not restore any setting

Restores Centralized Storage settings

Does not support blobless Backup Paths

Is designed for blobless Backup Paths and compatible with standard Backup Paths

Does restore item BLOBs

Does not restore item BLOBs

Restored objects get created anew

Restored objects maintain their original ID

What will be restored

  • Centralized Storage configuration and settings

  • Domains

  • Classes of Service

  • Distribution lists

  • Mailboxes

  • Mailbox preferences

  • Item metadata

What will not be restored

  • Item Blobs

Running a Raw Restore

The Raw Restore is only available through the zxsuite CLI tool:

[zimbra@mail ~]$ zxsuite backup doRawRestore
Perform a disaster recovery

Syntax:
   zxsuite backup doRawRestore {source_path} [attr1 value1 [attr2 value2...]]


PARAMETER LIST

NAME                     TYPE                  EXPECTED VALUES    DEFAULT
source_path(M)           String
notifications(O)         Email Address[,..]
skipProvisioning(O)      Boolean               true|false         false
deleteWhenConflict(O)    Boolean               true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doRawRestore /my/backup/path notifications user1@example.com,user2@example.com skipProvisioning false deleteWhenConflict false
Performs a Raw Restore without restoring provisioning or deleting a mailbox when ids are conflicting, and sends notifications to user1@example.com and user2@example.com
The disaster recovery operation does not perform blob restore, use doRestoreBlobs when needed.

Usage scenarios

Restore of a single-server infrastructure

  1. Setup a new server (install Zimbra, configure Global and Server settings).

  2. Manually create any local or independent 3rd-party volume as it was on the original server.

  3. Start a Raw Restore using to restore domains, CoS mailboxes, and item metadata (mailboxes won’t be accessible until this step completes).

  4. If the source backup was not running in Blobless Mode, run zxsuite backup doRestoreBlobs for all volumes to restore item BLOBS.

Loss of a single mailbox node in a multiserver infrastructure

  1. Add a new mailbox node to the infrastructure.

  2. Manually create any local or independent 3rd-party volume as it was on the original server.

  3. Start a Raw Restore using the skipProvisioning true parameter to restore item metadata (mailboxes won’t be accessible until this step completes).

  4. If the source backup was not running in Blobless Mode, run zxsuite backup doRestoreBlobs for all volumes to restore item BLOBS.

Loss of multiple mailbox servers in an infrastructure

  1. Setup a new empty infrastructure (all servers and roles, setting up Global and Server configuration).

  2. Delete default admin, gal, ham, and spam accounts.

  3. On all mailbox servers, manually create any local or independent 3rd-party volume as it was on the original server.

  4. On the first mailbox server, start a Raw Restore using to restore domains, CoS mailboxes, and item metadata (mailboxes won’t be accessible until this step completes).

  5. On all other mailbox servers, start a Raw Restore using the skipProvisioning true parameter to restore item metadata.

  6. Once steps 3 and 4 complete, If the source backup was not running in Blobless Mode run zxsuite backup doRestoreBlobs for all volumes on all mailbox servers to restore item BLOBS.

Disaster Recovery

The Disaster

What Can go Wrong

To classify a problem as a Disaster, one or more of the following must happened:

  • Hardware failure of one or more vital filesystems (such as / or /opt/zimbra/)

  • Contents of a vital filesystem made unusable by internal or external factors (like a careless rm * or an external intrusion)

  • Hardware failure of the physical machine hosting the Zimbra service or of the related virtualization infrastructure

  • A critical failure on a software or OS update/upgrade

Minimizing the Chances

Some suggestions to minimize the chances of a disaster:

  • Always keep vital filesystems on different drives (namely /, /opt/zimbra/ and your Zextras Backup path)

  • Use a monitoring/alerting tool for your server to become aware of problems as soon as they appear

  • Carefully plan your updates and migrations

The Recovery

How to Recover Your System

The recovery of a system is divided into 2 steps:

  • Base system recovery (OS installation and configuration, Zimbra installation and base configuration)

  • Data recovery (reimporting the last available data to the Zimbra server, including domain and user configurations, COS data and mailbox contents)

How can Zextras Backup Help with Recovery?

The Import Backup feature of Zextras Backup provides an easy and safe way to perform step 2 of a recovery.

Using the old server’s backup path as the import path allows you to restore a basic installation of Zimbra to the last valid moment of your old server.

This is just one possible Disaster Recovery scenario: more advanced scenarios and technicques are described in the Zimbra Wiki.

The Recovery Process

  • Install Zimbra on a new server and configure the Server and Global settings.

  • Install Zextras Suite on the new server.

  • Mount the backup folder of the old server onto the new one. If this is not available, use the last external backup available or the latest copy of either.

  • Begin an External Restore on the new server using the following CLI command:

zxsuite backup doExternalRestore /path/to/the/old/store

  • The External Restore operation will immediately create the domains, accounts and distribution lists, so as soon as the first part of the Restore is completed (check your Zextras Suite Notifications), the system will be ready for your users. Emails and other mailbox items will be restored afterwards.

Settings and Configs

Server and Global settings are backed up but are not restored automatically. Zextras Backup’s high-level integration with Zimbra allows you to restore your data to a server with a different OS/Zimbra Release/Networking/Storage setup without any constraints other than the minimum Zimbra version required to run Zextras Suite.

Whether you wish to create a perfect copy of the old server or just take a cue from the old server’s settings to adapt those to a new environment, Zextras Backup comes with a very handy CLI command: getServerConfig.

zimbra@test:~$ zxsuite backup getServerConfig
command getServerConfig requires more parameters


Syntax:
   zxsuite backup getServerConfig {standard|customizations} [attr1 value1 [attr2 value2...


PARAMETER LIST


NAME              TYPE               EXPECTED VALUES                       DEFAULT
type(M)           Multiple choice    standard|customizations
date(O)           String             `dd/MM/yyyy HH:mm:ss`|"last"|"all"
backup_path(O)    Path                                                     /opt/zimbra/backup/ng/
file(O)           String             Path to backup file
query(O)          String             section/id/key
verbose(O)        String                                                   false
colors(O)         String                                                   false


(M) == mandatory parameter, (O) == optional parameter


Usage example:


zxsuite backup getserverconfig standard date last
 Display the latest backup data for Server and Global configuration.
zxsuite backup getserverconfig standard file /path/to/backup/file
 Display the contents of a backup file instead of the current server backup.
zxsuite backup getserverconfig standard date last query zimlets/com_zimbra_ymemoticons colors true verbose true
 Displays all settings for the com_zimbra_ymemoticons zimlet, using colored output and high verbosity.

Specifically, this will display the latest backed up configurations:

zxsuite backup getServerConfig standard backup_path /your/backup/path/ date last query / | less

You can change the query argument to display specific settings, e.g.

zimbra@test:~$ zxsuite backup getServerConfig standard date last backup_path /opt/zimbra/backup/ng/ query serverConfig/zimbraMailMode/test.domain.com


config date_______________________________________________________________________________________________28/02/2014 04:01:14 CET
test.domain.com____________________________________________________________________________________________________________both

The {zimbrahome}/conf/ and {zimbrahome}/postfix/conf/ directories are backed up as well:

zimbra@test:~$ zxsuite backup getServerConfig customizations date last verbose true
ATTENTION: These files contain the directories {zimbraHome}/conf/ and {zimbraHome}/postfix/conf/ compressed into a single archive.
           Restore can only be performed manually. Do it only if you know what you're doing.




        archives


                filename                                                    customizations_28_02_14#04_01_14.tar.gz
                path                                                        /opt/zimbra/backup/ng/server/
                modify date                                                 28/02/2014 04:01:14 CET

VMs and Snapshots

Thanks to the advent of highly evolved virtualization solutions in the past years, virtual machines are now the most common way to deploy server solutions such as Zimbra Collaboration Suite.

Most hypervisors feature customizable snapshot capabilities and snapshot-based VM backup systems. In case of a disaster, it’s always possible to roll back to the latest snapshot and import the missing data using the External Restore feature of Zextras Backup - using the server’s backup path as the import path.

Disaster Recovery from a Previous VM State

Snapshot-based backup systems allow you to keep a frozen copy of a VM in a valid state and rollback to it at will. To 100% ensure data consistency, it’s better to take snapshot copies of switched off VMs, but this is not mandatory.

When using these kinds of systems, it’s vital to make sure that the Backup Path isn’t either part of the snapshot (e.g. by setting the vdisk to `Independent Persistent in VMWare ESX/i) or altered in any way when rolling back in order for the missing data to be available for import.

To perform a disaster recovery from a previous machine state with Zextras Backup, you need to:

  • Restore the last valid backup into a separate (clone) VM in an isolated network, making sure that users can’t access it and that both incoming and outgoing emails are not delivered.

  • Switch on the clone and wait for Zimbra to start.

  • Disable Zextras Backup’s RealTime Scanner.

  • Connect the Virtual Disk containing the untampered Backup Path to the clone and mount it (on a different path).

  • Start an External Restore using the Backup Path as the Import Path.

Doing so will parse all items in the Backup Path and import the missing ones, speeding up the disaster recovery. These steps can be repeated as many time as needed as long as user access and mail traffic is inhibited.

After the restore is completed, make sure that everything is functional and restore user access and mail traffic.

The Aftermath

What Now?

Should you need to restore any content from before the disaster, just initialize a new Backup Path and store the old one.

Unrestorable Items

How can I check if all of my items have been restored?

It’s very easy. Check the appropriate Operation Completed notification you received as soon as the restore operation finished. It can be viewed in the Notifications section of the Administration Zimlet, and it’s also emailed to the address you specified in the Core section of the Administration Zimlet as the Notification E-Mail recipient address.

The skipped items section contains a per-account list of unrestored items:

  [...]
  - stats -
  Restored Items: 15233
  Skipped Items:  125
  Unrestored Items: 10

  - unrestored items -
  account: account1@domain.com
  unrestored items: 1255,1369

  account: account2@domain.com
  unrestored items: 49965

  account: account14@domain.com
  unrestored items: 856,13339,45200, 45655
  [...]

Skipped Items vs. Unrestored Items

  • Skipped item: An item that has already been restored, either during the current restore or in a previous one.

  • Unrestored item: An item that has not been restored due to an issue in the restore process.

Why have some of my items not been restored?

There are different possible causes, the most common of which are:

  • Read Error: Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.

  • Broken item: Both the the raw item or the metadata file are readable by Zextras Backup but their content is broken/corrupted.

  • Invalid item: Both the the raw item or the metadata file are readable and the content is correct, but Zimbra refuses to inject the item.

How Can I Identify Unrestored Items?

There are two ways to do so: via the CLI and via the Zimbra Web Client. The first way can be used to search for the item within the backup/import path, and the second can be used to view the items in the source server.

Identifying Unrestorable Items through the CLI

The getItem CLI command can display an item and the related metadata, extracting all information from a backup path/external backup.

The syntax of the command is:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

To extract the raw data and metadata information of the item whose itemID is 49965 belonging to account2@domain.com ,also including the full dump of the item’s BLOB, the command would be:

zxsuite backup getItem account2@domain.com 49965 dump_blob true

Identifying Unrestorable Items through the Zimbra WebClient

The comma separated list of unrestored items displayed in the Operation Complete notification can be used as a search argument in the Zimbra Web Client to perform an item search.

To do so:

  • Log into the Zimbra Administration Console in the source server.

  • Use the View Mail feature to access the account containing the unrestored items.

  • In the search box, enter item: followed by the comma separated list of itemIDs.

e.g.
item: 856,13339,45200,45655

Remember that any search is executed only within the tab it is executed, so if you are running the search from the Email tab and get no results try to run the same search in the Address Book, Calendar, Tasks and Briefcase tabs

How Can I Restore Unrestored Items?

An item not being restored is a clear sign of an issue, either with the item itself or with your current Zimbra setup. In some cases, there are good chances of being able to restore an item even if it was not restored on the first try.

In the following paragraphs, you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.

Items Not Restored because of a Read Error

A dutiful distinction must be done about the read errors that can cause items not to be restored:

  • hard errors: Hardware failures and all other destructive errors that cause an unrecoverable data loss.

  • soft errors: non-destructive errors such as wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring), etc.

While there is nothing much to do about hard errors, you can prevent or mitigate soft errors by following these guidelines:

  • Run a filesystem check.

  • If using a RAID disk setup, check the array for possible issues (depending on RAID level).

  • Make sure that the 'zimbra' user has r/w access to the backup/import path, all its subfolders and all thereby contained files.

  • Carefully check the link quality of network-shared filesystems. If link quality is poor, consider transferring the data with rsync.

  • If using SSHfs to remotely mount the backup/import path, make sure to run the mount command as root using the -o allow_other option.

Items Not Restored because Identified as Broken Items

Unfortunately, this is the worst category of unrestored items in terms of salvageability.

Based on the degree of corruption of the item, it might be possible to recover either a previous state or the raw object (this is only valid for emails). To identify the degree of corruption, use the getItem CLI command:

   zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE               EXPECTED VALUES            DEFAULT
account(M)        Account Name/ID
item(M)           Integer
backup_path(O)    Path                                          /opt/zimbra/backup/ng/
dump_blob(O)      Boolean            true|false                 false
date(O)           Date               dd/mm/yyyy hh:mm:ss|all    last

(M) == mandatory parameter, (O) == optional parameter

Searching for the broken item, setting the backup_path parameter to the import path and the date parameter to all, will display all valid states for the item.

zimbra@test:~$ zxsuite backup getItem admin@example.com 24700 backup_path /mnt/import/ date all
       itemStates
               start_date                                                  12/07/2013 16:35:44
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  12/07/2013 17:04:33
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
               start_date                                                  15/07/2013 10:03:26
               type                                                        message
               deleted                                                     true
               blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

If the item is an email, you will be able to recover a standard .eml file through the following steps:

  • Identify the latest valid state

/mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
              start_date                                                  15/07/2013 10:03:26
              type                                                        message
              deleted                                                     true
              blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
  • Identify the blob path

blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

  • Use gzip to uncompress the BLOB file into an .eml file

zimbra@test:~$ gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml

zimbra@test:~$ cat /tmp/restored.eml

Return-Path: zimbra@test.example.com

Received: from test.example.com (LHLO test.example.com) (192.168.1.123)
by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Received: by test.example.com (Postfix, from userid 1001) id 4F34A120CC4;
Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
To: admin@example.com
From: admin@example.com
Subject: Service mailboxd started on test.example.com
Message-Id: <20130712143543.4F34A120CC4@test.example.com>
Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running
  • Done! You can now import the .eml file into the appropriate mailbox using your favorite client.

Items Not Restored because Identified as Invalid Items

An item is identified as Invalid when, albeit being formally correct, is discarded by Zimbra’s LMTP Validator upon injection. This is common when importing items created on an older version of Zimbra to a newer one, Validation rules are updated very often, so not all messages considered valid by a certain Zimbra version are still considered valid by a newer version.

If you experienced a lot of unrestored items during an import, it might be a good idea to momentarily disable the LMTP validator and repeat the import:

  • To disable Zimbra’s LMTP Validator, run the following command as the Zimbra user:

zmlocalconfig -e zimbra_lmtp_validate_messages=false

  • Once the import is completed, you can enable the LMTP validator running

zmlocalconfig -e zimbra_lmtp_validate_messages=true

This is a dirty workaround, as items deemed invalid by the LMTP validator might cause display or mobile synchronization errors. Use at your own risk.

doCoherencyCheck

What is the Coherency Check?

The Coherency Check performs a deeper check of a Backup Path than the one done by the SmartScan.

While the SmartScan works incrementally by only checking items that have been modified since the last SmartScan, the Coherency Check performs a thorough check of all metadata and BLOBs in the backup path.

It’s specifically designed to detect corrupted metadata and BLOBs.

How Does it Work?

The Coherency Check verifies the integrity of all metadata in the backup path and of the related BLOBs. Should any errors be found, running the check with the fixBackup option will move any orphaned or corrupted metadata/BLOB to a dedicated directory within the backup path.

When Should a Coherency Check be Executed?

  • At interval periods to make sure that everything is ok (e.g. every 3 or 6 months).

  • After a system crash.

  • After the filesystem or storage device containing the backup path experiences any issue.

Should the SmartScan detect a possible item corruption, a Coherency Check will be started automatically.

The Coherency Check is highly I/O consuming, so make sure to run it only during off-peak periods

Running a Coherency Check

Starting the Check via the Administration Zimlet

The Coherency Check is not available via the Administration Zimlet.

Starting the Check via the CLI

To start a Coherency Check via the CLI, use the doCoherencyCheck command:

Syntax:
   zxsuite backup doCoherencyCheck {backup_path} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                TYPE                    EXPECTED VALUES    DEFAULT
backup_path(M)      Path
accounts(O)         Account Name/ID[,..]                       all
checkZimbra(O)      Boolean                 true|false         false
fixBackup(O)        Boolean                 true|false         false
notifications(O)    Email Address[,..]

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ accounts jack@exmaple.com,john@exmaple.com
Performs a coherency check on /opt/zimbra/backup/ng/ for Jack's and John's accounts
zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ fixBackup true
Performs a coherency check on /opt/zimbra/backup/ng/ and moves corrupted backup files and blob files not referenced by any metadata out of backup

Checking the Status of a Running Check

To check the status of a running scan via the CLI, use the monitor command:

Syntax:
   zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid
operation_host(O)    String

(M) == mandatory parameter, (O) == optional parameter

Taking Additional and Offsite Backups of Zextras Backup’s Datastore

Who Watches the Watchmen?

Having backup systems is a great safety measure against data loss, but each backup system must be part of a broader backup strategy to ensure the highest possible level of reliability. The lack of a proper backup strategy gives a false sense of security, while actually turning even the best backup systems in the world into yet another breaking point.

Devising a backup strategy is no easy matter, and at some point you will most likely be confronted with the following question: What if I lose the data I backed up?. The chances of this happening ultimately only depend on how you make and manage your backups. It’s more likely that you will lose all of your backed up data if you store both your data and your backups in a single SATAII disk than if you store your backed up data on a dedicated SAN using a RAID 1+0 setup.

Here are some suggestions and best practices to improve your backup strategy by making a backup of the Backup NG’s datastore and storing it offsite.

Making an Additional Backup of Zextras Backup’s Datastore

  • Atomicity: Any transaction is committed and written to the disk only when completed.

  • Consistency: Any committed transaction is valid, and no invalid transaction will be committed and written to the disk.

  • Isolation: All transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.

  • Durability: Once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).

Due to this, it’s very easy to make a backup. The best (and easiest) way to do so is by using rsync. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport is usually much faster in transferring the data.

You won’t need to stop Zimbra or the Real Time Scanner to make an additional backup of Zextras Backup’s datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.

Storing Your Zextras Backup’s Datastore Backup Offsite

As seen in the previous section, making a backup of Zextras Backup’s Datastore is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.

To optimize your backup strategy when dealing with this kind of setup, the following best practices are recommended:

  • If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed.

  • Use the --delete options so that files that have been deleted in the source server are deleted in the destination server to avoid inconsistencies.

    • If you notice that using the --delete option takes too much time, schedule two different rsync instances: one with --delete to be run after the weekly purge and one without this option.

  • Make sure you transfer the whole folder tree recursively starting from Zextras Backup’s Backup Path. This includes server config backups and mapfiles.

  • Make sure the destination filesystem is case sensitive (just as Backup NG’s Backup Path must be).

  • If you plan to restore directly from the remote location, make sure that the zimbra user on your server has read and write permissions on the transferred data.

  • Expect to experience slowness if your transfer speed is much higher than your storage throughput (or vice versa).

Additional/Offsite Backup F.A.Q.

Why shouldn’t I use the Export Backup feature of Zextras Backup instead of rsync?

For many reasons:

  • The Export Backup feature is designed to perform migrations. It exports a snapshot that is an end in itself and was not designed to be managed incrementally. Each time an Export Backup is run, it’ll probably take just as much time as the previous one, while using rsync is much more time-efficient.

  • Being a Zextras Backup operation, any other operation started while the Export Backup is running will be queued until the Export Backup is completed.

  • An Export Backup operation has a higher impact on system resources than an rsync.

  • Should you need to stop an Export Backup operation, you won’t be able to reprise it, and you’ll need to start from scratch.

Can I use this for Disaster Recovery?

Yes. Obviously, if your Backup Path is still available. it’s better to use that, as it will restore all items and settings to the last valid state. However, should your Backup Path be lost, you’ll be able to use your additional/offsite backup.

Can I use this to restore data on the server the backup copy belongs to?

Yes, but not through the External Restore operation, since item and folder IDs are the same.

The most appropriate steps to restore data from a copy of the backup path to the very same server are as follows:

  • Stop the RealTime Scanner.

  • Change the Backup Path to the copy you wish to restore your data from.

  • Run either Restore on New Account or a Restore Deleted Account.

  • Once the restore is over, change the backup path to the original one.

  • Start the RealTime Scanner. A SmartScan will trigger to update the backup data.

Can I use this to create an Active/Standby infrastructure?

No, because the External Restore operation does not perform any deletions. By running several External Restores, you’ll end up filling up your mailboxes with unwanted content, since items deleted from the original mailbox will not be deleted on the standby server.

The External Restore operation has been designed so that accounts will be available for use as soon as the operation is started, so your users will be able to send and receive emails even if the restore is running.

Are there any other ways to do an Additional/Offsite backup of my system?

There are for sure, and some of them might even be better than the one described here. These are just guidelines that apply to the majority of cases.

Multistore Information

Zextras Backup in a Multistore Environment

Command Execution in a Multistore Environment

The new Network Administration Zimlet makes the management of multiple servers very easy. You can select a server from the Zextras Backup tab and perform all backup operations on that server, even if you are logged into the Zimbra Administration Console of another server.

Specific differences between Singlestore and Multistore environments are:

  • In a Multistore environment, Restore on New Account operations ALWAYS create the new account in the Source account’s mailbox server.

  • All operations are logged on the target server, not in the server that launched the operation.

  • If a wrong target server for an operation is chosen, Zimbra automatically proxies the operation request to the right server.

Backup and Restore

Backup and Restore in a Multistore environment will work exactly like in a Singlestore environment.

The different servers will be configured and managed separately via the Administration Zimlet, but certain operations like Live Full Scan and Stop All Operations can be 'broadcast' to all the mailstores via the zxsuite CLI using the --hostname all_servers option. This applies also to Zextras Backup settings.

Backup and Restore operations are managed as follows:

  • Smartscans can be executed on single servers via the Administration Zimlet or on multiple servers via the CLI.

  • Restores can be started from the Accounts tab in the Zimbra Admin Console, from each server tab in the Zextras Backup menu of the Administration Zimlet and via the CLI. The differences between these methods are:

Operation started from: Options

Accounts tab

The selected account’s restore is automatically started in the proper server.

Server tab

Any accounts eligible for a restore on the selected server can be chosen as the restore 'source'

CLI

Any account on any server can restored, but there is no automatic server selection.

Export and Import

Export and Import functions are those that differ the most when performed on a Multistore environment.

Here are the basic scenarios.

Export from a Singlestore and Import to a Multistore

Importing multiple accounts of a single domain to a different store will break the consistency of ALL the items that are shared from/to a mailbox on a different server.

A command in the CLI is available to fix the shares for accounts imported on different servers.

Export from a Multistore and Import to a Single or Multistore

Two different scenarios apply here:

  • Mirror import: Same number of source and destination mailstores. Each export is imported on a different server. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. The doCheckShares and doFixShares CLI commands are available to check and fix share consistency (see below).

  • Composite import: Same or different number of source and destination servers. Domains or accounts are manually imported into different servers. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. The doCheckShares and doFixShares CLI commands are available to check and fix share consistency (see below)

The doCheckShares and doFixShares Commands

The doCheckShares command will parse all share information in local accounts and report any error:

zimbra@test:~$ zxsuite help backup doCheckShares

Syntax:
   zxsuite backup doCheckShares


Usage example:

zxsuite backup doCheckShares
Check all shares on local accounts

The doFixShares will fix all share inconsistencies using a migration.

zimbra@test:~$ zxsuite help backup doFixShares

Syntax:
   zxsuite backup doFixShares {import_idmap_file}


PARAMETER LIST

NAME                    TYPE
import_idmap_file(M)    String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doFixShares idmap_file
Fixes the shares' consistency after an import according to the
mapping contained in the /opt/zimbra/backup/ng/idmap_file

Operation Queue and Queue Management

Zextras Backup’s Operation Queue

Every time a Zextras Backup operation is started, either manually or through scheduling, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).

The queue system affects the following operations:

  • External backup

  • All restore operations

  • Smartscan

Changes to Zextras Backup’s configuratito on are not enqueued and are applied immediately.

Operation Queue Management

Through the Administration Console

Viewing the Queue

To view the operation queue, access the Notifications tab in the Administration Zimlet and click the Operation Queue button.

The Administration Zimlet displays operations queued both by Zextras Backup and Zextras Powerstore in a single view. This is just a design choice, as the two queues are completely separate, meaning that one Zextras Backup operation and one Zextras Powerstore operation can be running at the same time.
Emptying the Queue

To stop the current operation and empty Zextras Backup’s operation queue, enter the Zextras Backup tab in the Administration Zimlet and click the Stop all Operations button.

Through the CLI

Viewing the Queue

To view Zextras Backup’s operation queue, use the getAllOperations command:

zimbra@server:~$ zxsuite help backup getAllOperations

Syntax:
   zxsuite backup getAllOperations [attr1 value1 [attr2 value2...


PARAMETER LIST

NAME          TYPE       EXPECTED VALUES    DEFAULT
verbose(O)    Boolean    true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup getAllOperations
Shows all running and queued operations
Emptying the Queue

To stop the current operation and empty Zextras Backup’s operation queue, use the doStopAllOperations command:

zimbra@mail:~$ zxsuite help backup doStopAllOperations

Syntax:
   zxsuite backup doStopAllOperations


Usage example:

zxsuite backup doStopAllOperations
Stops all running operations
Removing a Single Operation from the Queue

To stop the current operation or to remove a specific operation from the queue, use the doStopOperation command:

zimbra@mail:~$ zxsuite help backup doStopOperation

Syntax:
   zxsuite backup doStopOperation {operation_uuid}


PARAMETER LIST

NAME                 TYPE
operation_uuid(M)    Uiid

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite backup doStopOperation 30ed9eb9-eb28-4ca6-b65e-9940654b8601
Stops operation with id = 30ed9eb9-eb28-4ca6-b65e-9940654b8601

COS-level Backup Management

What is COS-level Backup Management?

COS-level Backup Management allows the administrator to disable ALL Zextras Backup functions for a whole Class of Service to lower storage usage.

How Does COS-level Backup Management Work?

What happens if I disable the Zextras Backup Module for a Class of Service?

  • The Real Time Scanner will ignore all accounts in the COS.

  • The Export Backup function WILL NOT EXPORT accounts in the COS.

  • Accounts in the COS will be treated as Deleted by the backup system. This means that after the data retention period expires, all data for such accounts will be purged from the backup store. Re-enabling the backup for a Class of Service will reset this.

How is the backup enabled/backup disabled information saved?

Disabling the backup for a Class of Service will add the following marker to the Class of Service’s Notes field: ${ZxBackup_Disabled}

While the Notes field remains fully editable and usable, changing or deleting this marker will re-enable the backup for the COS.

Incremental Migration with Zextras Backup

Description

  • This guide describes how to perform an Incremental Migration using Zextras Backup.

  • It’s specifically designed for the migration of a production environment, minimizing the downtime and aiming to be transparent for the users.

  • If correctly planned and executed, your mail system won’t suffer any downtime, and the impact on the users will be close to zero.

  • All the CLI commands in this guide must be executed as the Zimbra user unless otherwise specified.

What Will be Migrated?

  • Emails and email folders

  • Contacts and address books

  • Appointments and calendars

  • Tasks and task lists

  • Files and briefcases

  • Share informations

  • User preferences

  • User settings

  • Class of Service settings

  • Domain settings

What Will NOT be Migrated?

  • Server settings (migrated for reference but not restored)

  • Global settings (migrated for reference but not restored)

  • Customizations (Postfix, Jetty, etc…​)

  • Items moved or deleted during the process will not be moved or deleted on the destination server.

  • Preferences (e.g. passwords) changed during the process will be reset upon each import

The incremental migration is not designed to set up a server-to-server mirroring. Using multiple imports to create a mirrored copy of the source server won’t create a mirrored copy at all, since no deletions are performed by the import process.

Pre-Migration Checks

Servers

  • Source Server: Any Zimbra server can be the source of your migration, provided that it’s running Zextras Backup or Zimbra Suite Plus.

  • Destination Server: Any Zimbra server can be the destination of your migration, provided that it’s running Zextras Backup.

Storage

  • On the Source server: If Zextras Backup is not currently enabled on the source server, make sure you have an amount of free disk space comparable to the size of the /opt/zimbra/store/ folder (the exported data is compressed through the gzip algorithm, and all zimbra items are deduplicated, usually reducing the size of exported to 70% of the original size).

  • On the Destination server: Make sure you have an amount of free space greater than the size of the /opt/zimbra/store/ and of the export folders on the source server combined.

Data Transfer

While you can choose to transfer the data in any other way, rsync is our method of choice because it’s a good compromise between speed and convenience.

The main data transfer is executed while the source server is still active and functional. However, since the transfer is performed via network, carefully plan your transfer in advance so that you’ll have transferred all of your data before migrating.

Alternative Ways to Transfer Your Data

Anything spanning from the remote mount to physical move of the drive is ok as long as it suits your needs.

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
--Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

DNS

Set the TTL value of your MX record to 300 on your real DNS. This will allow a fast switch between source and destination servers.

The Setup

Step 1: Coherency Checks

To avoid any possible data-related issues, run the following checks on the source server:

Repair any error found.

Running a reindex of all mailboxes is also suggested.

Step 2: Zextras Backup Setup

Disable the Real Time Scanner on both servers:

zxsuite backup setProperty ZxBackup_RealTimeScanner false
A dedicated device for the data export is strongly recommended in order to improve the export performance and to lower the impact on the performances of the running system.

Any such device must be mounted on the /opt/zimbra/backup/ path, and the Zimbra user must have r/w permissions on it.

Step 3: Data Export (SmartScan)

Run a SmartScan on the source server:

zxsuite backup doSmartScan

All your data will be exported to the default backup path (/opt/zimbra/backup/ng/).

Pro-Tip: Single Domains Export

You can also choose to only migrate one or more domains instead of all of them. To do so, run the following command instead of the SmartScan:

zxsuite backup doExport /path/to/export/folder/ domains yourdomain.com,yourdomain2.com[..]

Mind that if you start with the SmartScan method, you’ll have to carry on the migration with this method. If you start with the Single Domains method you’ll have to carry on the migration with this method. The two methods cannot be mixed.

Data Export (SmartScan) via the Administration Zimlet

You can also choose to export your data using the Administration Zimlet.

Step 4: Data Synchronization

When you move the exported data to the destination server, make sure that the destination folder is not Zextras Backup’s backup path on the destination server to avoid any nuisances if you already use Zextras Backup or plan to do so on the destination server.

(You can skip this step if you choose to transfer your data by other means than rsync.)

Using rsync, copy the data contained in the /opt/zimbra/backup/ng/ onto a directory in the destination server (make sure the Zimbra user has r/w permissions on the folder). Use a terminal multiplexer like screen or tmux. This process might need A LOT of time depending on network speed and amount of data involved.

[run this command as Root]
rsync -avH /opt/zimbra/backup/ng/ root@desinationserver:/path/for/the/data/

Alternate Synchronization Method

While the suggested method is great for high-bandwidth situations, the first synchronization can involve a lot of data. If the rsync method is too slow, you might consider a physical move of the device (or the proper disk file if running on a virtual environment).

After moving the disk, you can remotely mount it back to the source server (e.g. via SSHFS), as the additional synchronizations needed for the migration will involve much less data. In this case, be sure to remount the device on the source server as /opt/zimbra/backup/ng/ with all due permissions.

Step 5: First Import

Import all exported data to the destination server.

zxsuite backup doExternalRestore /path/for/the/data/

Zextras Backup imports your data onto the destination server.

Do not edit or delete the backup path after this step.

First Import via the Administration Zimlet

You can also choose to import your data using the Administration Zimlet. While importing via the Administration Zimlet, be sure to remove all system accounts (like GalSync, Ham, Spam, Quarantine etc.) from the imported account list.

Step 5 (alternate): First Import for Large Migrations [ADVANCED Users Only]

If you are planning to migrate a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.

Instead of importing all of your data to the destination server, you can run a Provisioning Only import that will only create Domains, classes of service and accounts on the destination server, skipping all mailbox contents.

zxsuite backup doExternalRestore /path/for/the/data/ provisioning_only TRUE

After doing this, switch the mailflow to the new server. When the switch is completed, start the real import.

zxsuite backup doExternalRestore /path/for/the/data/

Your users will now connect to the new server where new emails will be delivered while old emails are being restored.

This approach has pros and cons.

Pros

  • Since items are only imported once and never modified or deleted afterwards, using this method will result in less discrepancies than the standard incremental migration.

  • This is the option that has less impact on the source server (e.g. good if you are in a hurry to decommission it).

Cons

  • Depending on the timing of the operation, this method has a higher impact on your users due to the fact that items are restored WHILE they work on their mailbox.

  • Since the import is done on a running system, you might notice some slowdowns.

The Situation so Far

Now the vast majority of the data has already been imported to the destination server. The source server is still active and functional, and you are ready to perform the actual migration.

The Migration

Step 6: Pre-Migration Checks

Before switching the mail flow, ALWAYS make sure that the new server is ready to become active (check your firewall, your DNS settings, your security systems, etc.)

Step 7: The Switch

At the end of this step the destination server will be active and functional.

  • Repeat step 3, step 4 and step 5 (only new data will be exported and synchronized).

  • Switch the mail flow to the new server.

  • Once NO MORE EMAILS arrive to the source server, repeat step 3, step 4 and step 5.

The Destination server is now active and functional.

Step 8: Post-Migration Checks

Run the following command to check for inconsistencies with shares:

zxsuite backup doCheckShares

Should this command report any inconsistency, this command will parse the import mapfile used as the first argument and fix any broken share:

zxsuite backup doFixShares

Mapfiles can be found in the Backup Path of the destination server as map_[source_serverID].

Step 9: Galsync

Delete any imported GalSync accounts from the Zimbra Administration Console. Then, if needed, create new GalSync accounts on all the imported domains and resync all the GalSync accounts with the following command:

zmgsautil forceSync -a galsync.randomstring@domain.com -n [resourcename]

Step 10: Message Deduplication

Running a Volume Deduplication using the Zextras Powerstore module is highly suggested after a migration.

What Now?

  • Initialize Zextras Backup on the new server to make sure all of your data is safe.

Incremental Migration FAQ

Q: Do I need a valid license to perform an incremental migration?

Yes. It can be either a trial license or a purchased one.

Q: What will be migrated?

Everything except the server configuration is migrated, including:

  • User data

  • User preferences

  • Classes of Service configurations

  • Domain configurations

Q: Will I lose my shares? Will I need to re-configure all my shares?

Absolutely not!

Q: How should I transfer the exported data between my servers?

Again, anything that suits your needs is ok. You just need to be very sure about what your needs are.

Do you need to move the data very fast? Physically moving an USB disk between your servers might not be a good idea.

Do you need to move the data in a very reliable way? Mounting the export folder via SSHFS to the destination server might not be a good idea if your internet connection is sloppy.

Zextras Backup CLI

zxsuite backup

Table 1. zxsuite backup
Command Description Usage

doAccountScan

Perform a Smart Scan on a single account

zxsuite backup doAccountScan {account} [attr1 value1 [attr2 value2…​]]

doBackupCluster

Perform a backup of Cluster

zxsuite backup doBackupCluster

doBackupLDAP

Perform a backup of LDAP

zxsuite backup doBackupLDAP

doBackupServerCustomizations

Perform a backup of customizations

zxsuite backup doBackupServerCustomizations

doCheckShares

Check all shares on local accounts

zxsuite backup doCheckShares

doCoherencyCheck

Check backup coherency

zxsuite backup doCoherencyCheck {backup_path} [attr1 value1 [attr2 value2…​]]

doEnableDisableCOS

Enable or disable the backup for a specific COS

zxsuite backup doEnableDisableCOS {cos_name} {enable|disable}

doExport

Perform an Export limited by domains

zxsuite backup doExport {destination_path} [attr1 value1 [attr2 value2…​]]

doExternalRestore

Perform an External Restore

zxsuite backup doExternalRestore {source_path} [attr1 value1 [attr2 value2…​]]

doFixShares

Try to fix all shares on local accounts

zxsuite backup doFixShares {import_idmap_file}

doItemRestore

Restores a single item

zxsuite backup doItemRestore {account_name} {item_id} [attr1 value1 [attr2 value2…​]]

doItemSearch

Search for an item

zxsuite backup doItemSearch {account} [attr1 value1 [attr2 value2…​]]

doPurge

Perform a Purge Operation

zxsuite backup doPurge [attr1 value1 [attr2 value2…​]]

doRawRestore

Perform a raw restore [BETA]

zxsuite backup doRawRestore {source_path} [attr1 value1 [attr2 value2…​]]

doRestartService

restart a given service

zxsuite backup doRestartService {service_name}

doRestoreBlobs

Start a restore blobs operation which tries to restore broken zimbra blobs

zxsuite backup doRestoreBlobs {volume_id} [attr1 value1 [attr2 value2…​]]

doRestoreOnNewAccount

Perform a Restore on New Account

zxsuite backup doRestoreOnNewAccount {Account name or id} {destination_account} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2…​]]

doSmartScan

Perform a Smart Scan

zxsuite backup doSmartScan [attr1 value1 [attr2 value2…​]]

doStartService

start a given service

zxsuite backup doStartService {service_name}

doStopAllOperations

Stops all running operations and empties the operations queue

zxsuite backup doStopAllOperations

doStopOperation

Stop a single running operation

zxsuite backup doStopOperation {operation_uuid}

doStopService

stop a given service

zxsuite backup doStopService {service_name}

doUndelete

Perform an Undelete Restore

zxsuite backup doUndelete {account} {"dd/MM/yyyy HH:mm:ss"|first} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2…​]]

getAccountInfo

Shows an account’s information

zxsuite backup getAccountInfo {account} [attr1 value1 [attr2 value2…​]]

getAllOperations

Displays all running and queued operations

zxsuite backup getAllOperations [attr1 value1 [attr2 value2…​]]

getAvailableAccounts

Displays all accounts available for restore

zxsuite backup getAvailableAccounts [attr1 value1 [attr2 value2…​]]

getAvailableDomains

Displays all domains available for restore

zxsuite backup getAvailableDomains {dd/MM/yyyy HH:mm:ss|"last"} {backup_path}

getBackupInfo

Displays informations about the backup system

zxsuite backup getBackupInfo [attr1 value1 [attr2 value2…​]]

getCOSBackupStatus

Displays the backup status of all COS

zxsuite backup getCOSBackupStatus [attr1 value1 [attr2 value2…​]]

getItem

Display an item in a human-readable format.

zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2…​]]

getMap

Show binary Map<String, String> object as human readable table

zxsuite backup getMap {file_path}

getProperty

Get configuration properties

zxsuite backup getProperty [attr1 value1 [attr2 value2…​]]

getServerConfig

Provides a list of stored server configs

zxsuite backup getServerConfig {standard|customizations} [attr1 value1 [attr2 value2…​]]

getServices

show current status of all services for this module

zxsuite backup getServices

monitor

Monitor a running operation

zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2…​]]

setProperty

Set configuration property

zxsuite backup setProperty {property_name} {property_value}