Zextras Backup

Zextras Backup separates the items' BLOBs from the items' Metatadata:

Whether the backup is local or remote, the system will read from and write to the path in real time via the RealTime Scanner, during scheduled operations such as the SmartScan and Purge and during Restores.

Most read and write operations affect the items' metadata, which are constantly updated on every new change, while BLOBs are only written when a new item is created and during restores. On average, the metadata makes up the 10% of the total backup disk usage but the 80% of read operations.

Both local and remote Backup Paths use the same basic structure:

The Backup Path can be set both via GUI and via CLI:

While BLOBs are momentarily cached on the local disk and uploaded to the remote volume as soon as possible, metadata are stored locally in the Backup Path.

An incremental copy of all metadata to the remote volume is performed every time the scheduled SmartScan runs, and every time a SmartScan is manually executed with the deep true option.

It is also possible to force a metadata upload using the remote_metadata_upload option in the following commands:

Remote metadata can be fetched to perform a Disaster Recovery or to update/fix local metadata with the zxsuite backup retrieveMetadataFromArchive command to download the latest metadata available in the remote storage to the Backup Path.

Data is stored in third-party storages using a structure very similar to the one of the Backup Path:

The only difference in how the content is saved is that all metadata for a mailbox are compressed within a single .tar.gz file instead of being stored uncompressed and spread across multiple subfolders.

New Zextras Backup users who don’t have an initialized backup just need to run the zxsuite backup setBackupVolume S3 command to set up the S3 backup. Running it without any further argument will display the full usage message:

Existing Zextras Backup users, on the other hand, must use the zxsuite backup migrateBackupVolume S3 command to setup the S3 backup and migrate their current data to it.

In both cases the command will create a bucket if provided with all the required information, or will use an existing bucket if the bucket_configuration_id parameter is used.

While at a first glance it might seem that due to the need of a local mountpoint specifically setting up the backup for NFS or FUSE has little utility, the backend differences in metadata handling ensure a greater degree of data safety.

Splitting the high-access metadata from the BLOBs ensures that disk failures, such as when the share becomes briefly available, are better handled thanks to the local cache granting a higher backup resilience.

To backup on "Local" shares such as NFS or Fuse, first mount the share and then use the appropriate command based on your need:

Both commands only require a single argument, which is the path to the local mountpoint of the NFS/FUSE share.

The Real Time Scanner is the big innovation in Zextras Backup. Each event on the system is recorded in real time by custom application triggers, which means that it is always possible to rollback an account to a previous state. Thanks to the Real Time Scanner, all the restore modes work with split-second precision.

The Real Time Scanner reads all the events of the mail server almost real-time through custom application hooks. Then it 'replicates' the same operations on its own data structure, creating items or updating their metadata. No information is ever overwritten in the backup, so every item has its own complete history.

These requirements must be met to enable Blobless Backup Mode:

Blobless Backup Mode is storage-agnostic and can be enabled on any server or infrastructure that meets the requirements above regardless of the specific storage vendor.

Blobless Backup Mode works exactly as its default counterpart: the RealTime Scanner takes care of backing up item changes while the SmartScan manages domain/cos/account consistency, the only difference between the two is that in Blobless Backup Mode the backup contains no items of kind blob while still saving all metadata and transaction history.

It’s essential to consider that once enabled, Blobless Backup Mode affects the entire server and no blobs get backed up regardless of the target volume and HSM policies.

To date, Blobless Backup Mode is only compatible with the "Raw Restore" operation. In the future, additional restore operations will become compatible with blobless datasets.

Blobless Backup Mode can be enabled or disabled through the backupBloblessMode Zextras config attribute at global and server level:

The SmartScan is the main coherency check for the health of your backup system. It’s Smart because it operates only on accounts modified since the last SmartScan, hence improving system performance and decreasing scan time exponentially.

By default, a SmartScan is scheduled to be executed each night (if Scan Operation Scheduling is enabled in the Zextras Backup section of the Administration Zimlet). Once a week, on a day set by the user, a Purge is executed together with the SmartScan to clear Zextras Backup’s datastore from any deleted item that exceeded the retention period.

The Zextras Backup engine scans all the items on the Zimbra Datastore, looking for items modified after the last SmartScan. It updates any outdated entry and creates any item not yet present in the backup while flagging as deleted any item found in the backup and not in the Zimbra datastore.

Finally, it updates all configuration metadata in the backup, so that domains, accounts, COSs and server configurations are stored along with a dump of all LDAP data and config.

The Backup Purge is a cleanup operation that removes from the Backup Path any deleted item that exceeded the retention time defined by the Data Retention Policy.

Zextras Backup allows to define two different data retention policies - one for items and one for accounts.

The "item" data retention policy is the main policy and defines for how long deleted items will remain in the backup. It can be set both via GUI in the "Backup" section of the Zextras Administration Zimlet and via CLI by changing the ZxBackup_DataRetentionDays config key via zxsuite config.

The "account" data retention policy, on the other hand, has a broader scope and defines for how long deleted mailboxes and their content will remain in the backup. It can be set via CLI by changing the backupAccountsRetentionDays server-level config key via zxsuite config.

Should the Account Retention Policy be higher than the item deletion policy, items contained in the deleted mailbox will not be deleted until the account’s retention period is over regardless of the Item Retention Policy value.

The Purge engine scans the metadata of all deleted items, and it removes any item whose last update (deletion) timestamp is higher than the retention time.

If an item BLOB is still referenced by one or more valid metadata files, due to Zextras Backup’s built-in deduplication, the BLOB itself will not be deleted.

Postfix customizations backed up by Zextras Backup also follow the backup path’s purge policies. This can be changed in the `Zextras Backup section of the Administration Zimlet by unchecking the Purge old customizations checkbox.

Should the Data Retention Policy be set to 0, meaning infinite retention, the Backup Purge will immediately exit since no deleted item will ever exceed the retention time.

The External Backup is one of the Backup Methods of Zextras Backup. It creates a snapshot of the mail system, which is ready to be used for a migration or for Disaster Recovery. Exported data is deduplicated and compressed to optimize disk utilization, transfer times and I/O rates.

The Zextras Backup engine scans all the data in the Zimbra datastore, saving all the items (deduplicated and compressed) into a folder of your choice.

The Zextras CLI can be used to schedule External Backup operations. This is handy when you need to keep a daily/weekly/monthly backup for corporate or legal reasons.

The Restore on New Account procedure allows you to restore the contents and preferences of a mailbox as it was in a moment in time, into a completely new account. The source account is not changed in any way, so it is possible to recover one or more deleted items in a user’s account without actually rolling back the whole mailbox. When you run this kind of restore, you can choose to hide the newly created account from the GAL as a security measure.

When a Restore on New Account starts, a new account is created (the destination account). All the items existing in the source account at the moment selected are recreated in the destination account, including the folder structure and all the user’s data. All restored items will be created in the current primary store unless the Obey HSM Policy box is checked.

A Restore on New Account can be run in two ways.

To start a Restore on New Account via the CLI, use the doRestoreOnNewAccount command:

Undelete Restore is one of the Restore Modes available in Zextras Backup. It allows an administrator to restore all items deleted from a mailbox in a period of time and put them into a dedicated Zimbra folder inside the mailbox itself.

During an Undelete Restore, the Zextras Backup engine searches the backup datastore for items flagged as DELETED and restores them in a dedicated folder in the selected mailbox.

The External Restore is one of the Restore Modes of Zextras Backup.

The External Restore adds to the current Zimbra server all the data, metadata and configuration data stored on an external backup.

The workflow of the import procedure is as follows:

PHASE1

PHASE2

PHASE3

If Zextras Backup is already initialized on the destination server, disable the RealTime Scanner to improve both memory usage and I/O performance.

To reduce the I/O overhead and the amount of disk space used for the migration, advanced users may tweak or disable Zimbra’s RedoLog for the duration of the import.

To further reduce the amount of disk space used, it’s possible to enable compression on your current primary volume before starting the import. If you do not wish to use a compressed primary volume after migration, it’s possible to create a new and uncompressed primary volume, set it to Current and switch the old one to Secondary. All of this can be done using the Zextras Powerstore module.

The concurrent_accounts parameter allows you to restore multiple accounts at the same time, thus greatly speeding up the restore process. This feature is not available via the Administration Console.

Running a volume-wide deduplication with the Zextras Powerstore module is highly recommended after an External Restore, since the native deduplication system might be ineffective when sequentially importing accounts.

The Restore Deleted Account procedure allows you to restore the contents and preferences of a mailbox, as it was when said mailbox was deleted, into a completely new account.

When a Restore Deleted Account starts, a new account is created (the Destination Account), and all the items existing in the source account at the moment of the deletion are recreated in the destination account, including the folder structure and all the user’s data. All restored items will be created in the current primary store unless the Obey HSM Policy box is checked.

The Item Restore is one of the Restore Modes of Zextras Backup.

A single item is restored from the backup to the owner’s account. Any type of item can be restored this way.

Let’s say a user moves one item to the trash…​

2013-07-18 15:22:01,495 INFO  [btpool0-4361://localhost/service/soap/MsgActionRequest [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailop - moving Message (id=339) to Folder Trash (id=3)

…​and empties the trash.

2013-07-18 15:25:08,962 INFO  [btpool0-4364://localhost/service/soap/FolderActionRequest] [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailbox - Emptying 9 items from /Trash, removeSubfolders=true.

She then calls the Administrator to restore the deleted item. Knowing the itemID and the email address, the Administrator runs the following as the zimbra user to restore the missing item:

zxsuite backup doItemRestore user@domain.com 339

The Raw Restore is only available through the zxsuite CLI tool:

Restore of a single-server infrastructure

Loss of a single mailbox node in a multiserver infrastructure

Loss of multiple mailbox servers in an infrastructure

Thanks to the advent of highly evolved virtualization solutions in the past years, virtual machines are now the most common way to deploy server solutions such as Zimbra Collaboration Suite.

Most hypervisors feature customizable snapshot capabilities and snapshot-based VM backup systems. In case of a disaster, it’s always possible to roll back to the latest snapshot and import the missing data using the External Restore feature of Zextras Backup - using the server’s backup path as the import path.

It’s very easy. Check the appropriate Operation Completed notification you received as soon as the restore operation finished. It can be viewed in the Notifications section of the Administration Zimlet, and it’s also emailed to the address you specified in the Core section of the Administration Zimlet as the Notification E-Mail recipient address.

The skipped items section contains a per-account list of unrestored items:

There are different possible causes, the most common of which are:

There are two ways to do so: via the CLI and via the Zimbra Web Client. The first way can be used to search for the item within the backup/import path, and the second can be used to view the items in the source server.

An item not being restored is a clear sign of an issue, either with the item itself or with your current Zimbra setup. In some cases, there are good chances of being able to restore an item even if it was not restored on the first try.

In the following paragraphs, you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.

The Coherency Check performs a deeper check of a Backup Path than the one done by the SmartScan.

While the SmartScan works incrementally by only checking items that have been modified since the last SmartScan, the Coherency Check performs a thorough check of all metadata and BLOBs in the backup path.

It’s specifically designed to detect corrupted metadata and BLOBs.

The Coherency Check verifies the integrity of all metadata in the backup path and of the related BLOBs. Should any errors be found, running the check with the fixBackup option will move any orphaned or corrupted metadata/BLOB to a dedicated directory within the backup path.

Should the SmartScan detect a possible item corruption, a Coherency Check will be started automatically.

Having backup systems is a great safety measure against data loss, but each backup system must be part of a broader backup strategy to ensure the highest possible level of reliability. The lack of a proper backup strategy gives a false sense of security, while actually turning even the best backup systems in the world into yet another breaking point.

Devising a backup strategy is no easy matter, and at some point you will most likely be confronted with the following question: What if I lose the data I backed up?. The chances of this happening ultimately only depend on how you make and manage your backups. It’s more likely that you will lose all of your backed up data if you store both your data and your backups in a single SATAII disk than if you store your backed up data on a dedicated SAN using a RAID 1+0 setup.

Here are some suggestions and best practices to improve your backup strategy by making a backup of the Backup NG’s datastore and storing it offsite.

Due to this, it’s very easy to make a backup. The best (and easiest) way to do so is by using rsync. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport is usually much faster in transferring the data.

You won’t need to stop Zimbra or the Real Time Scanner to make an additional backup of Zextras Backup’s datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.

As seen in the previous section, making a backup of Zextras Backup’s Datastore is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.

To optimize your backup strategy when dealing with this kind of setup, the following best practices are recommended:

Why shouldn’t I use the Export Backup feature of Zextras Backup instead of rsync?

For many reasons:

Can I use this for Disaster Recovery?

Yes. Obviously, if your Backup Path is still available. it’s better to use that, as it will restore all items and settings to the last valid state. However, should your Backup Path be lost, you’ll be able to use your additional/offsite backup.

Can I use this to restore data on the server the backup copy belongs to?

Yes, but not through the External Restore operation, since item and folder IDs are the same.

The most appropriate steps to restore data from a copy of the backup path to the very same server are as follows:

Can I use this to create an Active/Standby infrastructure?

No, because the External Restore operation does not perform any deletions. By running several External Restores, you’ll end up filling up your mailboxes with unwanted content, since items deleted from the original mailbox will not be deleted on the standby server.

The External Restore operation has been designed so that accounts will be available for use as soon as the operation is started, so your users will be able to send and receive emails even if the restore is running.

Are there any other ways to do an Additional/Offsite backup of my system?

There are for sure, and some of them might even be better than the one described here. These are just guidelines that apply to the majority of cases.

Every time a Zextras Backup operation is started, either manually or through scheduling, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).

The queue system affects the following operations:

Changes to Zextras Backup’s configuratito on are not enqueued and are applied immediately.

COS-level Backup Management allows the administrator to disable ALL Zextras Backup functions for a whole Class of Service to lower storage usage.

Set the TTL value of your MX record to 300 on your real DNS. This will allow a fast switch between source and destination servers.

To avoid any possible data-related issues, run the following checks on the source server:

Repair any error found.

Running a reindex of all mailboxes is also suggested.

Disable the Real Time Scanner on both servers:

Any such device must be mounted on the /opt/zimbra/backup/ path, and the Zimbra user must have r/w permissions on it.

Run a SmartScan on the source server:

All your data will be exported to the default backup path (/opt/zimbra/backup/).

(You can skip this step if you choose to transfer your data by other means than rsync.)

Using rsync, copy the data contained in the /opt/zimbra/backup/ng/ onto a directory in the destination server (make sure the Zimbra user has r/w permissions on the folder). Use a terminal multiplexer like screen or tmux. This process might need A LOT of time depending on network speed and amount of data involved.

Import all exported data to the destination server.

Zextras Backup imports your data onto the destination server.

If you are planning to migrate a very large infrastructure where an export/import lasts for hours or even days, there is an alternative way to handle the migration from this point forward.

Instead of importing all of your data to the destination server, you can run a Provisioning Only import that will only create Domains, classes of service and accounts on the destination server, skipping all mailbox contents.

After doing this, switch the mailflow to the new server. When the switch is completed, start the real import.

Your users will now connect to the new server where new emails will be delivered while old emails are being restored.

This approach has pros and cons.

Pros

Cons

Now the vast majority of the data has already been imported to the destination server. The source server is still active and functional, and you are ready to perform the actual migration.

Before switching the mail flow, ALWAYS make sure that the new server is ready to become active (check your firewall, your DNS settings, your security systems, etc.)

At the end of this step the destination server will be active and functional.

The Destination server is now active and functional.

Run the following command to check for inconsistencies with shares:

Should this command report any inconsistency, this command will parse the import mapfile used as the first argument and fix any broken share:

Mapfiles can be found in the Backup Path of the destination server as map_[source_serverID].

Delete any imported GalSync accounts from the Zimbra Administration Console. Then, if needed, create new GalSync accounts on all the imported domains and resync all the GalSync accounts with the following command:

Running a Volume Deduplication using the Zextras Powerstore module is highly suggested after a migration.