Backup Advanced Techniques#

This section contains various advanced techniques to deal with problems that may rarely arise using Carbonio, starting from a disaster recovery scenario: how to prevent it from happening and fixing it you find yourself in such a situation, then continuing with the introduction a number of possibilities to recover single lost items, and other advanced backup management options.

Disaster Recovery#

To classify a problem as a Disaster, one or more of the following must happen:

Hardware failure of one or more vital volumes or filesystems (such as / or /opt/zextras/)
Content of a vital filesystem made unusable due to internal or external factors (like a careless rm *, an external intrusion, a wrong file being overwritten, or other)
Issues in the infrastructure hosting Carbonio, either a broken piece of hardware or failing hypervisor including snapshots
A critical failure on a third-party software or during OS update/upgrade, for example a tainted kernel.

In other words, in a disaster scenario you face a data loss and need to either replace some hardware component or repair the virtualisation infrastructure, and repair or reinstall the system.

Minimise the Chances#

Preventing a disaster scenario may not be as easy task, because one of the failures mentioned in the previous section is unpredictable and may happen at any time.

However, to minimise the chances of a disaster, there are a number of good practices we can suggest, including the following:

Always keep vital filesystems on different volumes (namely / /opt/zextras/, or your Carbonio Backup Path)
Use a monitoring/alerting tool for your server to become aware of problems as soon as they appear
Carefully plan your updates and migrations
Consider implementing redundancy to replicate the services provided by Carbonio
Maintain multiple copies of the backups: for more information, please refer to section Taking Additional and Offsite Backups of Carbonio Backup’s Volume

The Recovery Process#

If, despite all your efforts, you end up facing a disaster scenario, you can proceed to recover the system in few steps:

Installation of the base system: Operating System installation (not covered in this guide)
Installation and bootstrap of Carbonio, covered in section How to install Roles
Recovery of data (reimporting the last available data to the Carbonio server, including domain and user configurations, COS data and mailbox contents)
Recovery of Settings and Configurations

The third point can take advantage of the Import Backup feature of Carbonio Backup, which provides an easy and safe way to recover from a disaster scenario.

Indeed, you can use the Backup Path of the old server as the import path. allows you to restore a basic installation of Carbonio to the last valid moment of your old server.

There are two equivalent procedures to data recovery, both recovering Carbonio to a given status saved in a backup:

A generic one, which can always be used
A VM-based that takes advantage of hypervisor’s snapshot feature

Data Recovery#

The Recovery procedure is quite easy and requires to carry out these steps. The time required to successfully complete the recovery depends on the number and type of items to be recovered and can not be easily quantified in advance.

Install Carbonio on a new server and configure the Server and Global settings.
Mount the backup folder of the old server onto the new one. If this is not available, use the last external backup available or the latest copy of either.
Begin an External Restore on the new server using the following CLI command
```
zextras$ carbonio backup doExternalRestore /path/to/the/old/store
```

The External Restore operation will immediately create the domains, accounts and distribution lists, so as soon as the first part of the Restore is completed (check your Carbonio Notifications), the system will be ready for your users. E-mails and other mailbox items will be restored right after.

Settings and Configs#

Server and Global settings are backed up but are not restored automatically. Carbonio Backup’s high-level integration with Carbonio allows you to restore your data to a server with a different OS/Carbonio Release/Networking/Storage setup without any constraints other than the minimum Carbonio version required.

Whether you wish to create a perfect copy of the old server or just take a cue from the old server’s settings to adapt those to a new environment, Carbonio Backup comes with a very handy CLI command.

Basic Usage Examples

zextras$ carbonio backup getserverconfig standard date last

Display the latest backup data for Server and Global configuration.

zextras$ carbonio backup getserverconfig standard file /path/to/backup/file

Display the contents of a backup file instead of the current server backup.

zextras$ carbonio backup getServerConfig standard backup_path /your/backup/path/ date last query / | less

Display the latest backed up configurations, using a pipe to show one page of output at a time.

Advanced Usage

Change the query argument to display specific settings

zextras$ carbonio backup getServerConfig standard date last backup_path /opt/zextras/backup/mail.example/ query serverConfig/zimbraMailMode/mail.example.com

config date_______________________________________________________________________________________________28/12/2022 15:14:29 CET
mail.example.com____________________________________________________________________________________________________________both

Use the verbose true parameter to show more details; for example, that the /opt/zextras/conf/ and /opt/zextras/postfix/conf/ directories are backed up as well.

zextras$ carbonio backup getServerConfig customizations date last verbose true
ATTENTION: These files contain the directories /opt/zextras//conf/ and /opt/zextras/postfix/conf/ compressed into a single archive.
Restore can only be performed manually. Do it only if you know what you're doing.

archives
   filename                                                    customizations_28_12_22#04_01_14.tar.gz
   path                                                        /opt/zextras/backup/ng/server/
   modify date                                                 28/12/2022:01:14 CET

Recovery from VMs and Snapshots#

Nowadays, one of most useful features of hypervisors are customisable snapshot capabilities and snapshot-based VM backup systems. In case of a disaster, it’s always possible to roll back to the latest snapshot and import the missing data using the External Restore feature of Carbonio Backup - using the server’s backup path as the import path.

Snapshot-based backup systems allow you to keep a frozen copy of a VM in a valid state and rollback to it at will. To ensure full data consistency, it’s better to take snapshot copies of switched off VMs, but this is not mandatory.

Warning

When using these kinds of systems, it is vital to make sure that the Backup Path is either not part of the snapshot (you can ensure this for example by setting the vdisk to Independent Persistent in VMWare ESX/i) or altered in any way when rolling back, in order for the missing data to be available for import.

To perform a disaster recovery from a previous VM state by using Carbonio Backup, you need to:

Restore the last valid backup into a separate (clone) VM in an isolated network, making sure that users can not access it and that both incoming and outgoing emails are not delivered.
Switch to the clone and wait for Carbonio to start.
Disable Carbonio Backup’s RealTime Scanner.
Connect the Virtual Disk containing the untampered Backup Path to the clone and mount it (on a different path).
Start an External Restore using the Backup Path as the Import Path.

This procedure parses all items in the Backup Path and import the missing ones, speeding up the disaster recovery. Moreover, these steps can be repeated as many time as needed as long as user access and mail traffic is inhibited.

After the restore is completed, make sure that everything is functional and restore user access and mail traffic.

Hint

At the end of the operation, you can check that the configuration of the new mailbox is the same by running the command carbonio config dump.

The Aftermath#

Should you need to restore any content from before the disaster, just initialise a new Backup Path and store the old one.

Unrestorable Items#

Items are called unrestorable when it was not possible to restore them automatically during the recovery procedure. The reasons why this happens may vary, the most common are:

Read Error

Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.

Broken item

Both the the raw item or the metadata file are readable by Carbonio Backup but their content is broken or corrupted.

Invalid item

Both the the raw item or the metadata file are readable and the content is correct, but there is some other issue during the restore.

Check for Unrestorable Items#

When the recovery process ends, a detailed notification of Operation Completed will be sent to the administrators, which also includes a skipped items section that contains a per-account list of items that were not restored, like shown by the following excerpt:

[...]
- stats -
Restored Items: 15233
Skipped Items:  125
Unrestored Items: 10

- unrestored items -
account: account1@example.com
unrestored items: 1255,1369

account: account2@example.com
unrestored items: 49965

account: account14@example.com
unrestored items: 856,13339,45200, 45655
[...]

In the above excerpt, we denote:

Skipped items: All items that had already been restored, either during the current restore or in a previous one. This is therefore just an informative message.
Unrestored items: An item that has not been restored due to an issue in the restore process. Write down all the IDs of these items if you plan to try to recover them. They will be referred to in the reminder of this Section.

Note

Recall that an Item can be an e-mail, a file, a contact, or any other object within an account.

Identify Unrestored Items#

There are two ways to do so: via the CLI and via the Carbonio Admin Panel. The first way can be used to search for the item within the backup/import path, and the second can be used to view the items in the Web interface.

Using the Carbonio Admin Panel

The comma separated list of unrestored items displayed in the Operation Complete notification can be used as a search argument in the Carbonio Admin Panel to perform an item search.

To do so:

Log into the Carbonio Admin Panel of the source server.
Use the View Mail feature to access the account containing the unrestored items
In the search box, enter item: followed by the comma separated list of itemIDs, for example: item: 856,13339,45200,45655

Warning

Remember that any search is executed only within the current tab, so if you are running the search from the Email tab and get no results try to run the same search in the other tabs, e.g., Address Book, Calendar, Tasks.

Using the CLI

The backup getItem CLI command can display an item and the related metadata, extracting all information from a backup path/external backup.

zextras$ carbonio backup getItem {account} {item} [attr1 value1 [attr2 value2...

For example

zextras$ carbonio backup getItem account2@example.com 49965 dump blob true

Extract the raw data and metadata information of the item whose itemID is 49965 belonging to account2@example.com ,also including the full dump of the item’s BLOB

Restore Unrestored Items#

An item not being restored is a clear sign of an issue, either with the item itself or with your current Carbonio setup. In some cases, there are good chances of being able to restore an item even if it was not restored on the first try.

In the following paragraphs, you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.

Items Not Restored Because of Read Errors

Read errors that can lead to items not to be restored are of two types:

Hard errors: Hardware failures and all other destructive errors that cause an unrecoverable data loss.
Soft errors: non-destructive errors, including for example wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring), and so on.

While there is nothing much to do about hard errors, you can prevent or mitigate soft errors by following these guidelines:

Run a filesystem check
If using a RAID disk setup, check the array for possible issues (depending on RAID level)
Make sure that the zextras user has r/w access to the backup/import path, all its subfolders and all thereby contained files
Carefully check the link quality of network-shared filesystems. If link quality is poor, consider transferring the data with rsync
If using SSHfs to remotely mount the backup/import path, make sure to run the mount command as root using the -o allow_other option

Items Not Restored Because Identified as Invalid Items

An item is identified as Invalid when, albeit being formally correct, is discarded by the LMTP Validator upon injection.

If you experienced a lot of unrestored items during an import, it might be a good idea to momentarily disable the LMTP validator and repeat the import:

To disable the LMTP Validator, run the following command as the zextras user.
```
zextras$ zmlocalconfig -e zimbra_lmtp_validate_messages=false
```
Once the import is completed, you can enable the LMTP validator by running
```
zextras$ zmlocalconfig -e zimbra_lmtp_validate_messages=true
```

Warning

This is a dirty workaround, as items deemed invalid by the LMTP validator might cause display or mobile synchronisation errors. Use at your own risk.

Items Not Restored Because Identified as Broken Items

Unfortunately, this is the worst category of unrestored items, and their recovery may be difficult when not impossible, depending on the degree of corruption of the item. However, it might be possible to recover either a previous state of the item or, in case of e-mails, the raw object. To identify the degree of corruption, use the backup getItem CLI command.

zextras$ carbonio backup getItem {account} {item} [attr1 value1 [attr2 value2...

Example of how to restore an item

To search for a broken item, setting the backup_path parameter to the import path and the date parameter to all, will display all valid states for the item:

zextras$ carbonio backup getItem admin@example.com 24700 backup path /mnt/import/ date all
     itemStates
             start date                                                  12/07/2013 16:35:44
             type                                                        message
             deleted                                                     true
             blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
             start date                                                  12/07/2013 17:04:33
             type                                                        message
             deleted                                                     true
             blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
             start date                                                  15/07/2013 10:03:26
             type                                                        message
             deleted                                                     true
             blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

If the item is an email, you will be able to recover a standard .eml file through the following steps:

Identify the latest valid state

From the above snippet, consider:

/mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
             start_date                                                  15/07/2013 10:03:26
             type                                                        message
             deleted                                                     true
             blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

Identify the blob path

Take the blob path from the previous step:

blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=

Use gzip to uncompress the BLOB file into an .eml file

# gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml

# cat /tmp/restored.eml

Return-Path: carbonio@test.example.com

Received: from test.example.com (LHLO test.example.com) (192.168.1.123)
by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Received: by test.example.com (Postfix, from userid 1001) id 4F34A120CC4;
Fri, 12 Jul 2013 16:35:43 +0200 (CEST)
To: admin@example.com
From: admin@example.com
Subject: Service mailboxd started on test.example.com
Message-Id: <20130712143543.4F34A120CC4@test.example.com>
Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST)

Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running

Done! You can now import the .eml file into the appropriate mailbox using your favorite client.

Taking Additional and Offsite Backups of Carbonio Backup’s Volume#

Having backup systems is a great safety measure against data loss, but each backup system must be part of a broader backup strategy to ensure the highest possible level of reliability. The lack of a proper backup strategy gives a false sense of security, while actually turning even the best backup systems in the world into yet another breaking point.

Devising a backup strategy is no easy matter, and at some point you will most likely be confronted with the following question: “What if I lose the data I backed up?”. The chances of this happening ultimately only depend on how you make and manage your backups. For example, it’s more likely that you will lose all of your backed up data if you store both your data and your backups in a same, single SATA-II disk than if you store your backed up data on a dedicated SAN using a RAID 1+0 setup.

Here are some suggestions and best practices to improve your backup strategy by making a backup of the Backup NG’s datastore and storing it offsite.

Making an Additional Backup of Carbonio Backup’s Volume#

In order to minimise the possible loss of data, a backup can take advantage of the well-known database properties called ACID, that guarantee data validity and integrity.

ACID properties

A set of database operations that satisfy the following four properties is called a transaction and represent a single logical unit of work. A transaction guarantees the logical consistency of the data stored and, in the context of Carbonio Backup, it allows for easy data back-up and roll-back to a previous state in case of serious database problems.

Atomicity: Any transaction is committed and written to the disk only when completed.
Consistency: Any committed transaction is valid, and no invalid transaction will be committed and written to the disk.
Isolation: All transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.
Durability: Once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).

By respecting these properties, it is very easy to make a backup of the Volume and make sure of the content’s integrity and validity. The best (and easiest) way to do so is by using the rsync software, designed around an algorithm that only transfers deltas (i.e., what actually changed) instead of the whole data, and works incrementally. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport is usually much faster in transferring the data.

You will not need to stop Carbonio or the Realtime Scanner to make an additional backup of Carbonio Backup’s datastore using rsync, and, thanks to the ACID properties, you will be always able to stop the sync at any time and reprise it at a later point.

Store Additional Offsite Backups#

As seen in the previous section, making a backup of Carbonio Backup’s Volume is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.

To optimize your backup strategy when dealing with this kind of setup, the following best practices are recommended:

If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed
Use the --delete options so that files that have been deleted in the source server are deleted in the destination server to avoid inconsistencies
- If you notice that using the --delete option takes too much time, schedule two different rsync instances: one with --delete to be run after the weekly purge and one without this option
Make sure you transfer the whole folder tree recursively, starting from Carbonio Backup’s Backup Path. This includes server config backups and map files
Make sure the destination filesystem is case sensitive
If you plan to restore directly from the remote location, make sure that the zextras user on your server has read and write permissions on the transferred data
Expect to experience slowness if your transfer speed is much higher than your storage throughput (or vice versa)

Additional F.A.Q. for Offsite Backup#

Why shouldn’t I use the Export Backup feature of Carbonio Backup instead of rsync?

For many reasons:

The Export Backup feature is designed to perform migrations. It exports a snapshot that is not designed to be managed incrementally. Each time an Export Backup is run, it will probably take just as much time as the previous one, while using rsync is much more time-efficient.
Being a Carbonio Backup operation, any other operation started while the Export Backup is running will be queued until the Export Backup is completed
An Export Backup operation has a higher impact on system resources than an rsync
If you need to stop an Export Backup operation, you will not be able to reprise it, and you will need to start from scratch

Can I use an Offsite Backup for Disaster Recovery?

Yes. Obviously, if your Backup Path is still available. it’s better to use that, as it will restore all items and settings to the last valid state. However, should your Backup Path be lost, you’ll be able to use your additional/offsite backup.

Can I use an Offsite Backup to restore data on the server the backup copy belongs to?

Yes, but not through the External Restore operation, since item and folder IDs are the same.

The most appropriate steps to restore data from a copy of the backup path to the very same server are as follows:

Stop the Realtime Scanner
Change the Backup Path to the copy you wish to restore your data from
Run either Restore on New Account or a Restore Deleted Account.
Once the restore is over, change the backup path to the original one.
Start the RealTime Scanner. A SmartScan will be triggered to update the backup data.

Can I use this to create an Active/Standby infrastructure?

No, because the External Restore operation does not perform any deletions. By running several External Restores, you’ll end up filling up your mailboxes with unwanted content, since items deleted from the original mailbox will not be deleted on the standby server.

The External Restore operation has been designed so that accounts will be available for use as soon as the operation is started, so your users will be able to send and receive emails even if the restore is running.

Are there any other ways to do an Additional/Offsite backup of my system?

There are for sure, and some of them might even be better than the one described here. These are just guidelines that apply to the majority of cases.

Operation Queue and Queue Management#

Carbonio Backup’s Operation Queue#

Every time a Carbonio Backup operation is started, either manually or through scheduling, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).

The queue system affects the following operations:

si’- External backup

All restore operations
SmartScan

Changes to Carbonio Backup's configuration are not enqueued and are applied immediately.

Operation Queue Management#

It is often good to know whether there are running operation within Carbonio Backup and manage the queue: three useful CLI commands help in these situations.

View the Queue

To view all running and queued operations, use command
```
zextras$ carbonio backup getAllOperations
```
Clear the Queue

To stop all the current running operations and to empty Carbonio Backup’s operation queue, use
```
zextras$ carbonio backup doStopAllOperations
```
Remove one single operation from the queue

To remove a specific operation from the queue, use the doStopOperation command with the ID of the operation. For example, to stop operation with ID 30ed9eb9-eb28-4ca6-b65e-9940654b8601, run
```
zextras$ carbonio backup doStopOperation 30ed9eb9-eb28-4ca6-b65e-9940654b8601
```

COS-level Backup Management#

COS-level Backup Management allows the administrator to disable all Carbonio Backup functions for a whole Class of Service. In other words, all members of the COS will never be part of a backup: this allows to lower storage usage.

Disable Backup for a COS#

Since it’s currently only possible to manage the enabling and disabling of this feature on a COS by command line, to disable the backup for a given COS use command:

carbonio config set cos <COS_NAME> backupEnabled false

For example to remove backup from COS called EXTERNAL_COLLABORATORS, use

zextras$ carbonio set cos EXTERNAL_COLLABORATORS backupEnabled false

To enable again the backup, run the unset command:

zextras$ carbonio unset cos EXTERNAL_COLLABORATORS backupEnabled

Or, in alternative, explicitly set to true:

zextras$ carbonio set cos EXTERNAL_COLLABORATORS backupEnabled true

You can also check the backup status for a COS, for example for a COS called EXTERNAL_COLLABORATORS, use

zextras$ carbonio config get cos EXTERNAL_COLLABORATORS backupEnabled

When Backup is disabled, the following happens in the COS:

The RealTime Scanner will ignore all accounts
The Export Backup function will not export the accounts
Accounts will be treated as Deleted by the backup system. This means that after the data retention period expires, all data for such accounts will be purged from the backup store. Re-enabling the backup for a Class of Service will reset this behaviour to the default one and mark accounts as Active.

Disable Backup for an Account#

As in the case of COS, enabling and disabling the backup functionality for individual accounts is currently only available via command line:

zextras$ carbonio config set account <ACCOUNT_NAME> backupEnabled true/false`

For example to remove the backup for an account called , use

zextras$ carbonio config set account john@example.com backupEnabled false

To enable again the backup, run the command:

zextras$ carbonio config set account john@example.com backupEnabled true

You can also check the backup status for an account, for example for an account called , use

zextras$ carbonio config get account john@example.com backupEnabled

Backup Advanced Techniques

Contents

Backup Advanced Techniques#

Disaster Recovery#

Minimise the Chances#

The Recovery Process#

Data Recovery#

Settings and Configs#

Recovery from VMs and Snapshots#

The Aftermath#

Unrestorable Items#

Check for Unrestorable Items#

Identify Unrestored Items#

Restore Unrestored Items#

Taking Additional and Offsite Backups of Carbonio Backup’s Volume#

Making an Additional Backup of Carbonio Backup’s Volume#

Store Additional Offsite Backups#

Additional F.A.Q. for Offsite Backup#

Operation Queue and Queue Management#

Carbonio Backup’s Operation Queue#

Operation Queue Management#

COS-level Backup Management#

Disable Backup for a COS#

Disable Backup for an Account#