Backup Advanced Techniques#
This section contains various advanced techniques to deal with problems that may rarely arise using Carbonio, starting from a disaster recovery scenario: how to prevent it from happening and fixing it you find yourself in such a situation, then continuing with the introduction a number of possibilities to recover single lost items, and other advanced backup management options.
Disaster Recovery#
To classify a problem as a Disaster, one or more of the following must happen:
Hardware failure of one or more vital volumes or filesystems (such as
/
or/opt/zextras/
)Content of a vital filesystem made unusable due to internal or external factors (like a careless rm *, an external intrusion, a wrong file being overwritten, or other)
Issues in the infrastructure hosting Carbonio, either a broken piece of hardware or failing hypervisor including snapshots
A critical failure on a third-party software or during OS update/upgrade, for example a tainted kernel.
In other words, in a disaster scenario you face a data loss and need to either replace some hardware component or repair the virtualisation infrastructure, and repair or reinstall the system.
Minimise the Chances#
Preventing a disaster scenario may not be as easy task, because one of the failures mentioned in the previous section is unpredictable and may happen at any time.
However, to minimise the chances of a disaster, there are a number of good practices we can suggest, including the following:
Always keep vital filesystems on different volumes (namely
/
/opt/zextras/
, or your Carbonio Backup Path)Use a monitoring/alerting tool for your server to become aware of problems as soon as they appear
Carefully plan your updates and migrations
Consider implementing redundancy to replicate the services provided by Carbonio
Maintain multiple copies of the backups: for more information, please refer to section Taking Additional and Offsite Backups of Carbonio Backup’s Volume
The Recovery Process#
If, despite all your efforts, you end up facing a disaster scenario, you can proceed to recover the system in few steps:
Installation of the base system: Operating System installation (not covered in this guide)
Installation and bootstrap of Carbonio, covered in section How to install Roles
Recovery of data (reimporting the last available data to the Carbonio server, including domain and user configurations, COS data and mailbox contents)
Recovery of Settings and Configurations
The third point can take advantage of the Import Backup feature of Carbonio Backup, which provides an easy and safe way to recover from a disaster scenario.
Indeed, you can use the Backup Path of the old server as the import path. allows you to restore a basic installation of Carbonio to the last valid moment of your old server.
There are two equivalent procedures to data recovery, both recovering Carbonio to a given status saved in a backup:
A generic one, which can always be used
A VM-based that takes advantage of hypervisor’s snapshot feature
Data Recovery#
The Recovery procedure is quite easy and requires to carry out these steps. The time required to successfully complete the recovery depends on the number and type of items to be recovered and can not be easily quantified in advance.
Install Carbonio on a new server and configure the Server and Global settings.
Mount the backup folder of the old server onto the new one. If this is not available, use the last external backup available or the latest copy of either.
-
Begin an External Restore on the new server using the following CLI command (please also refer to Sections External Restore and External Restore from an S3 Bucket) for details, especially if you use an external Bucket.
zextras$ carbonio backup doExternalRestore /path/to/the/old/store
The External Restore operation will immediately create the domains, accounts and distribution lists, so as soon as the first part of the Restore is completed (check your Carbonio Notifications), the system will be ready for your users. E-mails and other mailbox items will be restored right after.
Settings and Configs#
Server and Global settings are backed up but are not restored automatically. Carbonio Backup’s high-level integration with Carbonio allows you to restore your data to a server with a different OS/Carbonio Release/Networking/Storage setup without any constraints other than the minimum Carbonio version required.
Whether you wish to create a perfect copy of the old server or just take a cue from the old server’s settings to adapt those to a new environment, Carbonio Backup comes with a very handy CLI command.
zextras$ carbonio backup getserverconfig standard date last
Display the latest backup data for Server and Global configuration.
zextras$ carbonio backup getserverconfig standard file /path/to/backup/file
Display the contents of a backup file instead of the current server backup.
zextras$ carbonio backup getServerConfig standard backup_path /your/backup/path/ date last query / | less
Display the latest backed up configurations, using a pipe to show one page of output at a time.
Change the query
argument to display specific settings
zextras$ carbonio backup getServerConfig standard date last backup_path /opt/zextras/backup/mail.example/ query serverConfig/zimbraMailMode/mail.example.com
config date_______________________________________________________________________________________________28/12/2022 15:14:29 CET
mail.example.com____________________________________________________________________________________________________________both
Use the verbose true
parameter to show more details; for
example, that the /opt/zextras/conf/
and
/opt/zextras/postfix/conf/
directories are backed up as
well.
zextras$ carbonio backup getServerConfig customizations date last verbose true
ATTENTION: These files contain the directories /opt/zextras//conf/ and /opt/zextras/postfix/conf/ compressed into a single archive.
Restore can only be performed manually. Do it only if you know what you're doing.
archives
filename customizations_28_12_22#04_01_14.tar.gz
path /opt/zextras/backup/ng/server/
modify date 28/12/2022:01:14 CET
Recovery from VMs and Snapshots#
Nowadays, one of most useful features of hypervisors are customisable snapshot capabilities and snapshot-based VM backup systems. In case of a disaster, it’s always possible to roll back to the latest snapshot and import the missing data using the External Restore feature of Carbonio Backup - using the server’s backup path as the import path.
Snapshot-based backup systems allow you to keep a frozen copy of a VM in a valid state and rollback to it at will. To ensure full data consistency, it’s better to take snapshot copies of switched off VMs, but this is not mandatory.
Warning
When using these kinds of systems, it is vital to make sure that the Backup Path is either not part of the snapshot (you can ensure this for example by setting the vdisk to Independent Persistent in VMWare ESX/i) or altered in any way when rolling back, in order for the missing data to be available for import.
To perform a disaster recovery from a previous VM state by using Carbonio Backup, you need to:
Restore the last valid backup into a separate (clone) VM in an isolated network, making sure that users can not access it and that both incoming and outgoing emails are not delivered.
Switch to the clone and wait for Carbonio to start.
Disable Carbonio Backup’s RealTime Scanner.
Connect the Virtual Disk containing the untampered Backup Path to the clone and mount it (on a different path).
Start an External Restore using the Backup Path as the Import Path.
This procedure parses all items in the Backup Path and import the missing ones, speeding up the disaster recovery. Moreover, these steps can be repeated as many time as needed as long as user access and mail traffic is inhibited.
After the restore is completed, make sure that everything is functional and restore user access and mail traffic.
Hint
At the end of the operation, you can check that the configuration of the new mailbox is the same by running the command carbonio config dump.
The Aftermath#
Should you need to restore any content from before the disaster, just initialise a new Backup Path and store the old one.
Unrestorable Items#
Items are called unrestorable when it was not possible to restore them automatically during the recovery procedure. The reasons why this happens may vary, the most common are:
Read Error
Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.
Broken item
Both the the raw item or the metadata file are readable by Carbonio Backup but their content is broken or corrupted.
Invalid item
Both the the raw item or the metadata file are readable and the content is correct, but there is some other issue during the restore.
Check for Unrestorable Items#
When the recovery process ends, a detailed notification of Operation Completed will be sent to the administrators, which also includes a skipped items section that contains a per-account list of items that were not restored, like shown by the following excerpt:
[...]
- stats -
Restored Items: 15233
Skipped Items: 125
Unrestored Items: 10
- unrestored items -
account: account1@example.com
unrestored items: 1255,1369
account: account2@example.com
unrestored items: 49965
account: account14@example.com
unrestored items: 856,13339,45200, 45655
[...]
In the above excerpt, we denote:
- Skipped items
-
All items that had already been restored, either during the current restore or in a previous one. This is therefore just an informative message.
- Unrestored items
-
An item that has not been restored due to an issue in the restore process. Write down all the IDs of these items if you plan to try to recover them. They will be referred to in the reminder of this Section.
Note
Recall that an Item can be an e-mail, a file, a contact, or any other object within an account.
Identify Unrestored Items#
There are two ways to do so: via the CLI and via the Carbonio Admin Panel. The first way can be used to search for the item within the backup/import path, and the second can be used to view the items in the Web interface.
Using the Carbonio Admin Panel
The comma separated list of unrestored items displayed in the
Operation Complete
notification can be used as a search
argument in the Carbonio Admin Panel to perform an item search.
To do so:
Log into the Carbonio Admin Panel of the source server.
Use the View Mail feature to access the account containing the unrestored items
In the search box, enter item: followed by the comma separated list of itemIDs, for example:
item: 856,13339,45200,45655
Warning
Remember that any search is executed only within
the current tab, so if you are running the search from the
Email
tab and get no results try to run the same search
in the other tabs, e.g., Address Book
, Calendar
,
Tasks
.
Using the CLI
The backup getItem CLI command can display an item and the related metadata, extracting all information from a backup path/external backup.
zextras$ carbonio backup getItem {account} {item} [attr1 value1 [attr2 value2...
For example
zextras$ carbonio backup getItem account2@example.com 49965 dump blob true
Extract the raw data and metadata information of the item whose itemID is 49965 belonging to account2@example.com ,also including the full dump of the item’s BLOB
Restore Unrestored Items#
An item not being restored is a clear sign of an issue, either with the item itself or with your current Carbonio setup. In some cases, there are good chances of being able to restore an item even if it was not restored on the first try.
In the following paragraphs, you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.
Items Not Restored Because of Read Errors
Read errors that can lead to items not to be restored are of two types:
- Hard errors
-
Hardware failures and all other destructive errors that cause an unrecoverable data loss.
- Soft errors
-
non-destructive errors, including for example wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring), and so on.
While there is nothing much to do about hard errors, you can prevent or mitigate soft errors by following these guidelines:
Run a filesystem check
If using a RAID disk setup, check the array for possible issues (depending on RAID level)
Make sure that the
zextras
user has r/w access to the backup/import path, all its subfolders and all thereby contained filesCarefully check the link quality of network-shared filesystems. If link quality is poor, consider transferring the data with rsync
If using SSHfs to remotely mount the backup/import path, make sure to run the mount command as root using the
-o allow_other
option
Items Not Restored Because Identified as Invalid Items
An item is identified as Invalid when, albeit being formally correct, is discarded by the LMTP Validator upon injection.
If you experienced a lot of unrestored items during an import, it might be a good idea to momentarily disable the LMTP validator and repeat the import:
-
To disable the LMTP Validator, run the following command as the
zextras
user.zextras$ zmlocalconfig -e zimbra_lmtp_validate_messages=false
-
Once the import is completed, you can enable the LMTP validator by running
zextras$ zmlocalconfig -e zimbra_lmtp_validate_messages=true
Warning
This is a dirty
workaround, as items deemed
invalid by the LMTP validator might cause display or mobile
synchronisation errors. Use at your own risk.
Items Not Restored Because Identified as Broken Items
Unfortunately, this is the worst category of unrestored items, and their recovery may be difficult when not impossible, depending on the degree of corruption of the item. However, it might be possible to recover either a previous state of the item or, in case of e-mails, the raw object. To identify the degree of corruption, use the backup getItem CLI command.
zextras$ carbonio backup getItem {account} {item} [attr1 value1 [attr2 value2...
To search for a broken item, setting the backup_path
parameter to the import path and the date
parameter to
all
, will display all valid states for the item:
zextras$ carbonio backup getItem admin@example.com 24700 backup path /mnt/import/ date all
itemStates
start date 12/07/2013 16:35:44
type message
deleted true
blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
start date 12/07/2013 17:04:33
type message
deleted true
blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
start date 15/07/2013 10:03:26
type message
deleted true
blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
If the item is an email, you will be able to recover a standard .eml
file through the following steps:
-
Identify the latest valid state
From the above snippet, consider:
/mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= start_date 15/07/2013 10:03:26 type message deleted true blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
-
Identify the
blob path
Take the blob path from the previous step:
blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
-
Use gzip to uncompress the BLOB file into an
.eml
file# gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml # cat /tmp/restored.eml Return-Path: carbonio@test.example.com Received: from test.example.com (LHLO test.example.com) (192.168.1.123) by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST) Received: by test.example.com (Postfix, from userid 1001) id 4F34A120CC4; Fri, 12 Jul 2013 16:35:43 +0200 (CEST) To: admin@example.com From: admin@example.com Subject: Service mailboxd started on test.example.com Message-Id: <20130712143543.4F34A120CC4@test.example.com> Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST) Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running
Done! You can now import the
.eml
file into the appropriate mailbox using your favorite client.
Taking Additional and Offsite Backups of Carbonio Backup’s Volume#
Having backup systems is a great safety measure against data loss, but each backup system must be part of a broader backup strategy to ensure the highest possible level of reliability. The lack of a proper backup strategy gives a false sense of security, while actually turning even the best backup systems in the world into yet another breaking point.
Devising a backup strategy is no easy matter, and at some point you will most likely be confronted with the following question: “What if I lose the data I backed up?”. The chances of this happening ultimately only depend on how you make and manage your backups. For example, it’s more likely that you will lose all of your backed up data if you store both your data and your backups in a same, single SATA-II disk than if you store your backed up data on a dedicated SAN using a RAID 1+0 setup.
Here are some suggestions and best practices to improve your backup strategy by making a backup of the Backup NG’s datastore and storing it offsite.
Making an Additional Backup of Carbonio Backup’s Volume#
In order to minimise the possible loss of data, a backup can take advantage of the well-known database properties called ACID, that guarantee data validity and integrity.
By respecting these properties, it is very easy to make a backup of
the Volume and make sure of the content’s integrity and validity. The
best (and easiest) way to do so is by using the rsync
software, designed around an algorithm that only transfers deltas
(i.e., what actually changed) instead of the whole data, and works
incrementally. Specific options and parameters depend on many
factors, such as the amount of data to be synced and the storage in
use, while connecting to an rsync
daemon instead of using a remote
shell as a transport is usually much faster in transferring the data.
You will not need to stop Carbonio or the Realtime Scanner to make an additional backup of Carbonio Backup’s datastore using rsync, and, thanks to the ACID properties, you will be always able to stop the sync at any time and reprise it at a later point.
Store Additional Offsite Backups#
As seen in the previous section, making a backup of Carbonio Backup’s Volume is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.
To optimize your backup strategy when dealing with this kind of setup, the following best practices are recommended:
If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed
-
Use the
--delete
options so that files that have been deleted in the source server are deleted in the destination server to avoid inconsistenciesIf you notice that using the
--delete
option takes too much time, schedule two different rsync instances: one with--delete
to be run after the weekly purge and one without this option
Make sure you transfer the whole folder tree recursively, starting from Carbonio Backup’s Backup Path. This includes server config backups and map files
Make sure the destination filesystem is case sensitive
If you plan to restore directly from the remote location, make sure that the
zextras
user on your server has read and write permissions on the transferred dataExpect to experience slowness if your transfer speed is much higher than your storage throughput (or vice versa)
Additional F.A.Q. for Offsite Backup#
For many reasons:
The Export Backup feature is designed to perform migrations. It exports a snapshot that is not designed to be managed incrementally. Each time an Export Backup is run, it will probably take just as much time as the previous one, while using rsync is much more time-efficient.
Being a Carbonio Backup operation, any other operation started while the Export Backup is running will be queued until the Export Backup is completed
An Export Backup operation has a higher impact on system resources than an rsync
If you need to stop an Export Backup operation, you will not be able to reprise it, and you will need to start from scratch
Yes. Obviously, if your Backup Path is still available. it’s better to use that, as it will restore all items and settings to the last valid state. However, should your Backup Path be lost, you’ll be able to use your additional/offsite backup.
Yes, but not through the External Restore operation, since item and folder IDs are the same.
The most appropriate steps to restore data from a copy of the backup path to the very same server are as follows:
Stop the Realtime Scanner
Change the Backup Path to the copy you wish to restore your data from
Run either
Restore on New Account
or aRestore Deleted Account
.Once the restore is over, change the backup path to the original one.
Start the RealTime Scanner. A SmartScan will be triggered to update the backup data.
No, because the External Restore operation does not perform any
deletions. By running several External Restores, you’ll end up
filling up your mailboxes with unwanted content, since items
deleted from the original mailbox will not be deleted on the
standby
server.
The External Restore operation has been designed so that accounts will be available for use as soon as the operation is started, so your users will be able to send and receive emails even if the restore is running.
There are for sure, and some of them might even be better than the one described here. These are just guidelines that apply to the majority of cases.
Operation Queue and Queue Management#
Carbonio Backup’s Operation Queue#
Every time a Carbonio Backup operation is started, either manually or through scheduling, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).
The queue system affects the following operations:
si’- External backup
All restore operations
SmartScan
Changes to Carbonio Backup's configuration are not enqueued and are applied immediately.
Operation Queue Management#
It is often good to know whether there are running operation within Carbonio Backup and manage the queue: three useful CLI commands help in these situations.
-
View the Queue
To view all running and queued operations, use command
zextras$ carbonio backup getAllOperations
-
Clear the Queue
To stop all the current running operations and to empty Carbonio Backup’s operation queue, use
zextras$ carbonio backup doStopAllOperations
-
Remove one single operation from the queue
To remove a specific operation from the queue, use the
doStopOperation
command with the ID of the operation. For example, to stop operation with ID 30ed9eb9-eb28-4ca6-b65e-9940654b8601, runzextras$ carbonio backup doStopOperation 30ed9eb9-eb28-4ca6-b65e-9940654b8601
COS-level Backup Management#
COS-level Backup Management allows the administrator to disable all Carbonio Backup functions for a whole Class of Service. In other words, all members of the COS will never be part of a backup: this allows to lower storage usage.
Disable Backup for a COS#
Since it’s currently only possible to manage the enabling and disabling of this feature on a COS by command line, to disable the backup for a given COS use command:
carbonio config set cos <COS_NAME> backupEnabled false
For example to remove backup from COS called EXTERNAL_COLLABORATORS, use
zextras$ carbonio config set cos EXTERNAL_COLLABORATORS backupEnabled false
To enable again the backup, run the unset command:
zextras$ carbonio config unset cos EXTERNAL_COLLABORATORS backupEnabled
Or, in alternative, explicitly set to true:
zextras$ carbonio config set cos EXTERNAL_COLLABORATORS backupEnabled true
You can also check the backup status for a COS, for example for a COS called EXTERNAL_COLLABORATORS, use
zextras$ carbonio config get cos EXTERNAL_COLLABORATORS backupEnabled
When Backup is disabled, the following happens in the COS:
The RealTime Scanner will ignore all accounts
The Export Backup function will not export the accounts
Accounts will be treated as Deleted by the backup system. This means that after the data retention period expires, all data for such accounts will be purged from the backup store. Re-enabling the backup for a Class of Service will reset this behaviour to the default one and mark accounts as Active.
Disable Backup for an Account#
As in the case of COS, enabling and disabling the backup functionality for individual accounts is currently only available via command line:
zextras$ carbonio config set account <ACCOUNT_NAME> backupEnabled true/false`
For example to remove the backup for an account called john@example.com, use
zextras$ carbonio config set account john@example.com backupEnabled false
To enable again the backup, run the command:
zextras$ carbonio config set account john@example.com backupEnabled true
You can also check the backup status for an account, for example for an account called john@example.com, use
zextras$ carbonio config get account john@example.com backupEnabled