Zextras Backup
- The Backup Path
- Backup on a Local Disk
- Backup on a Third Party Store [BETA]
- Real Time Scanner
- SmartScan
- Purge
- External Backup
- Restore on New Account
- Undelete Restore
- External Restore
- Restore Deleted Account
- Item Restore
- Disaster Recovery
- Unrestorable Items
- doCoherencyCheck
- Taking Additional and Offsite Backups of Zextras Backup’s Datastore
- Multistore Information
- Operation Queue and Queue Management
- COS-level Backup Management
- Zextras Backup CLI
The Backup Path
Zextras Backup stores its data in the so called "Backup Path", which can either be:
-
A local directory or mountpoint (default, /opt/zimbra/backup/zextras/)
-
A third party S3 volume
Metadata and Blobs
Zextras Backup separates the items' BLOBs from the items' Metatadata:
-
BLOB: a file containing the item itself, such as the headers and body of an email.
-
Metadata: a file containing all information and transactions of an item, such as the folder in which it is stored and when it was moved there.
This split is made to provide faster search and interaction times along with additional security. Both BLOBs and Metadata are stored in the same backup path, but in different subdirectories. |
Whether the backup is local or remote, the system will read from and write to the path in real time via the RealTime Scanner, during scheduled operations such as the SmartScan and Purge and during Restores.
If for any reason the defined Backup Path is not available (for example, an external USB disk has been disconnected or the network connection to the remote Backup Path is interrupted) when SmartScan starts, it will detect the missing path and report it via email to the administrator. |
Most read and write operations affect the items' metadata, which are constantly updated on every new change, while BLOBs are only written when a new item is created and during restores. On average, the metadata makes up the 10% of the total backup disk usage but the 80% of read operations.
Backup Path Structure
Both local and remote Backup Paths use the same basic structure:
/opt/zimbra/backup/zextras/ |-- accounts |-- backupstat |-- items |-- lock_a8aad4af-6b1d-4a6b-8493-95979826c548 `-- server
Name | Type | Description | Content |
---|---|---|---|
|
Directory |
Stores all Backup Metadata. |
|
|
File |
Contains the basic backup usage information, such as the first and latest SmartScan date. |
Raw data |
|
Directory |
Contains all item BLOBs. |
Compressed Blobs, stored into subdirectories named after the last two character’s of the item’s hash. |
|
File |
Marks the Backup Path as belonging to the server, and prevents other servers to use the same Backup Path. |
Always stored in the local Backup Path, even when Remote Backup is enabled |
|
Directory |
Contains server and other configs all done by the SmartScan |
Compressed files, named |
Backup of Team Database
Zextras Team is an instant messaging platform with a number of features, including file sharing, Web conferencing, and more. Since Team keeps track of everything (uploaded files, chat, and so on), its database can grow quickly to a large size: This slows down any Backup operations and is not usable for a restore operation.
For this reason, the backup of Team’s DB has been disabled by default. An Administrator may enable it, in theory, but only after having contacted beforehand a TSE (Technical Support Engineer).
Backup on a Local Disk
This is the default Zextras Backup setting, where all backup files are saved in a local directory (by default, /opt/zimbra/backup/zextras/
).
In order to be eligible to be used as the Backup Path, a directory must:
-
Be both readable and writable by the "zimbra" user.
-
Use a case sensitive filesystem.
Setting the Backup Path
The Backup Path can be set both via GUI and via CLI:
-
Via GUI: in the "Backup" section of the Zextras Administration Zimlet, under "Backup Path".
-
Via CLI: using the
zxsuite config server
command to change theZxBackup_DestPath
config key.
Backup paths are unique and not reusable. Copying a Backup Path to a new server and setting it as its current Backup Path will return an error, and forcing this in any way by tampering with the backup file will cause corruption of both old and new backup data. |
Backup on a Third Party Store [BETA]
This feature is currently in beta, usage in production environment is not endorsed |
Zextras Backup also allows to store its data on a third party S3 bucket or on an NFS/Fuse share.
The standard local Backup Path will still be used to store metadata, a copy of which will then be uploaded to S3 every night, while BLOBs will be stored in the bucket straight away.
This allows to effectively split the live data, keeping the metadata on a local disk and the BLOBs on a remote storage. Due to the fact that Metadata usually makes up the 10% of the backup space but the 90% of the total read/write operations, this design ensures a good level of performances (especially on metadata I/O heavy tasks, such as the Purge and the SmartScan) even when the largest part of the data is hosted remotely.
Even when backing up on a Third Party Store, the local Backup Path will still be used for metadata and to cache BLOB writes. |
Metadata management
While BLOBs are momentarily cached on the local disk and uploaded to the remote volume as soon as possible, metadata are stored locally in the Backup Path.
An incremental copy of all metadata to the remote volume is performed every time the scheduled SmartScan runs, and every time a SmartScan is manually executed with the deep true
option.
It is also possible to force a metadata upload using the remote_metadata_upload
option in the following commands:
-
doSmartScan
-
doAccountScan
-
doBackupServerCustomizations
-
doBackupLDAP
-
doBackupCluster
Remote metadata can be fetched to perform a Disaster Recovery or to update/fix local metadata with the zxsuite backup retrieveMetadataFromArchive
command to download the latest metadata available in the remote storage to the Backup Path.
Data stored in a Third-party storage
Data is stored in third-party storages using a structure very similar to the one of the Backup Path:
|-- accounts |-- items |-- server `-- backupstat
The only difference in how the content is saved is that all metadata for a mailbox are compressed within a single .tar.gz
file instead of being stored uncompressed and spread across multiple subfolders.
How to backup on S3
New Zextras Backup users who don’t have an initialized backup just need to run the zxsuite backup setBackupVolume S3
command to set up the S3 backup. Running it without any further argument will display the full usage message:
zxsuite backup setBackupVolume S3 [attr1 value1 [attr2 value2...]] NAME TYPE EXPECTED VALUES bucket_name(O) String Amazon AWS bucket access_key(O) String Service username secret(O) String Service password volume_prefix(O) String Prefix to the server id used in all objects keys (Volume configuration) bucket_configuration_id(O) String UUID for already existing S3 service credentials (zxsuite config global get attribute s3BucketConfigurations) region(O) String Amazon AWS Region url(O) String S3 API compatible service url (ex: s3api.service.com) prefix(O) String Prefix added to blobs keys (Bucket configuration) use_infrequent_access(O) Boolean true|false infrequent_access_threshold(O) String use_intelligent_tiering(O) Boolean true|false
Existing Zextras Backup users, on the other hand, must use the zxsuite backup migrateBackupVolume S3
command to setup the S3 backup and migrate their current data to it.
zxsuite backup migrateBackupVolume S3 PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT bucket_name(O) String Amazon AWS bucket access_key(O) String Service username secret(O) String Service password volume_prefix(O) String Prefix to the server id used in all objects keys (Volume configuration) bucket_configuration_id(O) String UUID for already existing S3 service credentials (zxsuite config global get attribute s3BucketConfigurations) region(O) String Amazon AWS Region url(O) String S3 API compatible service url (ex: s3api.service.com) prefix(O) String Prefix added to blobs keys (Bucket configuration) use_infrequent_access(O) Boolean true|false infrequent_access_threshold(O) String use_intelligent_tiering(O) Boolean true|false threads(O) Integer 10 (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup migrateBackupVolume S3 bucket_configuration_id {uuid}
In both cases the command will create a bucket if provided with all the required information, or will use an existing bucket if the bucket_configuration_id
parameter is used.
The backup migration to a third party store also copies the metadata archives, the server’s configuration, and the map files; unlike what happened in previous releases, SmartScan is not run at the end of the migration, to reduce bandwidth consumption. Moreover, if the server’s logging is set to debug level, a line for each file uploaded will be added to the log file.
While similar in concept, Zextras Backup and Zextras Powerstore buckets are not compatible with each other - if Powerstore data is stored in a bucket it is not possible to store Backup data on the same bucket and vice-versa. |
S3 Backup in a multi-mailbox environment
In multi-mailbox environments, it’s not necessary to create multiple buckets: enter the bucket configuration information when enabling the remote backup on the first server and then just use the bucket_configuration_id
and prefix
parameters to store other server’s data on a separate directory of the same storage.
Example:
On Server 1, set up the S3 backup by creating a new bucket:
zxsuite backup setBackupVolume S3 \ bucket_name "Backup Bucket" access_key a1b2c3e4 \ secret f5g6h7i8j9k0 region EuWest \ url s3api.myvendor.com volume_prefix "server1" \
After the backup is created, list all buckets and take note of the bucket_configuration_id
of the backup bucket:
zxsuite core listBuckets all
On Server 2, set up the S3 backup using the previously created bucket:
zxsuite backup setBackupVolume S3 bucket_configuration_id vw12xy34z56 volume_prefix "server 2"
How to backup on NFS/Fuse
While at a first glance it might seem that due to the need of a local mountpoint specifically setting up the backup for NFS or FUSE has little utility, the backend differences in metadata handling ensure a greater degree of data safety.
Splitting the high-access metadata from the BLOBs ensures that disk failures, such as when the share becomes briefly available, are better handled thanks to the local cache granting a higher backup resilience.
To backup on "Local" shares such as NFS or Fuse, first mount the share and then use the appropriate command based on your need:
-
No pre-existing backup:
zxsuite backup setBackupVolume Local
-
Running backup:
zxsuite backup migrateBackupVolume Local
Both commands only require a single argument, which is the path to the local mountpoint of the NFS/FUSE share.
Real Time Scanner
What is the Real Time Scanner?
The Real Time Scanner is the big innovation in Zextras Backup. Each event on the system is recorded in real time by custom application triggers, which means that it is always possible to rollback an account to a previous state. Thanks to the Real Time Scanner, all the restore modes work with split-second precision.
How Does it Work?
The Real Time Scanner reads all the events of the mail server almost real-time through custom application hooks. Then it 'replicates' the same operations on its own data structure, creating items or updating their metadata. No information is ever overwritten in the backup, so every item has its own complete history.
Managing the Real Time Scanner
Enabling the Real Time Scanner
Via the Administration Zimlet
-
Select the Zextras Backup Tab.
-
Under Real Time Scanner, press the
Enable
button.
When the Real Time Scanner is enabled for the first time or re-enabled after a stop, a SmartScan is required. A warning will be displayed after enabling the Real Time Scanner, and you will be prompted to start the SmartScan. |
Disabling the Real Time Scanner
Via the Administration Zimlet
-
Select the Zextras Backup Tab.
-
Under Real Time Scanner, press the
Disable
button.
Via the CLI
To disable the Real Time Scanner via the CLI, the
ZxBackup_RealTimeScanner
property of the Zextras Backup module must be set
to false
:
zxsuite config server set $(zmhostname) attribute ZxBackup_RealTimeScanner value TRUE
Why Should I Disable the Real Time Scanner?
The only time you should disable the Real Time Scanner is while performing an External Restore of multiple domains. This is a safety measure to avoid high load on your server. After the import, re-enable the Real Time Scanner and perform a SmartScan when prompted.
Limitations and Safety Scan
The main limitation when restoring data acquired via the Real Time Scanner is:
-
Emptied Folder - when a user uses the
Empty Folder
button in the right-click context menu
In this case, and any time Zextras Backup cannot determine the status of an item by reading the metadata saved by the Real Time Scan, an Account Scan on the given account is triggered BEFORE the restore.
This fixes any misaligned data and sanitizes the backed up metadata for the mailbox.
SmartScan
What is the SmartScan?
The SmartScan is the main coherency check for the health of your backup
system. It’s Smart
because it operates only on accounts modified since
the last SmartScan, hence improving system performance and decreasing
scan time exponentially.
By default, a SmartScan is scheduled to be executed each night (if Scan
Operation Scheduling
is enabled in the Zextras Backup section of the
Administration Zimlet). Once a week, on a day set by the user, a Purge
is executed together with the SmartScan to clear Zextras Backup’s datastore
from any deleted item that exceeded the retention period.
How Does it Work?
The Zextras Backup engine scans all the items on the Zimbra Datastore, looking for items modified after the last SmartScan. It updates any outdated entry and creates any item not yet present in the backup while flagging as deleted any item found in the backup and not in the Zimbra datastore.
Since the LDAP dump is not used by the restore operation, by default the SmartScan does not execute the dump. |
Finally, it updates all configuration metadata in the backup, so that domains, accounts, COSs and server configurations are stored along with a dump of all config.
When is a SmartScan Executed?
-
When the Zextras Backup module is started.
-
Daily, if the Scan Operation Scheduling is enabled in the Administration Zimlet.
-
When the Real Time Scanner is re-enabled via the Administration Zimlet after being previously disabled.
Running a SmartScan
Starting the Scan via the Administration Zimlet
To start a SmartScan via the Administration Zimlet,
-
Open the Administration Zimlet.
-
Click the Zextras Backup tab (be sure to have a valid license).
-
Click
Run Smartscan
.
Starting the Scan via the CLI
To start a FullScan via the CLI, use the doSmartScan
command:
Syntax: zxsuite backup doSmartScan [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE notifications(O) Email Address[,..] (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup dosmartscan notifications user1@example.com,user2@example.com Performs a smart scan and sends notifications to user1@example.com and user2@example.com
Checking the Status of a Running Scan
To check the status of a running scan via the CLI, use the monitor
command:
Syntax: zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE operation_uuid(M) Uiid operation_host(O) String (M) == mandatory parameter, (O) == optional parameter
Purge
What is the Backup Purge?
The Backup Purge is a cleanup operation that removes from the Backup
Path any deleted item that exceeded the retention time defined by the
Data Retention Policy
.
Data Retention Policy
Zextras Backup allows to define two different data retention policies - one for items and one for accounts.
The "item" data retention policy is the main policy and defines for how long deleted items will remain in the backup. It can be set both via GUI in the "Backup" section of the Zextras Administration Zimlet and via CLI by changing the ZxBackup_DataRetentionDays
config key via zxsuite config
.
The "account" data retention policy, on the other hand, has a broader scope and defines for how long deleted mailboxes and their content will remain in the backup. It can be set via CLI by changing the backupAccountsRetentionDays
server-level config key via zxsuite config
.
Should the Account Retention Policy be higher than the item deletion policy, items contained in the deleted mailbox will not be deleted until the account’s retention period is over regardless of the Item Retention Policy value.
The default value for both the retention policies is 30 days. |
How Does it Work?
The Purge engine scans the metadata of all deleted items, and it removes any item whose last update (deletion) timestamp is higher than the retention time.
If an item BLOB is still referenced by one or more valid metadata files, due to Zextras Backup’s built-in deduplication, the BLOB itself will not be deleted.
Postfix customizations backed up by Zextras Backup also follow the backup path’s purge policies. This can
be changed in the `Zextras Backup
section of the Administration Zimlet by
unchecking the Purge old customizations
checkbox.
When is a Backup Purge Executed?
-
Weekly, if the Scan Operation Scheduling is enabled in the Administration Zimlet.
-
When manually started either via the Administration Console or the CLI.
Infinite Retention
Should the Data Retention Policy
be set to 0
, meaning infinite
retention, the Backup Purge will immediately exit since no deleted item
will ever exceed the retention time.
Running a Backup Purge
Starting the Backup Purge via the Administration Zimlet
To start a BackupPurge via the Administration Zimlet:
-
Click the Zextras Backup tab (be sure to have a valid license).
-
Click the
Run Purge
button in the top-right part of the UI.
Starting the Backup Purge via the CLI
To start a BackupPurge via the CLI, use the doPurge
command:
Syntax: zxsuite backup doPurge [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE purgeDays(O) String backup_path(O) Path (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup dopurge purgeDays 30 backup_path /opt/zimbra/backup/backup_name
Checking the Status of a Running Backup Purge
To check the status of a running Purge via the CLI, use the monitor
command:
Syntax: zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE operation_uuid(M) Uiid operation_host(O) String (M) == mandatory parameter, (O) == optional parameter
External Backup
What is the External Backup?
The External Backup is one of the Backup Methods of Zextras Backup. It creates a snapshot of the mail system, which is ready to be used for a migration or for Disaster Recovery. Exported data is deduplicated and compressed to optimize disk utilization, transfer times and I/O rates.
How Does it Work?
The Zextras Backup engine scans all the data in the Zimbra datastore, saving all the items (deduplicated and compressed) into a folder of your choice.
Folder Permissions
The destination folder must be readable and writable by the zimbra user.
To create a valid export directory, run the following commands:
mkdir /opt/zimbra/backup/yourdestfolder
chown -R zimbra:zimbra /opt/zimbra/backup/yourdestfolder
Preparing the Migration
To minimize the risk of errors, please perform the following maintenance procedures before migrating:
-
Double check Zimbra permissions with the following command (must be ran as root):
/opt/zimbra/libexec/zmfixperms --verbose --extended
-
Reindex all mailboxes.
-
Check the BLOB consistency with the
zxsuite hsm doCheckBlobs
utility.
Running an External Backup
Via the Administration Zimlet
To start an External Backup via the Administration Zimlet:
-
Click the Zextras Backup tab.
-
Click the
Export Backup
button underImport/Export
to open the Export Backup wizard. -
Enter the Destination Path in the textbox, and press Next. The software will check if the destination folder is empty and whether the 'zimbra' user has R/W permissions.
-
Select the domains you want to export, and press Next.
-
Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the Admin account and the user who started the restore procedure are notified by default.
Via the CLI
To start an External Backup via the CLI, use doExport
command:
Syntax: zxsuite backup doExport {destination_path} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE DEFAULT destination_path(M) Path domains(O) Domain Name[,..] all notifications(O) Email Address[,..] (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup doexport /opt/zimbra/backup/ domains example.com notifications john@example.com Exports a backup of example.com to /opt/zimbra/backup/ and notifies john@example.com
Restore on New Account
What is the Restore on New Account?
The Restore on New Account procedure allows you to restore the contents and preferences of a mailbox as it was in a moment in time, into a completely new account. The source account is not changed in any way, so it is possible to recover one or more deleted items in a user’s account without actually rolling back the whole mailbox. When you run this kind of restore, you can choose to hide the newly created account from the GAL as a security measure.
How Does it Work?
When a Restore on New Account starts, a new account is created (the
destination account). All the items existing in the source account at
the moment selected are recreated in the destination account, including
the folder structure and all the user’s data. All restored items will be
created in the current primary store unless the Obey HSM Policy
box is
checked.
When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account’s ID, not to the restored account. |
Running a Restore on New Account via the Administration Zimlet
A Restore on New Account can be run in two ways.
From the Account List
Running Restore from the Accounts
tab in the Zimbra
Administration Console allows you to operate on users currently existing on
the server.
If you need to restore a deleted user, please proceed to Restore via
the Administration Zimlet.
-
Select
Accounts
in the left pane of the Administration Console to show the Accounts List. -
Browse the list and click the account to be restored (Source).
-
On the top bar, press the wheel and then the `Restore ` button.
-
Select
Restore on New Account
as the Restore Mode and enter the name of the new account (Destination) into the text box. You can then choose whether to Hide in GAL the new account or not. When you’re done choosing, pressNext
. -
Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and minute and second via two text boxes. Click
Next
. -
Verify all your choice in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Pleas notice that the admin account and the user who started the restore procedure are notified by default.
Click Finish
to start the restore.
Running a Restore on New Account via the CLI
To start a Restore on New Account via the CLI, use the doRestoreOnNewAccount command:
Syntax: zxsuite backup doRestoreOnNewAccount {source_account} {destination_account} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES source_account(M) Account Name destination_account(M) Account Name/ID date(M) Date `dd/MM/yyyy HH:mm:ss`|last restore_chat_buddies(O) Boolean true|false notifications(O) Email Address[,..] (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup dorestoreonnewaccount John NewJohn `28/09/2012 10:15:10` Restores John's account in a new account named NewJohn
At the end of the operation, you can check that the
configuration of the new mailbox is the same by running the command
zxsuite config dump (See cli.adoc#_zextras_config_cli)
|
Undelete Restore
What is Undelete Restore?
Undelete Restore is one of the Restore Modes available in Zextras Backup. It allows an administrator to restore all items deleted from a mailbox in a period of time and put them into a dedicated Zimbra folder inside the mailbox itself.
How Does it Work?
During an Undelete Restore, the Zextras Backup engine searches the
backup datastore for items flagged as DELETED
and restores them in
the selected mailbox. The default behaviour is to put the restored
items in a dedicated mailbox.
New behaviour
Starting from version 3.1.9, using the CLI it is possible to restore
items under the same folder from where they were deleted. All the
restored items will be tagged with a string named undelete_DD_MM_YY
,
in which DD_MM_YY represents the date when the undelete has been made.
The |
The structure of the undeleted items is preserved, with each
sub-folder restored using its latest attribute available in the time
window that is being restored. If no information is available, the
folder will be called unknown_XX
.
Corner cases
There are two points that is worth highlighting:
-
In case an item or folder for some reason had the trash bin as its first position and this was its only position, they will be restored in the trash bin.
-
Suppose you have a folder called Conference 2021, then delete all of the item it contains and rename it to Conference 2022. You later—on 15th of November 2021—carry out an Undelete Restore on the mailbox. All of the items and content will be restored under folder Conference 2021 and tagged as
undelete_15_11_21
.
To deal with IMAP-deleted emails in a more comfortable way
for the user, the deleted IMAP flag will now be stripped from any
restored item so that the item itself is visible in the Zimbra Web
Client.
|
Running an Undelete Restore
Via the Administration Console
-
Select `Accounts`in the left pane of the Administration Console to show the Accounts List.
-
Browse the list and click the account to be restored (Source).
-
On the top bar, press the wheel and then the `Restore ` button".
-
Select
Undelete
as the Restore Mode and pressNext
. -
Choose the restore date-time slot. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and the minute and second via two text boxes. Click
Next
. -
Verify your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the restore procedure are notified by default.
-
Click
Finish
to start the Restore.
Via the CLI
To start an Undelete Restore operation, use the doUndelete
command:
Syntax: zxsuite backup doUndelete {account} {"dd/MM/yyyy HH:mm:ss"|first} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES account(M) Account Name start_date(M) Date `dd/MM/yyyy HH:mm:ss`|first end_date(M) Date `dd/MM/yyyy HH:mm:ss`|last notifications(O) Email Address[,..] target_original_folder Boolean true|false (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup doUndelete John `08/10/2012 10:15:00` last Performs an undelete on John's account of all items created between 08/10/2012 10:15:00 and the latest data available zxsuite backup doUndelete John `08/10/2012 10:15:00` last target_original_folder true Performs an undelete on John's account of all items created between 08/10/2012 10:15:00 and the latest data available. The items will be restored under the same folder from which they were deleted. This is the new behaviour.
At the end of the operation, you can check that the
configuration of the new mailbox is the same by running the command
zxsuite config dump (See cli.adoc#_zextras_config_cli)
|
External Restore
How Does it Work?
The External Restore adds to the current Zimbra server all the data, metadata and configuration data stored on an external backup.
The workflow of the import procedure is as follows:
PHASE1
-
''Operation Started'' notification
-
Read Server Backup Data
-
Create empty Domains
-
Create needed COS (only those effectively used by the imported accounts)
-
Create empty DLs
-
Create empty Accounts
-
Restore all Accounts' attributes
-
Restore all Domains' attributes
-
Restore all DLs' attributes and share informations
-
''PHASE1 Feedback'' Notification
PHASE2
-
Restore all Items
PHASE3
-
Restore all Mountpoints and Datasources
-
''Operation Ended'' notification with complete feedback
Folder restore
Suppose you have created a folder called Inbox/Zextras
(which is also
its Backup Path), and later deleted from it some messages, which are
in some backup. When an External Restore is carried out, those
messages are restored, along with any existent message, in the
Inbox/Zextras
folder. In other words, since the restored folder
shares the same Backup Path with an existing folder, then the restored
messages end up there.
In more details, the following happens:
- Local folder
-
If a folder with the same path was already created by a filter, the backup folder id will be mapped to the existing folder id. Moreover, all items that were in the original folder will be restored to the same path.
- Remote mailbox
-
If a folder with that same path was already created by a filter, the mountpoint will be restored. Additionally, all items in the folder (created by the filter) are moved to the mountpoint target; also the filter to write to the restored mountpoint will be updated.
Before You Start
If Zextras Backup is already initialized on the destination server, disable the RealTime Scanner to improve both memory usage and I/O performance.
To reduce the I/O overhead and the amount of disk space used for the migration, advanced users may tweak or disable Zimbra’s RedoLog for the duration of the import.
To further reduce the amount of disk space used, it’s possible
to enable compression on your current primary volume before starting the
import. If you do not wish to use a compressed primary volume after
migration, it’s possible to create a new and uncompressed primary
volume, set it to Current
and switch the old one to Secondary
.
All of this can be done using the Zextras Powerstore module.
Running an External Restore
Via the Administration Zimlet
-
Click the Zextras Backup tab.
-
Click the
Import Backup
button underImport/Export
to open the Import Backup wizard. -
Enter the Destination Path into the text box and press Forward. The software will check if the destination folder contains a valid backup and whether the 'zimbra' user has Read permissions.
-
Select the domains you want to import and press Forward.
-
Select the accounts you want to import and press Forward.
-
Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the restore procedure are notified by default.
doExternalRestore
zxsuite backup doExternalRestore source_path [param VALUE[,VALUE]]
PARAMETER LIST
NAME |
TYPE |
EXPECTED VALUES |
DEFAULT |
source_path(M) |
Path |
||
accounts(O) |
Account Name[,..] |
all |
|
input_file(O) |
String |
||
domains(O) |
Domain Name[,..] |
all |
|
filter_deleted(O) |
Boolean |
true|false |
true |
skip_system_accounts(O) |
Boolean |
true|false |
true |
skip_aliases(O) |
Boolean |
true|false |
false |
skip_distribution_lists(O) |
Boolean |
true|false |
false |
skip_coses(O) |
Boolean |
true|false |
false |
skip_account_provisioning(O) |
Boolean |
true|false |
false |
provisioning_only(O) |
Boolean |
true|false |
false |
notifications(O) |
Email Address |
||
concurrent_accounts(O) |
Integer |
||
max_file_size(O) |
Integer |
||
restore_datasource(O) |
Boolean |
true|false |
true |
force_as_external_restore(O) |
Boolean |
true|false |
false |
blobs_archive(O) |
String |
(M) == mandatory parameter, (O) == optional parameter
zxsuite backup doExternalRestore /path/to/data/ accounts john@example.com,jack@example.com domains example.com filter_deleted false skip_system_accounts false
Restores the example.com domain, including all system accounts, and the john@example.com and jack@example.com accounts from a backup located in /path/to/data/
At the end of the operation, you can check that the
configuration of the new mailbox is the same by running the command
zxsuite config dump (See cli.adoc#_zextras_config_cli)
|
Speeding up the Restore through Multithreading
The concurrent_accounts
parameter
allows you to restore multiple accounts at the same time, thus greatly
speeding up the restore process. This feature is not available via
the Administration Console.
Albeit resource consumption does not grow linearly with the number of accounts restored at the same time, it can easily become taxing. Start from a low number of concurrent accounts, and raise it according to your server’s performance. |
Usage example: zxsuite backup doExternalRestore /tmp/external1 domains example0.com,example1.com concurrent_accounts 5 Restores the example0.com and example1.com domain, excluding system accounts, restoring 5 accounts at same time from a backup located in /tmp/external1
Restore Deleted Account
What is the Restore Deleted Account?
The Restore Deleted Account procedure allows you to restore the contents and preferences of a mailbox, as it was when said mailbox was deleted, into a completely new account.
How Does it Work?
When a Restore Deleted Account starts, a new account is created (the
Destination Account), and all the items existing in the source account at
the moment of the deletion are recreated in the destination account,
including the folder structure and all the user’s data. All restored
items will be created in the current primary store unless the Obey HSM
Policy
box is checked.
When restoring data on a new account, shared items consistency is not preserved. This is because the original share rules refer to the original account’s ID, not to the restored account. |
From the Zextras Backup tab
-
Select
`Zextras Backup
in the left pane of the Administration Console to show the Zextras Backup tab. -
On the top bar, push the
Restore Deleted Account
button. -
Choose the restore date. Day/Month/Year can be selected via a minical, the hour via a drop-down menu and the minute and second via two text boxes. Click
Next
. -
Browse the list and click the account to be restored (Source).
-
Enter the name of the new account (Destination) in the text box. You can then choose whether to Hide in GAL the new account or not. When you’re done choosing, press
Next
. -
Verify all your choices in the Operation Summary window. You can also add additional email addresses to be notified when the restore operation is finished. Please notice that the admin account and the user who started the Restore procedure are notified by default.
-
Click
Finish
to start the Restore.
Item Restore
How Does it Work?
A single item is restored from the backup to the owner’s account. Any type of item can be restored this way.
Running an Item Restore
Via the CLI
To start an Item Restore operation, use the doItemRestore
command:
Syntax: zxsuite backup doItemRestore {account_name or id} {item_id} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE account_name(M) Account Name item_id(M) Integer restore_folder(O) String (M) == mandatory parameter, (O) == optional parameter
zxsuite backup doItemRestore john@example.com 4784
zxsuite backup doItemRestore 968df11c-8f8b-429a-9f29-4503d08544b3 5923
The first command restores item 4784 in the john@example.com
mailbox; while the second restores item 5923 in the
968df11c-8f8b-429a-9f29-4503d08544b3
mailbox
How to Obtain the itemID
The itemID
is one of the metadata
of an item consisting in
an univoque code that identifies an item in a mailbox.
Along with all other metadata, it is stored in a file inside the items
directory of the proper account in
[backup path]/accounts/[accountID]/items/[last 2 digits of itemID]/[itemID]
e.g.:
Item 2057 of account 4a217bb3-6861-4c9f-80f8-f345ae2897b5, default backup path
/opt/zimbra/backup/ng/accounts/4a217bb3-6861-4c9f-80f8-f345ae2897b5/items/57/2057
Metadata are stored in a plain text file, so tools like grep
and find
can be used to search for contents. To see the metadata
contained in a file in a more readable format, you can use the zxsuite
backup getItem
command:
Syntax: zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT account(M) Account Name/ID item(M) Integer backup_path(O) Path /opt/zimbra/backup/ng/ dump_blob(O) Boolean true|false false date(O) Date dd/mm/yyyy hh:mm:ss|all last (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1 Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636 zimbra@simone:~$ zxsuite backup getitem command getItem requires more parameters Syntax: zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT account(M) Account Name/ID item(M) Integer backup_path(O) Path /opt/zimbra/backup/ng/ dump_blob(O) Boolean true|false false date(O) Date dd/mm/yyyy hh:mm:ss|all last (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup getitem a7300a00-56ec-46c3-9773-c6ef7c4f3636 1 Shows item with id = 1 belonging to account a7300a00-56ec-46c3-9773-c6ef7c4f3636
''Real Life'' Example
Let’s say a user moves one item to the trash…
2013-07-18 15:22:01,495 INFO [btpool0-4361://localhost/service/soap/MsgActionRequest [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailop - moving Message (id=339) to Folder Trash (id=3)
…and empties the trash.
2013-07-18 15:25:08,962 INFO [btpool0-4364://localhost/service/soap/FolderActionRequest] [name=user@domain.com;mid=2538;oip=258.236.789.647;ua=zclient/7.2.4_GA_2900;] mailbox - Emptying 9 items from /Trash, removeSubfolders=true.
She then calls the Administrator to restore the deleted item.
Knowing the itemID and the email address, the Administrator runs the following as the zimbra
user to restore the missing item:
zxsuite backup doItemRestore user@domain.com 339
Disaster Recovery
The Disaster
What Can go Wrong
To classify a problem as a Disaster
, one or more of the following
must happened:
-
Hardware failure of one or more vital filesystems (such as / or /opt/zimbra/)
-
Contents of a vital filesystem made unusable by internal or external factors (like a careless rm * or an external intrusion)
-
Hardware failure of the physical machine hosting the Zimbra service or of the related virtualization infrastructure
-
A critical failure on a software or OS update/upgrade
Minimizing the Chances
Some suggestions to minimize the chances of a disaster:
-
Always keep vital filesystems on different drives (namely /, /opt/zimbra/ and your Zextras Backup path)
-
Use a monitoring/alerting tool for your server to become aware of problems as soon as they appear
-
Carefully plan your updates and migrations
The Recovery
How to Recover Your System
The recovery of a system is divided into 2 steps:
-
Base system recovery (OS installation and configuration, Zimbra installation and base configuration)
-
Data recovery (reimporting the last available data to the Zimbra server, including domain and user configurations, COS data and mailbox contents)
How can Zextras Backup Help with Recovery?
The Import Backup
feature of Zextras Backup provides an easy and safe way
to perform step 2 of a recovery.
Using the old server’s backup path as the import path allows you to restore a basic installation of Zimbra to the last valid moment of your old server.
This is just one possible Disaster Recovery scenario: more advanced scenarios and technicques are described in the Zimbra Wiki.
The Recovery Process
-
Install Zimbra on a new server and configure the Server and Global settings.
-
Install Zextras Suite on the new server.
-
Mount the backup folder of the old server onto the new one.
-
If this is not available, use the last external backup available;
-
If your backup is on a third party storage, see the box below;
-
-
Begin an External Restore on the new server using the following CLI command:
zxsuite backup doExternalRestore /path/to/the/old/store
-
The External Restore operation will immediately create the domains, accounts and distribution lists, so as soon as the first part of the Restore is completed (check your Zextras Suite Notifications), the system will be ready for your users. Emails and other mailbox items will be restored afterwards.
If you are using the "Backup on Third Party Storage" feature:
-
create a new bucket with the same settings as the old one
-
don’t set it as the BLOB store
-
-
fetch the metadata with the
zxsuite backup retrieveMetadataFromArchive
command -
use the
blobArchive
option when running the External Restore to make sure that the operation will fetch the BLOBs from the old backup bucket
Settings and Configs
Server and Global settings are backed up but are not restored automatically. Zextras Backup’s high-level integration with Zimbra allows you to restore your data to a server with a different OS/Zimbra Release/Networking/Storage setup without any constraints other than the minimum Zimbra version required to run Zextras Suite.
Whether you wish to create a perfect copy of the old server or just take
a cue from the old server’s settings to adapt those to a new
environment, Zextras Backup comes with a very handy CLI command:
getServerConfig
.
zimbra@test:~$ zxsuite backup getServerConfig command getServerConfig requires more parameters Syntax: zxsuite backup getServerConfig {standard|customizations} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT type(M) Multiple choice standard|customizations date(O) String `dd/MM/yyyy HH:mm:ss`|"last"|"all" backup_path(O) Path /opt/zimbra/backup/ng/ file(O) String Path to backup file query(O) String section/id/key verbose(O) String false colors(O) String false (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup getserverconfig standard date last Display the latest backup data for Server and Global configuration. zxsuite backup getserverconfig standard file /path/to/backup/file Display the contents of a backup file instead of the current server backup. zxsuite backup getserverconfig standard date last query zimlets/com_zimbra_ymemoticons colors true verbose true Displays all settings for the com_zimbra_ymemoticons zimlet, using colored output and high verbosity.
Specifically, this will display the latest backed up configurations:
zxsuite backup getServerConfig standard backup_path /your/backup/path/ date last query / | less
You can change the query
argument to display specific settings, e.g.
zimbra@test:~$ zxsuite backup getServerConfig standard date last backup_path /opt/zimbra/backup/ng/ query serverConfig/zimbraMailMode/test.domain.com config date_______________________________________________________________________________________________28/02/2014 04:01:14 CET test.domain.com____________________________________________________________________________________________________________both
The {zimbrahome}/conf/ and {zimbrahome}/postfix/conf/ directories are backed up as well:
zimbra@test:~$ zxsuite backup getServerConfig customizations date last verbose true ATTENTION: These files contain the directories {zimbraHome}/conf/ and {zimbraHome}/postfix/conf/ compressed into a single archive. Restore can only be performed manually. Do it only if you know what you're doing. archives filename customizations_28_02_14#04_01_14.tar.gz path /opt/zimbra/backup/ng/server/ modify date 28/02/2014 04:01:14 CET
VMs and Snapshots
Thanks to the advent of highly evolved virtualization solutions in the past years, virtual machines are now the most common way to deploy server solutions such as Zimbra Collaboration Suite.
Most hypervisors feature customizable snapshot capabilities and
snapshot-based VM backup systems. In case of a disaster, it’s always
possible to roll back to the latest snapshot and import the missing data
using the External Restore
feature of Zextras Backup - using the server’s
backup path as the import path.
Disaster Recovery from a Previous VM State
Snapshot-based backup systems allow you to keep a frozen
copy of a VM
in a valid state and rollback to it at will. To 100% ensure data
consistency, it’s better to take snapshot copies of switched off VMs, but
this is not mandatory.
When using these kinds of systems, it’s vital to make sure that the Backup Path isn’t either part of the snapshot (e.g. by setting the vdisk to `Independent Persistent in VMWare ESX/i) or altered in any way when rolling back in order for the missing data to be available for import.
To perform a disaster recovery from a previous machine state with Zextras Backup, you need to:
-
Restore the last valid backup into a separate (clone) VM in an isolated network, making sure that users can’t access it and that both incoming and outgoing emails are not delivered.
-
Switch on the clone and wait for Zimbra to start.
-
Disable Zextras Backup’s RealTime Scanner.
-
Connect the Virtual Disk containing the untampered Backup Path to the clone and mount it (on a different path).
-
Start an External Restore using the Backup Path as the Import Path.
Doing so will parse all items in the Backup Path and import the missing ones, speeding up the disaster recovery. These steps can be repeated as many time as needed as long as user access and mail traffic is inhibited.
After the restore is completed, make sure that everything is functional and restore user access and mail traffic.
At the end of the operation, you can check that the
configuration of the new mailbox is the same by running the command
zxsuite config dump (See cli.adoc#_zextras_config_cli)
|
Unrestorable Items
How can I check if all of my items have been restored?
It’s very easy. Check the appropriate Operation Completed
notification you received as soon as the restore operation finished.
It can be viewed in the Notifications
section of the
Administration Zimlet, and it’s also emailed to the address you specified
in the Core
section of the Administration Zimlet as the Notification
E-Mail recipient address
.
The skipped items
section contains a per-account list of unrestored
items:
[...] - stats - Restored Items: 15233 Skipped Items: 125 Unrestored Items: 10 - unrestored items - account: account1@domain.com unrestored items: 1255,1369 account: account2@domain.com unrestored items: 49965 account: account14@domain.com unrestored items: 856,13339,45200, 45655 [...]
Why have some of my items not been restored?
There are different possible causes, the most common of which are:
-
Read Error: Either the raw item or the metadata file is not readable due to an I/O exception or a permission issue.
-
Broken item: Both the the raw item or the metadata file are readable by Zextras Backup but their content is broken/corrupted.
-
Invalid item: Both the the raw item or the metadata file are readable and the content is correct, but Zimbra refuses to inject the item.
How Can I Identify Unrestored Items?
There are two ways to do so: via the CLI and via the Zimbra Web Client. The first way can be used to search for the item within the backup/import path, and the second can be used to view the items in the source server.
Identifying Unrestorable Items through the CLI
The getItem
CLI command can display an item and the related
metadata, extracting all information from a backup path/external backup.
The syntax of the command is:
zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT account(M) Account Name/ID item(M) Integer backup_path(O) Path /opt/zimbra/backup/ng/ dump_blob(O) Boolean true|false false date(O) Date dd/mm/yyyy hh:mm:ss|all last (M) == mandatory parameter, (O) == optional parameter
To extract the raw data and metadata information of the item whose itemID is 49965 belonging to account2@domain.com ,also including the full dump of the item’s BLOB, the command would be:
zxsuite backup getItem account2@domain.com 49965 dump_blob true
Identifying Unrestorable Items through the Zimbra WebClient
The comma separated list of unrestored items displayed in the Operation
Complete
notification can be used as a search argument in the Zimbra
Web Client to perform an item search.
To do so:
-
Log into the Zimbra Administration Console in the source server.
-
Use the
View Mail
feature to access the account containing the unrestored items. -
In the search box, enter item: followed by the comma separated list of itemIDs.
e.g.
item: 856,13339,45200,45655
Remember that any search is executed only within the tab it is
executed, so if you are running the search from the Email tab and get
no results try to run the same search in the Address Book , Calendar ,
Tasks and Briefcase tabs
|
How Can I Restore Unrestored Items?
An item not being restored is a clear sign of an issue, either with the item itself or with your current Zimbra setup. In some cases, there are good chances of being able to restore an item even if it was not restored on the first try.
In the following paragraphs, you will find a collections of tips and tricks that can be helpful when dealing with different kinds of unrestorable items.
Items Not Restored because of a Read Error
A dutiful distinction must be done about the read errors that can cause items not to be restored:
-
hard errors: Hardware failures and all other
destructive
errors that cause an unrecoverable data loss. -
soft errors:
non-destructive
errors such as wrong permissions, filesystem errors, RAID issues (e.g.: broken RAID1 mirroring), etc.
While there is nothing much to do about hard errors, you can prevent or mitigate soft errors by following these guidelines:
-
Run a filesystem check.
-
If using a RAID disk setup, check the array for possible issues (depending on RAID level).
-
Make sure that the 'zimbra' user has r/w access to the backup/import path, all its subfolders and all thereby contained files.
-
Carefully check the link quality of network-shared filesystems. If link quality is poor, consider transferring the data with rsync.
-
If using SSHfs to remotely mount the backup/import path, make sure to run the mount command as root using the
-o allow_other
option.
Items Not Restored because Identified as Broken Items
Unfortunately, this is the worst category of unrestored items in terms
of salvageability
.
Based on the degree of corruption of the item, it might be possible to
recover either a previous state or the raw object (this is only valid
for emails). To identify the degree of corruption, use the getItem
CLI
command:
zxsuite backup getItem {account} {item} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT account(M) Account Name/ID item(M) Integer backup_path(O) Path /opt/zimbra/backup/ng/ dump_blob(O) Boolean true|false false date(O) Date dd/mm/yyyy hh:mm:ss|all last (M) == mandatory parameter, (O) == optional parameter
Searching for the broken item, setting the backup_path
parameter to the
import path and the date
parameter to all
, will display all valid
states for the item.
zimbra@test:~$ zxsuite backup getItem admin@example.com 24700 backup_path /mnt/import/ date all itemStates start_date 12/07/2013 16:35:44 type message deleted true blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= start_date 12/07/2013 17:04:33 type message deleted true blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= start_date 15/07/2013 10:03:26 type message deleted true blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
If the item is an email, you will be able to recover a standard .eml
file
through the following steps:
-
Identify the latest valid state
/mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= start_date 15/07/2013 10:03:26 type message deleted true blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
-
Identify the
blob path
blob path /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ=
-
Use gzip to uncompress the BLOB file into an
.eml
file
zimbra@test:~$ gunzip -c /mnt/import/items/c0/c0,gUlvzQfE21z6YRXJnNkKL85PrRHw0KMQUqo,pMmQ= > /tmp/restored.eml zimbra@test:~$ cat /tmp/restored.eml Return-Path: zimbra@test.example.com Received: from test.example.com (LHLO test.example.com) (192.168.1.123) by test.example.com with LMTP; Fri, 12 Jul 2013 16:35:43 +0200 (CEST) Received: by test.example.com (Postfix, from userid 1001) id 4F34A120CC4; Fri, 12 Jul 2013 16:35:43 +0200 (CEST) To: admin@example.com From: admin@example.com Subject: Service mailboxd started on test.example.com Message-Id: <20130712143543.4F34A120CC4@test.example.com> Date: Fri, 12 Jul 2013 16:35:43 +0200 (CEST) Jul 12 16:35:42 test zmconfigd[14198]: Service status change: test.example.com mailboxd changed from stopped to running
-
Done! You can now import the
.eml
file into the appropriate mailbox using your favorite client.
Items Not Restored because Identified as Invalid Items
An item is identified as Invalid
when, albeit being formally correct,
is discarded by Zimbra’s LMTP Validator upon injection. This is common
when importing items created on an older version of Zimbra to a newer
one, Validation rules are updated very often, so not all messages
considered valid by a certain Zimbra version are still
considered valid by a newer version.
If you experienced a lot of unrestored items during an import, it might be a good idea to momentarily disable the LMTP validator and repeat the import:
-
To disable Zimbra’s LMTP Validator, run the following command as the Zimbra user:
zmlocalconfig -e zimbra_lmtp_validate_messages=false
-
Once the import is completed, you can enable the LMTP validator running
zmlocalconfig -e zimbra_lmtp_validate_messages=true
This is a dirty workaround, as items deemed invalid by the
LMTP validator might cause display or mobile synchronization errors. Use
at your own risk.
|
doCoherencyCheck
What is the Coherency Check?
The Coherency Check
performs a deeper check of a Backup Path than the one done by the
SmartScan.
While the SmartScan works incrementally
by only checking items that
have been modified since the last SmartScan, the Coherency Check
performs a thorough check of all metadata and BLOBs in the backup
path.
It’s specifically designed to detect corrupted metadata and BLOBs.
How Does it Work?
The Coherency Check verifies the integrity of all metadata in the
backup path and of the related BLOBs. Should any errors be found,
running the check with the fixBackup
option will move any orphaned or
corrupted metadata/BLOB to a dedicated directory within the backup path.
When Should a Coherency Check be Executed?
-
At interval periods to make sure that everything is ok (e.g. every 3 or 6 months).
-
After a system crash.
-
After the filesystem or storage device containing the backup path experiences any issue.
Should the SmartScan detect a possible item corruption, a Coherency Check will be started automatically.
The Coherency Check is highly I/O consuming, so make sure to run it only during off-peak periods |
Running a Coherency Check
Starting the Check via the Administration Zimlet
The Coherency Check is not available via the Administration Zimlet.
Starting the Check via the CLI
To start a Coherency Check via the CLI, use the doCoherencyCheck
command:
Syntax: zxsuite backup doCoherencyCheck {backup_path} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT backup_path(M) Path accounts(O) Account Name/ID[,..] all checkZimbra(O) Boolean true|false false fixBackup(O) Boolean true|false false notifications(O) Email Address[,..] (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ accounts jack@exmaple.com,john@exmaple.com Performs a coherency check on /opt/zimbra/backup/ng/ for Jack's and John's accounts zxsuite backup docoherencycheck /opt/zimbra/backup/ng/ fixBackup true Performs a coherency check on /opt/zimbra/backup/ng/ and moves corrupted backup files and blob files not referenced by any metadata out of backup
Checking the Status of a Running Check
To check the status of a running scan via the CLI, use the monitor
command:
Syntax: zxsuite backup monitor {operation_uuid} [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE operation_uuid(M) Uiid operation_host(O) String (M) == mandatory parameter, (O) == optional parameter
Taking Additional and Offsite Backups of Zextras Backup’s Datastore
Who Watches the Watchmen?
Having backup systems is a great safety measure against data loss, but
each backup system must be part of a broader backup strategy
to ensure
the highest possible level of reliability. The lack of a proper backup
strategy gives a false sense of security, while actually turning even
the best backup systems in the world into yet another breaking point.
Devising a backup strategy is no easy matter, and at some point you will
most likely be confronted with the following question: What if I lose
the data I backed up?
. The chances of this happening ultimately only
depend on how you make and manage your backups. It’s more likely that you will lose
all of your backed up data if you store both your data and your backups
in a single SATAII disk than if you store your backed up data on a
dedicated SAN using a RAID 1+0 setup.
Here are some suggestions and best practices to improve your backup strategy by making a backup of the Backup NG’s datastore and storing it offsite.
Making an Additional Backup of Zextras Backup’s Datastore
-
Atomicity: Any transaction is committed and written to the disk only when completed.
-
Consistency: Any committed transaction is valid, and no invalid transaction will be committed and written to the disk.
-
Isolation: All transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.
-
Durability: Once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).
Due to this, it’s very easy to make a backup. The best (and
easiest) way to do so is by using rsync
.
Specific options and parameters depend on many factors, such as the
amount of data to be synced and the storage in use, while connecting to
an rsync daemon instead of using a remote shell as a transport is
usually much faster in transferring the data.
You won’t need to stop Zimbra or the Real Time Scanner to make an additional backup of Zextras Backup’s datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.
Storing Your Zextras Backup’s Datastore Backup Offsite
As seen in the previous section, making a backup of Zextras Backup’s Datastore is very easy, and the use of rsync makes it just as easy to store your backup in a remote location.
To optimize your backup strategy when dealing with this kind of setup, the following best practices are recommended:
-
If you schedule your rsync backups, make sure that you leave enough time between an rsync instance and the next one in order for the transfer to be completed.
-
Use the --delete options so that files that have been deleted in the source server are deleted in the destination server to avoid inconsistencies.
-
If you notice that using the
--delete
option takes too much time, schedule two different rsync instances: one with--delete
to be run after the weekly purge and one without this option.
-
-
Make sure you transfer the whole folder tree recursively starting from Zextras Backup’s Backup Path. This includes server config backups and mapfiles.
-
Make sure the destination filesystem is case sensitive (just like the Backup Path must be).
-
If you plan to restore directly from the remote location, make sure that the zimbra user on your server has read and write permissions on the transferred data.
-
Expect to experience slowness if your transfer speed is much higher than your storage throughput (or vice versa).
Additional/Offsite Backup F.A.Q.
Why shouldn’t I use the Export Backup
feature of Zextras Backup instead of
rsync?
For many reasons:
-
The
Export Backup
feature is designed to perform migrations. It exports asnapshot
that is an end in itself and was not designed to be managed incrementally. Each time an Export Backup is run, it’ll probably take just as much time as the previous one, while using rsync is much more time-efficient. -
Being a Zextras Backup operation, any other operation started while the Export Backup is running will be queued until the Export Backup is completed.
-
An
Export Backup
operation has a higher impact on system resources than an rsync. -
Should you need to stop an Export Backup operation, you won’t be able to reprise it, and you’ll need to start from scratch.
Can I use this for Disaster Recovery?
Yes. Obviously, if your Backup Path is still available. it’s better to use that, as it will restore all items and settings to the last valid state. However, should your Backup Path be lost, you’ll be able to use your additional/offsite backup.
Can I use this to restore data on the server the backup copy belongs to?
Yes, but not through the External Restore
operation, since item and
folder IDs are the same.
The most appropriate steps to restore data from a copy of the backup path to the very same server are as follows:
-
Stop the RealTime Scanner.
-
Change the Backup Path to the copy you wish to restore your data from.
-
Run either
Restore on New Account
or aRestore Deleted Account
. -
Once the restore is over, change the backup path to the original one.
-
Start the RealTime Scanner. A SmartScan will trigger to update the backup data.
Can I use this to create an Active/Standby infrastructure?
No, because the External Restore
operation does not perform any
deletions. By running several External Restores, you’ll end
up filling up your mailboxes with unwanted content, since items deleted
from the original mailbox will not be deleted on the standby
server.
The External Restore
operation has been designed so that accounts will
be available for use as soon as the operation is started, so your
users will be able to send and receive emails even if the restore is
running.
Are there any other ways to do an Additional/Offsite backup of my system?
There are for sure, and some of them might even be better than the one described here. These are just guidelines that apply to the majority of cases.
Multistore Information
Zextras Backup in a Multistore Environment
Command Execution in a Multistore Environment
The new Network Administration Zimlet makes the management of multiple servers very easy. You can select a server from the Zextras Backup tab and perform all backup operations on that server, even if you are logged into the Zimbra Administration Console of another server.
Specific differences between Singlestore and Multistore environments are:
-
In a Multistore environment,
Restore on New Account
operations ALWAYS create the new account in the Source account’s mailbox server. -
All operations are logged on the target server, not in the server that launched the operation.
-
If a wrong target server for an operation is chosen, Zimbra automatically proxies the operation request to the right server.
Backup and Restore
Backup and Restore in a Multistore environment will work exactly like in a Singlestore environment.
The different servers will be configured and managed separately via
the Administration Zimlet, but certain operations like Live Full
Scan and Stop All Operations can be 'broadcast' to all the mailstores
via the zxsuite
CLI using the --hostname all_servers
option. This
applies also to Zextras Backup settings.
Backup and Restore operations are managed as follows:
-
Smartscans can be executed on single servers via the Administration Zimlet or on multiple servers via the CLI.
-
Restores can be started from the
Accounts
tab in the Zimbra Admin Console, from each server tab in the Zextras Backup menu of the Administration Zimlet and via the CLI. The differences between these methods are:
Operation started from: | Options |
---|---|
|
The selected account’s restore is automatically started in the proper server. |
|
Any accounts eligible for a restore on the selected server can be chosen as the restore 'source' |
|
Any account on any server can restored, but there is no automatic server selection. |
Export and Import
Export and Import functions are those that differ the most when performed on a Multistore environment.
Here are the basic scenarios.
Export from a Singlestore and Import to a Multistore
Importing multiple accounts of a single domain to a different store will break the consistency of ALL the items that are shared from/to a mailbox on a different server.
A command in the CLI is available to fix the shares for accounts imported on different servers.
Export from a Multistore and Import to a Single or Multistore
Two different scenarios apply here:
-
Mirror
import: Same number of source and destination mailstores. Each export is imported on a different server. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. ThedoCheckShares
anddoFixShares
CLI commands are available to check and fix share consistency (see below). -
Composite
import: Same or different number of source and destination servers. Domains or accounts are manually imported into different servers. This will break the consistency of ALL the items that are shared from/to a mailbox on a different server. ThedoCheckShares
anddoFixShares
CLI commands are available to check and fix share consistency (see below)
The doCheckShares
and doFixShares
Commands
The doCheckShares
command will parse all share information in local
accounts and report any error:
zimbra@test:~$ zxsuite help backup doCheckShares Syntax: zxsuite backup doCheckShares Usage example: zxsuite backup doCheckShares Check all shares on local accounts
The doFixShares
will fix all share inconsistencies using a migration.
zimbra@test:~$ zxsuite help backup doFixShares Syntax: zxsuite backup doFixShares {import_idmap_file} PARAMETER LIST NAME TYPE import_idmap_file(M) String (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup doFixShares idmap_file Fixes the shares' consistency after an import according to the mapping contained in the /opt/zimbra/backup/ng/idmap_file
Operation Queue and Queue Management
Zextras Backup’s Operation Queue
Every time a Zextras Backup operation is started, either manually or through scheduling, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is executed as soon as any preceding operation is dequeued (either because it has been completed or terminated).
The queue system affects the following operations:
-
External backup
-
All restore operations
-
Smartscan
Changes to Zextras Backup’s configuratito on are not enqueued and are applied immediately.
Operation Queue Management
Through the Administration Console
Viewing the Queue
To view the operation queue, access the Notifications
tab in
the Administration Zimlet and click the Operation Queue
button.
The Administration Zimlet displays operations queued both by Zextras Backup and Zextras Powerstore in a single view. This is just a design choice, as the two queues are completely separate, meaning that one Zextras Backup operation and one Zextras Powerstore operation can be running at the same time. |
Through the CLI
Viewing the Queue
To view Zextras Backup’s operation queue, use the getAllOperations
command:
zimbra@server:~$ zxsuite help backup getAllOperations Syntax: zxsuite backup getAllOperations [attr1 value1 [attr2 value2... PARAMETER LIST NAME TYPE EXPECTED VALUES DEFAULT verbose(O) Boolean true|false false (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup getAllOperations Shows all running and queued operations
Emptying the Queue
To stop the current operation and empty Zextras Backup’s operation
queue, use the doStopAllOperations
command:
zimbra@mail:~$ zxsuite help backup doStopAllOperations Syntax: zxsuite backup doStopAllOperations Usage example: zxsuite backup doStopAllOperations Stops all running operations
Removing a Single Operation from the Queue
To stop the current operation or to remove a specific operation
from the queue, use the doStopOperation
command:
zimbra@mail:~$ zxsuite help backup doStopOperation Syntax: zxsuite backup doStopOperation {operation_uuid} PARAMETER LIST NAME TYPE operation_uuid(M) Uiid (M) == mandatory parameter, (O) == optional parameter Usage example: zxsuite backup doStopOperation 30ed9eb9-eb28-4ca6-b65e-9940654b8601 Stops operation with id = 30ed9eb9-eb28-4ca6-b65e-9940654b8601
COS-level Backup Management
What is COS-level Backup Management?
COS-level Backup Management allows the administrator to disable ALL Zextras Backup functions for a whole Class of Service to lower storage usage.
How Does COS-level Backup Management Work?
What happens if I disable the Zextras Backup Module for a Class of Service?
-
The Real Time Scanner will ignore all accounts in the COS.
-
The Export Backup function WILL NOT EXPORT accounts in the COS.
-
Accounts in the COS will be treated as
Deleted
by the backup system. This means that after the data retention period expires, all data for such accounts will be purged from the backup store. Re-enabling the backup for a Class of Service will reset this.
How is the backup enabled
/backup disabled
information saved?
Disabling the backup for a Class of Service will add the following
marker to the Class of Service’s Notes
field: ${ZxBackup_Disabled}
While the Notes field remains fully editable and usable, changing or deleting this marker will re-enable the backup for the COS.
Zextras Backup CLI
Zextras Backup CLI
zxsuite backup
Name | Description | Usage |
---|---|---|
doAccountScan |
Perform a Smart Scan on a single account |
zxsuite backup doAccountScan [object Object] [attr1 value1 [attr2 value2…]] |
doBackupAuthToken |
Perform a backup of authentication tokens |
zxsuite backup doBackupAuthToken [object Object] [attr1 value1 [attr2 value2…]] |
doBackupChat |
Perform a backup of Zextras Team data |
zxsuite backup doBackupChat [object Object] |
doBackupCluster |
Perform a backup of Cluster |
zxsuite backup doBackupCluster [object Object] [attr1 value1 [attr2 value2…]] |
doBackupLDAP |
Perform a backup of LDAP |
zxsuite backup doBackupLDAP [object Object] [attr1 value1 [attr2 value2…]] |
doBackupServerCustomizations |
Perform a backup of customizations |
zxsuite backup doBackupServerCustomizations [object Object] [attr1 value1 [attr2 value2…]] |
doCheckShares |
Check all shares on local accounts |
zxsuite backup doCheckShares [object Object] |
doCoherencyCheck |
Check backup coherency |
zxsuite backup doCoherencyCheck [object Object] [attr1 value1 [attr2 value2…]] |
doEnableDisableCOS |
Enable or disable the backup for a specific COS |
zxsuite backup doEnableDisableCOS [object Object] {enable|disable} |
doExport |
Perform an Export limited by domains |
zxsuite backup doExport [object Object] [attr1 value1 [attr2 value2…]] |
doExternalRestore |
Perform an External Restore |
zxsuite backup doExternalRestore [object Object] [attr1 value1 [attr2 value2…]] |
doFixShares |
Try to fix all shares on local accounts |
zxsuite backup doFixShares [object Object] |
doItemRestore |
Restores a single item |
zxsuite backup doItemRestore [object Object] [object Object] [attr1 value1 [attr2 value2…]] |
doItemSearch |
Search for an item |
zxsuite backup doItemSearch [object Object] [attr1 value1 [attr2 value2…]] |
doPurge |
Perform a Purge Operation |
zxsuite backup doPurge [attr1 value1 [attr2 value2…]] |
doRestartService |
restart a given service |
zxsuite backup doRestartService [object Object] |
doRestoreBlobs |
Start a restore blobs operation which tries to restore broken zimbra blobs |
zxsuite backup doRestoreBlobs [object Object] [attr1 value1 [attr2 value2…]] |
doRestoreOnNewAccount |
Perform a Restore on New Account |
zxsuite backup doRestoreOnNewAccount {Account name or id} [object Object] {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2…]] |
doSmartScan |
Perform a Smart Scan |
zxsuite backup doSmartScan [object Object] [attr1 value1 [attr2 value2…]] |
doStartService |
start a given service |
zxsuite backup doStartService [object Object] |
doStopAllOperations |
Stops all running operations and empties the operations queue |
zxsuite backup doStopAllOperations |
doStopOperation |
Stop a single running operation |
zxsuite backup doStopOperation [object Object] |
doStopService |
stop a given service |
zxsuite backup doStopService [object Object] |
doUndelete |
Perform an Undelete Restore |
zxsuite backup doUndelete [object Object] {"dd/MM/yyyy HH:mm:ss"|first} {"dd/MM/yyyy HH:mm:ss"|last} [attr1 value1 [attr2 value2…]] |
getAccountInfo |
Shows an account’s information |
zxsuite backup getAccountInfo [object Object] [attr1 value1 [attr2 value2…]] |
getAllOperations |
Displays all running and queued operations |
zxsuite backup getAllOperations [attr1 value1 [attr2 value2…]] |
getAvailableAccounts |
Displays all accounts available for restore Note: In case a domain does not exist, a warning message will be shown |
zxsuite backup getAvailableAccounts [attr1 value1 [attr2 value2…]] |
getAvailableDomains |
Displays all domains available for restore |
zxsuite backup getAvailableDomains {dd/MM/yyyy HH:mm:ss|"last"} [object Object] |
getBackupInfo |
Displays informations about the backup system |
zxsuite backup getBackupInfo [attr1 value1 [attr2 value2…]] |
getCOSBackupStatus |
Displays the backup status of all COS |
zxsuite backup getCOSBackupStatus [attr1 value1 [attr2 value2…]] |
getItem |
Display an item in a human-readable format. |
zxsuite backup getItem [object Object] [object Object] [attr1 value1 [attr2 value2…]] |
getMap |
Show binary Map<String, String> object as human readable table |
zxsuite backup getMap [object Object] |
getProperty |
Get configuration properties Warning: this command is deprecated |
zxsuite backup getProperty [attr1 value1 [attr2 value2…]] |
getServerConfig |
Provides a list of stored server configs |
zxsuite backup getServerConfig {standard|customizations} [attr1 value1 [attr2 value2…]] |
getServices |
show current status of all services for this module |
zxsuite backup getServices |
migrateBackupVolume |
Migrate backup volume to a new volume |
zxsuite backup migrateBackupVolume |
monitor |
Monitor a running operation |
zxsuite backup monitor [object Object] [attr1 value1 [attr2 value2…]] |
retrieveMetadataFromArchive |
Retrieve metadata archives |
zxsuite backup retrieveMetadataFromArchive |
setBackupVolume |
Set backup volume location |
zxsuite backup setBackupVolume |
setProperty |
Set configuration property Warning: this command is deprecated |
zxsuite backup setProperty [object Object] [object Object] |