Zextras Powerstore

Hierarchical Storage Management

The Hierarchical Storage Management Technique

HSM is a data storage technique that moves data between different stores according to a defined policy.

The most common use of the HSM technique is the move of older data from a faster-but-expensive storage device to a slower-but-cheaper one based on the following premises:

  • Fast storage costs more.

  • Slow storage costs less.

  • Old data will be accessed much less frequently than new data.

The advantages of the HSM technique are clear: Lowering the overall storage cost since only a small part of your data needs to be on costly storage, and improving the overall user experience.

Stores, Volumes and Policies

Using HSM requires a clear understanding of some related terms:

  • Primary Store: The fast-but-expensive store where all your data is initially placed.

  • Secondary Store: The slow-but-cheap store where older data will be moved to.

Zimbra Stores

The Basics: Types of Stores and Their Uses

Zimbra allows for two different types of stores:

  • Index Store: A store that contains information about your data that is used by Apache Lucene to provide indexing and search functions.

  • Data Store: A store that contains all your Zimbra data organized in a MySql database.

You can have multiple stores of each type, but only one Index Store, one Primary Data Store and one Secondary Data Store can be set as Current (meaning that is currently used by Zimbra).

Primary and Secondary Data Stores

A data store in Zimbra can be either a Primary Data Store or a Secondary Data Store.

Data is moved between the current Primary Data Store and the current Secondary Data Store according to a defined policy.

Zextras Powerstore: Moving Items between Stores

The main feature of the Zextras Powerstore module is the ability to apply defined HSM policies.

The move can be triggered in three ways:

  • Click the Apply Policy button in the Administration Zimlet.

  • Start the doMoveBlobs operation through the CLI.

  • Enable Policy Application Scheduling in the Administration Zimlet and wait for it to start automatically.

Once the move is started, the following operations are performed:

  • Zextras Powerstore scans through the Primary Store to see which items comply with the defined policy.

  • All the Blobs of the items found in the first step are copied to the Secondary Store.

  • The database entries related to the copied items are updated to reflect the move.

  • If the second and the third steps are completed successfully (and only in this case), the old Blobs are deleted from the Primary Store.

The Move operation is stateful - each step is executed only if the previous step has been completed successfully - so the risk of data loss during a Move operation is nonexistent.

doMoveBlobs

The doMoveBlobs Operation of Zextras Powerstore

The doMoveBlobs is the heart of Zextras Powerstore.

It moves items between the Current Primary Store and the Current Secondary Store according to the proper HSM policy.

The move is performed by a transactional algorithm. Should an error occur during one of the steps of the operation, a rollback takes place and no change will be made to the data.

Once Zextras Powerstore identifies the items to be moved, the following steps are performed:

  • A copy of the Blob to the Current Secondary Store is created.

  • The Zimbra Database is updated to notify Zimbra of the item’s new position.

  • The original Blob is deleted from the Current Primary Store.

What is Moved?

Every item that complies with the specified HSM policy is moved.

Example:

The following policy

message,document:before:-20day
message:before:-10day has:attachment

will move all emails and documents older than 20 days along with all emails older than 10 days that contain an attachment.

Policy Order

All conditions for a policy are executed in the exact order they are specified. Zextras Powerstore will loop on all items in the Current Primary Store and apply each separate condition before starting the next one.

This means that the following policies

message,document:before:-20day
message:before:-10day has:attachment
message:before:-10day has:attachment
message,document:before:-20day

applied daily on a sample server that sends/receives a total of 1000 emails per day, 100 of which contain one or more attachments, will have the same final result. However, the execution time of the second policy will probably be slightly higher (or much higher, depending on the number and size of the emails on the server).

This is because in the first policy, the first condition (message,document:before:-20day) will loop on all items and move many of them to the Current Secondary Store, leaving fewer items for the second condition to loop on.

Likewise, having the message:before:-10day has:attachment as the first condition will leave more items for the second condition to loop on.

This is just an example and does not apply to all cases, but gives an idea of the need to carefully plan your HSM policy.

Executing the doMoveBlobs Operation (a.k.a. Applying the HSM Policy)

Applying a policy means running the doMoveBlobs operation in order to move items between the Primary and Secondary store according to the defined policy.

Zextras Powerstore gives you three different options:

  • Via the Administration Zimlet

  • Via the CLI

  • Through Scheduling

Apply the HSM Policy via the Administration Zimlet

To apply the HSM Policy via the Administration Zimlet:

  • Log into the Zimbra Administration Console.

  • Click the Zextras Powerstore entry in the Administration Zimlet.

  • Click the Apply Policy button.

Apply the HSM Policy via the CLI

To apply the HSM Policy via the CLI, run the following command as the zimbra user:

` zxsuite powerstore doMoveBlobs`

Apply the HSM Policy through Scheduling

To schedule a daily execution of the doMoveBlobs operation:

  • Log into the Zimbra Administration Console.

  • Click the Zextras Powerstore entry in the Administration Zimlet.

  • Enable scheduling by selecting the Enable HSM Session scheduling: button.

  • Select the hour to run the operation under HSM Session scheduled for:.

doMoveBlobs Stats and Info

Information about disk space savings, operation performances and more are available by clicking the Stats button under the Secondary Volumes list in the Zextras Powerstore tab of the Administration Zimlet.

Volume Management

Both primary and secondary volumes can be created on either local storage or on supported third-party storage solutions.

Zimbra Volumes

A volume is a distinct entity (path) on a filesystem with all the associated properties that contain Zimbra Blobs.

Volume Properties

All Zimbra volumes are defined by the following properties:

  • Name: A unique identifier for the volume.

  • Path: The path where the data is going to be saved.

    The zimbra user must have r/w permissions on this path.
  • Compression: Enable or Disable the file compression for the volume.

  • Compression Threshold: The minimum file size that will trigger the compression. 'Files under this size will never be compressed even if the compression is enabled.'

  • Current: A Current volume is a volume where data will be written upon arrival (Primary Current) or HSM policy application (Secondary Current).

Local Volumes

Local Volumes (i.e. FileBlob type) can be hosted on any mountpoint on the system regardless of the mountpoint’s destination and are defined by the following properties:

  • Name: A unique identifier for the volume.

  • Path: The path where the data is going to be saved. The zimbra user must have r/w permissions on this path.

  • Compression: Enable or Disable file compression for the volume.

  • Compression Threshold: the minimum file size that will trigger the compression.

    Files under this size will never be compressed even if compression is enabled.

Current Volumes

A Current Volume is a volume where data will be written upon arrival (Primary Current) or HSM Policy Application (Secondary Current). Volumes not set as Current won’t be written upon except by specific manual operations such as the Volume-to-Volume move.

Volume Management with Zextras Powerstore - Administration Zimlet

Creating a New Volume with the Administration Zimlet

To create a new volume from the Zextras Powerstore tab of the Administration Zimlet:

  • Click the appropriate Add option in the Volumes Management section according to the type of volume you want to create.

  • Select the store type, choosing between local mount point or S3 Bucket.

  • Enter the new volume’s name.

  • Enter a path for the new volume.

  • Check the Enable Compression button if you wish to activate data compression on the new volume.

  • Select the Compression Threshold.

  • If you are using an S3 Bucket, it’s possible to store information for multiple buckets.

  • Press OK to create the new volume. Should the operation fail, a notification containing any related errors will be generated.

Editing a Volume with the Administration Zimlet

To edit a volume from the Administration Zimlet, simply select an existing volume and press the appropriate Edit button.

Deleting a Volume with the Administration Zimlet

To delete a volume from the Administration Zimlet, select an existing volume and press the appropriate Delete button. Remember that only empty volumes can be deleted.

Volume Management with Zextras Powerstore - From the CLI

Beginning with release 8.8.9, all volume creation and update commands have been updated, as the storeType argument is now required.

The storeType argument is mandatory, it is always the on the first position and accepts any one value corresponding to the S3-Compatible Services listed previously. The arguments that follow in the command now depend on the selected storeType.

FileBlob (Local)

Updated zxsuite syntax to create new FileBlob zimbra volume:

# Add volume, run as zimbra user
zxsuite powerstore doCreateVolume FileBlob name secondary /path/to/store
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume FileBlob name current_volume true

zxsuite powerstore doCreateVolume FileBlob

Syntax:
    zxsuite powerstore doCreateVolume FileBlob {volume_name} {primary|secondary|index} {volume_path} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
volume_name(M)                  String
volume_type(M)                  Multiple choice     primary|secondary|index
volume_path(M)                  Path
volume_compressed(O)            Boolean             true|false                  false
compression_threshold_bytes(O)  Long                                            4096

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume FileBlob volumeName secondary /path/to/store volume_compressed true compression_threshold_bytes 4096

zxsuite powerstore doUpdateVolume FileBlob

Syntax:
    zxsuite powerstore doUpdateVolume FileBlob {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_type(O)                  String              primary|secondary|index
volume_name(O)                  String
volume_path(O)                  Path
current_volume(O)               Boolean             true|false                  false
volume_compressed(O)            String
compression_threshold(O)        String

(M) == mandatory parameter, (O) == optional parameter
S3 (Amazon and any S3-compatible solution not explicitly supported)
# Add volume, run as zimbra user
zxsuite powerstore doCreateVolume S3 name secondary bucket_name bucket access_key accessKey secret secretString region EU_WEST_1
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume S3 name current_volume true

zxsuite powerstore doCreateVolume S3

Syntax:
    zxsuite powerstore doCreateVolume S3 {Name of the zimbra store} {primary|secondary} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES
volume_name(M)                  String              Name of the zimbra store
volume_type(M)                  Multiple choice     primary|secondary
bucket_name(O)                  String              Amazon AWS bucket
access_key(O)                   String              Service username
secret(O)                       String              Service password
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing S3 service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
region(O)                       String              Amazon AWS Region
url(O)                          String              S3 API compatible service url (ex: s3api.service.com)
prefix(O)                       String              Prefix added to blobs keys
use_infrequent_access(O)        Boolean             true|false
infrequent_access_threshold(O)  String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

S3 AWS Bucket:
    zxsuite powerstore doCreateVolume S3 volumeName primary bucket_name bucket access_key accessKey secret secretKey prefix objectKeysPrefix region EU_WEST_1 user_infrequent_access TRUE infrequent_access_threshold 4096

S3 compatible object storage:
    zxsuite powerstore doCreateVolume S3 volumeName primary bucket_name bucket access_key accessKey secret secretKey url http://host/service

Using existing bucket configuration:
    zxsuite powerstore doCreateVolume S3 volumeName primary bucket_configuration_id 316813fb-d3ef-4775-b5c8-f7d236fc629c

zxsuite powerstore doUpdateVolume S3

Syntax:
    zxsuite powerstore doUpdateVolume S3 {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_name(O)                  String
volume_type(O)                  String              primary|secondary
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
use_infrequent_access(O)        Boolean             true|false
infrequent_access_threshold(O)  String
current_volume(O)               Boolean             true|false                  false

(M) == mandatory parameter, (O) == optional parameter
Scality (S3 compatible object storage)
# Add volume, run as zimbra user
zxsuite powerstore doCreateVolume ScalityS3 name secondary bucket_name mybucket access_key accessKey1 secret verySecretKey1 url http://{IP_ADDRESS}:{PORT}
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume ScalityS3 name current_volume true

zxsuite powerstore doCreateVolume ScalityS3

Syntax:
    zxsuite powerstore doCreateVolume ScalityS3 {volume_name} {primary|secondary} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES
volume_name(M)                  String
volume_type(M)                  Multiple choice     primary|secondary
bucket_name(O)                  String              Bucket name
url(O)                          String              S3 API compatible service url (ex: s3api.service.com)
access_key(O)                   String              Service username
secret(O)                       String              Service password
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
prefix(O)                       String              Prefix added to blobs keys

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume ScalityS3 volumeName primary bucket_name bucket url http://host/service access_key accessKey secret secretKet
zxsuite powerstore doCreateVolume ScalityS3 volumeName primary bucket_configuration_id uuid

zxsuite powerstore doUpdateVolume ScalityS3

Syntax:
    zxsuite powerstore doUpdateVolume ScalityS3 {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_name(O)                  String
volume_type(O)                  String              primary|secondary
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing S3 service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
current_volume(O)               Boolean             true|false                  false

(M) == mandatory parameter, (O) == optional parameter
EMC (S3 compatible object storage)
# Add volume, run as zimbra user
zxsuite powerstore docreatevolume EMC name secondary bucket_name bucket access_key ACCESSKEY secret SECRET url https://url.of.storage
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume EMC name current_volume true

zxsuite powerstore doCreateVolume EMC

Syntax:
    zxsuite powerstore doCreateVolume EMC {volume_name} {primary|secondary} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES
volume_name(M)                  String
volume_type(M)                  Multiple choice     primary|secondary
bucket_name(O)                  String              Bucket name
url(O)                          String              S3 API compatible service url (ex: s3api.service.com)
access_key(O)                   String              Service username
secret(O)                       String              Service password
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
prefix(O)                       String              Prefix added to blobs keys

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume EMC volumeName primary bucket_name bucket url http://host/service access_key accessKey secret secretKet
zxsuite powerstore doCreateVolume EMC volumeName primary bucket_configuration_id uuid

zxsuite powerstore doUpdateVolume EMC

Syntax:
    zxsuite powerstore doUpdateVolume EMC {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_name(O)                  String
volume_type(O)                  String              primary|secondary
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
current_volume(O)               Boolean             true|false                  false

(M) == mandatory parameter, (O) == optional parameter
OpenIO
# add volume, run as zimbra user
zxsuite powerstore doCreateVolume OpenIO name secondary http://{IP_ADDRESS} ZeXtras OPENIO
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume OpenIO name current_volume true

zxsuite powerstore doCreateVolume OpenIO

Syntax:
    zxsuite powerstore doCreateVolume OpenIO {volume_name} {primary|secondary} {url} {account} {namespace} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES
volume_name(M)                  String
volume_type(M)                  Multiple choice     primary|secondary
url(M)                          String
account(M)                      String
namespace(M)                    String
proxy_port(O)                   Integer
account_port(O)                 Integer

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume OpenIO volumeName primary http://host/service

accountName namespaceString proxy_port 6006 account_port 6009

Syntax:
zxsuite powerstore doUpdateVolume OpenIO {current_volume_name} [attr1 value1
[attr2 value2...]]
PARAMETER LIST
NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_name(O)                  String
volume_type(O)                  String              primary|secondary
url(O)                          String
account(O)                      String
namespace(O)                    String
proxy_port(O)                   Integer
account_port(O)                 Integer
current_volume(O)               Boolean             true|false                  false

(M) == mandatory parameter, (O) == optional parameter
Swift
# add volume, run as zimbra user
zxsuite powerstore doCreateVolume Swift name secondary http://{IP_ADDRESS}:8080/auth/v1.0/ user:username password maxDeleteObjectsCount 100
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume Swift name current_volume true

zxsuite powerstore doCreateVolume Swift

Syntax:
    zxsuite powerstore doCreateVolume Swift {volume_name} {primary|secondary} {url} {username} {password} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                        TYPE        EXPECTED VALUES             DEFAULT
volume_name(O)              String
volume_type(O)              String      primary|secondary
url(O)                      String
username(O)                 String
password(O)                 String
maxDeleteObjectsCount(O)    Integer     Number of object in a single bulk delete request
                                                                    500

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume Swift volumeName primary http://host/service accountName password max_delete_objects_count 100

zxsuite powerstore doUpdateVolume Swift

Syntax:
    zxsuite powerstore doUpdateVolume Swift {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                        TYPE        EXPECTED VALUES             DEFAULT
current_volume_name(M)      String
volume_name(O)              String
volume_type(O)              String      primary|secondary
url(O)                      String
username(O)                 String
password(O)                 String
maxDeleteObjectsCount(O)    Integer     Number of object in a single bulk delete request
                                                                    500
current_volume(O)           Boolean     true|false                  false

(M) == mandatory parameter, (O) == optional parameter
Cloudian (S3 compatible object storage)
# add volume, run as zimbra user
zxsuite powerstore doCreateVolume Cloudian name secondary bucket_name bucket access_key ACCESSKEY secret SECRET url https://url.of.storage
# Delete volume
zxsuite powerstore doDeleteVolume name
# set current
zxsuite powerstore doUpdateVolume Cloudian name current_volume true

zxsuite powerstore doCreateVolume Cloudian

Syntax:
    zxsuite powerstore doCreateVolume Cloudian {volume_name} {primary|secondary} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES
volume_name(M)                  String
volume_type(M)                  Multiple choice     primary|secondary
bucket_name(O)                  String              Bucket name
url(O)                          String              S3 API compatible service url (ex: s3api.service.com)
access_key(O)                   String              Service username
secret(O)                       String              Service password
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
prefix(O)                       String              Prefix added to blobs keys

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doCreateVolume Cloudian volumeName primary bucket_name bucket url http://host/service access_key accessKey secret secretKet
zxsuite powerstore doCreateVolume Cloudian volumeName primary bucket_configuration_id uuid

zxsuite powerstore doUpdateVolume Cloudian

Syntax:
    zxsuite powerstore doUpdateVolume Cloudian {current_volume_name} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                            TYPE                EXPECTED VALUES             DEFAULT
current_volume_name(M)          String
volume_name(O)                  String
volume_type(O)                  String              primary|secondary
server_prefix(O)                String              Prefix to the server id used in all objects keys
bucket_configuration_id(O)      String              UUID for already existing service credentials
                                                    (zxsuite config global get attribute s3BucketConfigurations)
current_volume(O)               Boolean             true|false                  false

(M) == mandatory parameter, (O) == optional parameter

Volume Deletion

zxsuite powerstore doDeleteVolume

Syntax:
    zxsuite powerstore doDeleteVolume {volume_name}

PARAMETER LIST

NAME                            TYPE
volume_name(M)                  String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore dodeletevolume hsm
Deletes volume with name hsm

Move all data from a volume to another

Syntax:
    zxsuite powerstore doVolumeToVolumeMove {source_volume_name} {destination_volume_name}

PARAMETER LIST

NAME                            TYPE
source_volume_name(M)           String
destination_volume_name(M)      String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doVolumeToVolumeMove sourceVolume destVolume
Moves the whole contents of sourceVolume to destVolume

Centralized Storage

The Centralized Storage feature allows to use an S3 bucket to host data coming from multiple servers at the same time sharing the same directory structure, as opposed to "independent" volumes which are self-contained and whose directory structure is strictly related to the server and volume itself.

This allows for better data management in large multistore environments and greatly improves mailbox move speed.

Enabling Centralized Storage

  1. Create the centralized volume on any one of your servers using the zxsuite powerstore doCreateVolume command.

    1. All volume types except for FileBlob are compatible;

    2. Make sure to add the centralized TRUE flag to set the volume as a Centralized Storage;

    3. The full syntax for the command depends on the storage type;

  2. Once the Centralized Volume has been created, use the zxsuite doCreateVolume Centralized command on all other mailbox servers to copy the Centralized Volume’s configuration from the first server and add it to the volume list.

    1. The full syntax for the command is zxsuite powerstore doCreateVolume Centralized {server_name} {volume_name}

Centralized Storage Structure

Storage Structure Data is stored in a Centralized Volume plainly, as the main directory of the volume contains a single empty directory for each server connected to the volume and a directory for each mailbox stored in it at the very same level.

In the following example, servers 3aa2d376-1c59-4b5a-94f6-101602fa69c6 and 595a4409-6aa1-413f-9f45-3ef0f1e560f5 are both connected to the same Centralized volume, where 3 mailboxes are stored. As you can see, the effective server where the mailboxes are hosted is irrelevant to the storage.

_
|- 3aa2d376-1c59-4b5a-94f6-101602fa69c6/
|- 595a4409-6aa1-413f-9f45-3ef0f1e560f5/
|- ff46e039-28e3-4343-9d66-92adc60e60c9/
\
 |-- 357-104.msg
 |-- 368-115.msg
 |-- 369-116.msg
 |-- 373-120.msg
 |-- 374-121.msg
 |-- 375-122.msg
 |-- 376-123.msg
 |-- 383-130.msg
|- 4c022592-f67d-439c-9ff9-e3d48a8c801b/
\
 |-- 315-63.msg
 |-- 339-87.msg
 |-- 857-607.msg
 |-- 858-608.msg
 |-- 859-609.msg
 |-- 861-611.msg
 |-- 862-612.msg
 |-- 863-613.msg
 |-- 864-614.msg
 |-- 865-615.msg
 |-- 866-616.msg
 |-- 867-617.msg
 |-- 868-618.msg
|- dafd5569-4114-4268-9201-14f4a895a3d5/
\
 |-- 357-104.msg
 |-- 368-115.msg
 |-- 369-116.msg
 |-- 373-120.msg
 |-- 374-121.msg
 |-- 375-122.msg
 |-- 376-123.msg
 |-- 383-130.msg
 |-- 384-131.msg

Policy Management

What is a Policy?

An HSM policy is a set of rules that define what items will be moved from the Primary Store to the Secondary Store when the doMoveBlobs operation of Zextras Powerstore is triggered, either manually or by scheduling.

A policy can consist of a single rule that is valid for all item types (Simple policy) or multiple rules valid for one or more item types (Composite policy). Also, an additional sub-rule can be defined using Zimbra’s search syntax.

Policy Examples

Here are some policy examples. To see how to create the policies in the Zextras Powerstore module, see below.

  • Move all items older than 30 days

  • Move emails older than 15 days and items of all other kinds older than 30 days

  • Move calendar items older than 15 days, briefcase items older than 20 days and all emails in the "Archive" folder

Defining a Policy

Policies can be defined both from the Zextras Powerstore tab of the Administration Zimlet and from the CLI. You can specify a Zimbra Search in both cases.

From the Administration Zimlet

To define a policy from the Administration Zimlet:

  • Log into the Zimbra Administration Console.

  • Click Zextras Powerstore on the Administration Zimlet.

  • Click the Add button in the Storage Management Policy section.

  • Select the Item Types from the Items to Move: list.

  • Enter the Item Age from the Move Items older than: box.

  • OPTIONAL: Add a Zimbra Search in the Additional Options box.

  • You can add multiple lines to narrow down your policy. Every line will be evaluated and executed after the line before has been applied.

From the CLI

Two policy management commands are available in the CLI:

  • setHsmPolicy

  • +setHsmPolicy

zxsuite powerstore setHsmPolicy {policy}

This command resets the current policy and creates a new one as specified by the policy parameter.

The policy parameter must be specified in the following syntax

itemType1[,itemType2,itemtype3,etc]:query

zxsuite powerstore +setHsmPolicy {policy}

This command adds the query specified by the policy parameter to the current HSM Policy.

The policy parameter must be specified in the following syntax

itemType1[,itemType2,itemtype3,etc]:query

Zextras Powerstore and S3 buckets

Primary and Secondary volumes created with Zextras Powerstore can be hosted on S3 buckets, effectively moving the largest part of your data to secure and durable cloud storage.

S3-compatible Services

While any storage service compatible with the Amazon S3 API should work out of the box with Zextras Powerstore, listed here are the only officially supported platforms:

  • FileBlob (standard local volume)

  • Amazon S3

  • EMC

  • OpenIO

  • Swift

  • Scality S3

  • Cloudian

  • Custom S3 (any unsupported S3-compliant solution)

Primary Volumes and the "Incoming" directory

In order to create a remote Primary Store on a mailbox server a local "Incoming" directory must exist on that server. The default directory is /opt/zimbra/incoming, but you can check or modify the current value using these commands:

zxsuite config server get $(zmhostname) attribute incomingPath
zxsuite config server set $(zmhostname) attribute incomingPath value /path/to/dir

Local Cache

Storing a volume on third-party remote storage solutions requires a local directory to be used for item caching, which must be readable and writable by the zimbra user.

The local directory must be created manually and its path must be entered in the Zextras Powerstore section of the Administration Zimlet in the Zimbra Administration Console.

If the Local Cache directory is not set, you won’t be able to create any secondary volume on an S3-compatible device or service.

Failing to correctly configure the cache directory will cause items to be unretrievable, meaning that users will get a No such BLOB error when trying to access any item stored on an S3 volume.

Bucket Setup

Zextras Powerstore doesn’t need any dedicated setting or configuration on the S3 side, so setting up a bucket for your volumes is easy. Although creating a dedicated user bucket and access policy are not required, they are strongly suggested because they make it much easier to manage.

All you need to start storing your secondary volumes on S3 is:

  • An S3 bucket. You need to know the bucket’s name and region in order to use it.

  • A user’s Access Key and Secret.

  • A policy that grants the user full rights on your bucket.

Bucket Management

A centralized Bucket Management UI is available in the Zimbra Administration Console. This facilitates saving bucket information to be reused when creating a new volume on an S3-compatible storage instead of entering the information each time.

To access the Bucket Management UI:

  • Access the Zimbra Administration Console

  • Select the "Configure" entry on the left menu

  • Select the "Global Settings" entry

  • Select the S3 Buckets entry

Any bucket added to the system will be available when creating a new volume of the following type: Amazon S3, Cloudian, EMC, Scality S3, Custom S3.

Bucket paths and naming

Files are stored in a bucket according to a well-defined path, which can be customized at will in order to make your bucket’s contents easier to understand even on multi-server environments with multiple secondary volumes:

/Bucket Name/Destination Path/[Volume Prefix-]serverID/

  • The Bucket Name and Destination Path are not tied to the volume itself, and there can be as many volumes under the same destination path as you wish.

  • The Volume Prefix, on the other hand, is specific to each volume and it’s a quick way to differentiate and recognize different volumes within the bucket.

Creating Volumes with Zextras Powerstore

To create a new volume with Zextras Powerstore from the Zimbra Administration Console:

  • Enter the HSM Section of the Zextras Administration Zimlet in the Zimbra Administration Console

  • Click on Add under either the Primary Volumes or Secondary Volumes list

  • Select the Volume Type among the available storage choices

  • Enter the required volume information

    Each volume type will require different information to be set up, please refer to your storage provider’s online resources to obtain those details.

Editing Volumes with Zextras Powerstore

To edit a volume with Zextras Powerstore from the Zimbra Administration Console:

  • Enter the HSM Section of the Zextras Administration Zimlet in the Zimbra Administration Console

  • Select a volume

  • Click on Edit

  • When done, click Save

Deleting Volumes with Zextras Powerstore

To delete a volume with Zextras Powerstore from the Zimbra Administration Console:

  • Enter the HSM Section of the Zextras Administration Zimlet in the Zimbra Administration Console

  • Select a volume

  • Click on Delete

Only empty volumes can be deleted.

Amazon S3 Tips

Bucket

Storing your secondary Zimbra volumes on Amazon S3 doesn’t have any specific bucket requirements, but we suggest that you create a dedicated bucket and disable Static Website Hosting for easier management.

User

To obtain an Access Key and the related Secret, a Programmatic Access user is needed. We suggest that you create a dedicated user in Amazon’s IAM Service for easier management.

Rights Management

In Amazon’s IAM, you can set access policies for your users. It’s mandatory that the user of your Access Key and Secret has a set of appropriate rights both on the bucket itself and on its contents. For easier management, we recommend granting full rights as shown in the following example:

{
    `Version`: `[LATEST API VERSION]`,
    `Statement`: [
        {
            `Sid`: `[AUTOMATICALLY GENERATED]`,
            `Effect`: `Allow`,
            `Action`: [
                `s3:*`
            ],
            `Resource`: [
                `[BUCKET ARN]/*`,
                `[BUCKET ARN]`
            ]
        }
    ]
}
This is not a valid configuration policy. Don’t copy and paste it into your user’s settings as it won’t be validated.

If you only wish to grant minimal permissions, change the Action section to:

"Action": [
                `s3:PutObject`,
                `s3:GetObject`,
                `s3:DeleteObject`,
                `s3:AbortMultipartUpload`
              ],

The bucket’s ARN is expressed according to Amazon’s standard naming format: arn:partition:service:region:account-id:resource. For more information about this topic, please see Amazon’s documentation.

Bucket Paths and Naming

Files are stored in a bucket according to a well-defined path, which can be customized at will to make your bucket’s contents easier to understand (even on multi-server environments with multiple secondary volumes):

/Bucket Name/Destination Path/serverID/

The Bucket Name and Destination Path are not tied to the volume itself, and there can be as many volumes under the same destination path as you wish.

The Volume Prefix, on the other hand, is specific to each volume and it’s a quick way to differentiate and recognize different volumes within the bucket.

Infrequent Access Storage Class

Zextras Powerstore is compatible with the Amazon S3 Standard - Infrequent access storage class and will set any file larger than the Infrequent Access Threshold value to this storage class as long as the option has been enabled on the volume.

For more information about Infrequent Access, please refer to the official Amazon S3 Documentation.

Intelligent Tiering Storage Class

Zextras Powerstore is compatible with the Amazon S3 - Intelligent Tiering storage class and will set the appropriate Intelligent Tiering flag on all files, as long as the option has been enabled on the volume.

For more information about Intelligent Tiering, please refer to the official Amazon S3 Documentation.

Item Deduplication

What is Item Deduplication

Item deduplication is a technique that allows you to save disk space by storing a single copy of an item and referencing it multiple times instead of storing multiple copies of the same item and referencing each copy only once.

This might seem like a minor improvement. However, in practical use, it makes a significant difference.

Item Deduplication in Zimbra

Item deduplication is performed by Zimbra at the moment of storing a new item in the Current Primary Volume.

When a new item is being created, its message ID is compared to a list of cached items. If there is a match, a hard link to the cached message’s BLOB is created instead of a whole new BLOB for the message.

The dedupe cache is managed in Zimbra through the following config attributes:

zimbraPrefDedupeMessagesSentToSelf

Used to set the deduplication behavior for sent-to-self messages.

<attr id="144" name="zimbraPrefDedupeMessagesSentToSelf" type="enum" value="dedupeNone,secondCopyifOnToOrCC,dedupeAll" cardinality="single"
optionalIn="account,cos" flags="accountInherited,domainAdminModifiable">
  <defaultCOSValue>dedupeNone</defaultCOSValue>
  <desc>dedupeNone|secondCopyIfOnToOrCC|moveSentMessageToInbox|dedupeAll</desc>
</attr>

zimbraMessageIdDedupeCacheSize

Number of cached Message IDs.

<attr id="334" name="zimbraMessageIdDedupeCacheSize" type="integer" cardinality="single" optionalIn="globalConfig" min="0">
  <globalConfigValue>3000</globalConfigValue>
  <desc>
    Number of Message-Id header values to keep in the LMTP dedupe cache.
    Subsequent attempts to deliver a message with a matching Message-Id
    to the same mailbox will be ignored.  A value of 0 disables deduping.
  </desc>
</attr>

zimbraPrefMessageIdDedupingEnabled

Manage deduplication at account or COS-level.

<attr id="1198" name="zimbraPrefMessageIdDedupingEnabled" type="boolean" cardinality="single" optionalIn="account,cos" flags="accountInherited"
 since="8.0.0">
  <defaultCOSValue>TRUE</defaultCOSValue>
  <desc>
    Account-level switch that enables message deduping.  See zimbraMessageIdDedupeCacheSize for more details.
  </desc>
</attr>

zimbraMessageIdDedupeCacheTimeout

Timeout for each entry in the dedupe cache.

<attr id="1340" name="zimbraMessageIdDedupeCacheTimeout" type="duration" cardinality="single" optionalIn="globalConfig" since="7.1.4">
  <globalConfigValue>0</globalConfigValue>
  <desc>
    Timeout for a Message-Id entry in the LMTP dedupe cache. A value of 0 indicates no timeout.
    zimbraMessageIdDedupeCacheSize limit is ignored when this is set to a non-zero value.
  </desc>
</attr>

(older Zimbra versions might use different attributes or lack some of them)

Item Deduplication and Zextras Powerstore

The Zextras Powerstore features a doDeduplicate operation that parses a target volume to find and deduplicate any duplicated item.

Doing so you will save even more disk space, as while Zimbra’s automatic deduplication is bound to a limited cache, Zextras Powerstore’s deduplication will also find and take care of multiple copies of the same email regardless of any cache or timing.

Running the doDeduplicate operation is also highly suggested after a migration or a large data import in order to optimize your storage usage.

Running a Volume Deduplication

Via the Administration Zimlet

To run a volume deduplication via the Administration Zimlet, simply click on the Zextras Powerstore tab, select the volume you wish to deduplicate and press the Deduplicate button.

Via the CLI

To run a volume deduplication through the CLI, use the doDeduplicate command:

zimbra@mailserver:~$ zxsuite powerstore doDeduplicate

command doDeduplicate requires more parameters

Syntax:
   zxsuite powerstore doDeduplicate {volume_name} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE           EXPECTED VALUES    DEFAULT
volume_name(M)    String[,..]
dry_run(O)        Boolean        true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore dodeduplicate secondvolume
Starts a deduplication on volume secondvolume

To list all available volumes, you can use the `zxsuite powerstore getAllVolumes` command.

doDeduplicate Stats

The doDeduplicate operation is a valid target for the monitor command, meaning that you can watch the command’s statistics while it’s running through the zxsuite powerstore monitor [operationID] command.

Sample Output

Current Pass (Digest Prefix):  63/64
 Checked Mailboxes:             148/148
 Deduplicated/duplicated Blobs: 64868/137089
 Already Deduplicated Blobs:    71178
 Skipped Blobs:                 0
 Invalid Digests:               0
 Total Space Saved:             21.88 GB
  • Current Pass (Digest Prefix): The doDeduplicate command will analyze the BLOBS in groups based on the first character of their digest (name).

  • Checked Mailboxes: The number of mailboxes analyzed for the current pass.

  • Deduplicated/duplicated Blobs: Number of BLOBS deduplicated by the current operation / Number of total duplicated items on the volume.

  • Already Deduplicated Blobs: Number of deduplicated blobs on the volume (duplicated blobs that have been deduplicated by a previous run).

  • Skipped Blobs: BLOBs that have not been analyzed, usually because of a read error or missing file.

  • Invalid Digests: BLOBs with a bad digest (name different from the actual digest of the file).

  • Total Space Saved: Amount of disk space freed by the doDeduplicate operation.

Looking at the sample output above we can see that:

  • The operation is running the second to last pass on the last mailbox.

  • 137089 duplicated BLOBs have been found, 71178 of which have already been deduplicated previously.

  • The current operation deduplicated 64868 BLOBs, for a total disk space saving of 21.88GB.

Advanced Volume Operations

Zextras Powerstore: More than Meets the Eye

At first sight, Zextras Powerstore seems to be strictly dedicated to HSM. However, it also features some highly useful volume-related tools that are not directly related to HSM.

Due to the implicit risks in volume management, these tools are only available through the CLI.

Volume Operations at a Glance

The following volume operations are available:

doCheckBlobs: Perform BLOB coherency checks on one or more volumes.

doDeduplicate: Start Item Deduplication on a volume.

doVolumeToVolumeMove: Move all items from one volume to another.

getVolumeStats: Display information about a volume’s size and number of thereby contained items/blobs.

Volume Operation Analysis

doCheckBlobs

Usage

zimbra@mail:~$ zxsuite powerstore doCheckBlobs

command doCheckBlobs requires more parameters

Syntax:
   zxsuite powerstore doCheckBlobs {start} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                           TYPE            EXPECTED VALUES    DEFAULT
action(M)                      String          start
volume_ids(O)                  Integer[,..]    1,3
mailbox_ids(O)                 Integer[,..]    2,9,27
missing_blobs_crosscheck(O)    Boolean         true|false         true
traced(O)                      Boolean         true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

Usage examples:
zxsuite powerstore doCheckBlobs start: Perform a BLOB coherency check on all message volumes.

zxsuite powerstore doCheckBlobs start volume_ids 1,3: Perform a BLOB coherency check on volumes 1 and 3.

zxsuite powerstore doCheckBlobs start mailbox_ids 2,9,27: Perform a BLOB coherency check on mailboxes 2,9 and 27.

zxsuite powerstore doCheckBlobs start missing_blobs_crosscheck false: Perform a BLOB coherency check without checking on other volumes.

zxsuite powerstore doCheckBlobs start traced true: Perform a BLOB coherency check, logging even the correct checked items.

Description and Tips

The doCheckBlobs operation can be used to run BLOB coherency checks on volumes and mailboxes. This can be useful when experiencing issues related to broken or unviewable items, which are often caused because either Zimbra cannot find or access the BLOB file related to an item or there is an issue with the BLOB content itself.

Specifically, the following checks are made:

  • DB-to-BLOB coherency: For every Item entry in Zimbra’s DB, check whether the appropriate BLOB file exists.

  • BLOB-to-DB coherency: For every BLOB file in a volume/mailbox, check whether the appropriate DB data exists.

  • Filename coherency: Checks the coherency of each BLOB’s filename with its content (as BLOBs are named after their file’s SHA hash).

  • Size coherency: For every BLOB file in a volume/mailbox, checks whether the BLOB file’s size is coherent with the expected size (stored in the DB).

The old zmblobchk command is deprecated and replaced by zxsuite powerstore doCheckBlobs on all infrastructures using Zextras Powerstore module.

doDeduplicate

Usage

zimbra@mail:~$ zxsuite powerstore doDeduplicate

command doDeduplicate requires more parameters

Syntax:
   zxsuite powerstore doDeduplicate {volume_name} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME              TYPE           EXPECTED VALUES    DEFAULT
volume_name(M)    String[,..]
dry_run(O)        Boolean        true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore dodeduplicate secondvolume
Starts a deduplication on volume secondvolume

doVolumeToVolumeMove

Usage

zimbra@mail:~$ zxsuite powerstore doVolumeToVolumeMove

command doVolumeToVolumeMove requires more parameters

Syntax:
   zxsuite powerstore doVolumeToVolumeMove {source_volume_name} {destination_volume_name}

PARAMETER LIST

NAME                          TYPE
source_volume_name(M)         String
destination_volume_name(M)    String

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doVolumeToVolumeMove sourceVolume destVolume
Moves the whole sourceVolume to destVolume

Description and Tips

This command can prove highly useful in all situations where you need to stop using a volume, such as:

  • Decommissioning old hardware: If you want to get rid of an old disk in a physical server, create new volumes on other/newer disks and move your data there.

  • Fixing little mistakes: If you accidentally create a new volume in the wrong place, move the data to another volume.

  • Centralize volumes: Centralize and move volumes as you please, for example, if you redesigned your storage infrastructure or you are tidying up your Zimbra volumes.

getVolumeStats

Usage

zimbra@mail:~$ zxsuite powerstore getVolumeStats

command getVolumeStats requires more parameters

Syntax:
   zxsuite powerstore getVolumeStats {volume_id} [attr1 value1 [attr2 value2...

PARAMETER LIST

NAME                   TYPE       EXPECTED VALUES    DEFAULT
volume_id(M)           Integer
show_volume_size(O)    Boolean    true|false         false
show_blob_num(O)       Boolean    true|false         false

(M) == mandatory parameter, (O) == optional parameter

Usage example:

**BE CAREFUL** show_volume_size and show_blob_num options are IO intensive and thus disabled by default

zxsuite powerstore getVolumeStats 2
Shows stats for the volume with ID equal to 2

Description and Tips

This command provides the following information about a volume:

name description

id

The ID of the volume

name

The Name of the volume

path

The Path of the volume

compressed

Compression enabled/disabled

threshold

Compression threshold (in bytes)

lastMoveOutcome

Exit status of the latest doMoveBlobs operation

lastMoveTimestamp

End timestamp of the latest doMoveBlobs operation

lastMoveDuration

Duration of the last doMoveBlobs operation

lastItemMovedCount

Number of items moved to the current secondary volume during the latest doMoveBlobs operation

bytesSaved

Total amount of disk space freed up thanks to deduplication and compression

bytesSavedLast

Amount of disk space freed up thanks to deduplication and compression during the latest doMoveBlobs operation

The show_volume_size and show_blob_num options will add the following data to the output:

option name description

show_volume_size

totSize

Total disk space used up by the volume

show_blob_num

blobNumber

Number of BLOB files in the volume

Moving Mailboxes Between Mailstores

The doMailboxMove command allows you to move a single mailbox or all accounts from a given domain, from one mailbox server to another within the same Zimbra infrastructure.

If the Zextras Powerstore module is installed and enabled, this command replaces the old zmmboxmove and zmmailboxmove commands. Using any of the legacy commands will return an error and won’t move any data.

Syntax

Syntax:
   zxsuite powerstore doMailboxMove {destinationHost} [attr1 value1 [attr2 value2...]]

PARAMETER LIST

NAME                       TYPE             EXPECTED VALUES                                      DEFAULT
destinationHost(M)         String
accounts(O)                String[,..]      john@example.com,smith@example.com[,...]
domains(O)                 String[,..]      example.com,test.com[,...]
input_file(O)              String
stages(O)                  String[,..]      blobs|backup|data|account data=blobs+backup[,...]    blobs,backup,account
compress(O)                Boolean          true|false                                           true
checkDigest(O)             Boolean          if false skip digest calculation and check           true
overwrite(O)               Boolean          true|false                                           false
threads(O)                 Integer                                                               1
hsm(O)                     Boolean          true|false                                           true
notifications(O)           Email Address
ignore_partial(O)          Boolean          true|false                                           false
drop_network_backup(O)     Boolean          true|false                                           false
read_error_threshold(O)    Integer

(M) == mandatory parameter, (O) == optional parameter

Usage example:

zxsuite powerstore doMailboxMove mail2.example.com accounts john@example.com stages data,account
Move mailbox for account john@example.com to mail2.example.com host

doMailboxMove Details

  • When moving a domain, each account from the current server is enumerated and moved sequentially.

  • The mailbox is set to maintenance mode when it’s moved and will be placed into its original state after all emails are moved (after the LDAP stage).

  • Operation is stopped if 5% or more write errors are encountered on items being moved. Beware that the current mailbox may remain in maintenance mode.

  • Single-mailbox moves will not start if the destination server does not have enough space available or the user just belongs to the destination host.

  • All data is moved at a low-level and will not be changed except for some small things like mailbox id.

  • The operation is made up of 3 stages: blobs|backup|account. For each mailbox:

    • blobs: All blobs are copied from the source server to the destination server.

    • backup: All backup entries are copied from the source server to the destination server.

    • ldap: zimbraMailHost LDAP attribute is updated and all account caches are flushed.

  • All of the stages are executed sequentially. If a single stage is specified, the mailbox is parked in maintenance mode throughout the entire operation. On success, the mailbox will be placed into its original state.

  • Initially, all blob items will be stored in the destination server primary volume.

  • All volumes' compression options are taken.

  • The MailboxMove operation can be executed if and only if no others operations are running on the source server.

  • The HSM option applies current HSM policies. It runs after each mailbox is successfully moved. On any run new, items will be moved.

Zextras Powerstore Attachment Indexing

How Indexing Works

A new indexing engine has been added to Zextras Powerstore to index attachment contents.

The indexing engine works together with Zimbra’s default engine. The main Zimbra indexing process analyzes the contents of an item, splitting it into several parts based on the MIME parts of the object. Next, Zimbra handles the indexing of known contents - plaintext - and passes the datastream on to the Zextras Powerstore handlers for all other content.

The indexing engine includes an indexing cache that speeds up the indexing process of any content that has already been analyzed. Datastreams over 10Kb are cached by default, and the cache hold 10000 entries, while smaller datastreams are not cached as the cache benefits only apply to large datastreams.

Indexed Formats

Web

Extension Parser Content-type

asp

HtmlParser

application/x-asp

htm

HtmlParser

application/xhtml+xml

html

HtmlParser

application/xhtml+xml, text/html

shtml

HtmlParser

application/xhtml+xml

xhtml

HtmlParser

application/xhtml+xml

Documents

Extension Parser Content-type

rtf

RTFParser

application/rtf

pdf

PDFParser

application/pdf

pub

OfficeParser

application/x-mspublisher

xls

OfficeParser

application/vnd.ms-excel

xlt

OfficeParser

application/vnd.ms-excel

xlw

OfficeParser

application/vnd.ms-excel

ppt

OfficeParser

application/vnd.ms-powerpoint

pps

OfficeParser

application/vnd.ms-powerpoint

mpp

OfficeParser

application/vnd.ms-project

doc

OfficeParser

application/msword

dot

OfficeParser

application/msword

msg

OfficeParser

application/vnd.ms-outlook

vsd

OfficeParser

application/vnd.visio

vst

OfficeParser

application/vnd.visio

vss

OfficeParser

application/vnd.visio

vsw

OfficeParser

application/vnd.visio

xlsm

OOXMLParser

application/vnd.ms-excel.sheet.macroenabled.12

pptm

OOXMLParser

application/vnd.ms-powerpoint.presentation.macroenabled.12

xltx

OOXMLParser

application/vnd.openxmlformats-officedocument.spreadsheetml.template

docx

OOXMLParser

application/vnd.openxmlformats-officedocument.wordprocessingml.document

potx

OOXMLParser

application/vnd.openxmlformats-officedocument.presentationml.template

xlsx

OOXMLParser

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

pptx

OOXMLParser

application/vnd.openxmlformats-officedocument.presentationml.presentation

xlam

OOXMLParser

application/vnd.ms-excel.addin.macroenabled.12

docm

OOXMLParser

application/vnd.ms-word.document.macroenabled.12

xltm

OOXMLParser

application/vnd.ms-excel.template.macroenabled.12

dotx

OOXMLParser

application/vnd.openxmlformats-officedocument.wordprocessingml.template

ppsm

OOXMLParser

application/vnd.ms-powerpoint.slideshow.macroenabled.12

ppam

OOXMLParser

application/vnd.ms-powerpoint.addin.macroenabled.12

dotm

OOXMLParser

application/vnd.ms-word.template.macroenabled.12

ppsx

OOXMLParser

application/vnd.openxmlformats-officedocument.presentationml.slideshow

odt

OpenDocumentParser

application/vnd.oasis.opendocument.text

ods

OpenDocumentParser

application/vnd.oasis.opendocument.spreadsheet

odp

OpenDocumentParser

application/vnd.oasis.opendocument.presentation

odg

OpenDocumentParser

application/vnd.oasis.opendocument.graphics

odc

OpenDocumentParser

application/vnd.oasis.opendocument.chart

odf

OpenDocumentParser

application/vnd.oasis.opendocument.formula

odi

OpenDocumentParser

application/vnd.oasis.opendocument.image

odm

OpenDocumentParser

application/vnd.oasis.opendocument.text-master

ott

OpenDocumentParser

application/vnd.oasis.opendocument.text-template

ots

OpenDocumentParser

application/vnd.oasis.opendocument.spreadsheet-template

otp

OpenDocumentParser

application/vnd.oasis.opendocument.presentation-template

otg

OpenDocumentParser

application/vnd.oasis.opendocument.graphics-template

otc

OpenDocumentParser

application/vnd.oasis.opendocument.chart-template

otf

OpenDocumentParser

application/vnd.oasis.opendocument.formula-template

oti

OpenDocumentParser

application/vnd.oasis.opendocument.image-template

oth

OpenDocumentParser

application/vnd.oasis.opendocument.text-web

sxw

OpenDocumentParser

application/vnd.sun.xml.writer

Packages and Archives

Extension Parser Content-Type

z

CompressorParser

application/x-compress

bz

CompressorParser

application/x-bzip

boz

CompressorParser

application/x-bzip2

bz2

CompressorParser

application/x-bzip2

gz

CompressorParser

application/gzip

gz

CompressorParser

application/x-gzip

gzip

CompressorParser

application/x-gzip

xz

CompressorParser

application/x-xz

tar

PackageParser

application/x-tar

jar

PackageParser

application/java-archive

7z

PackageParser

application/x-7z-compressed

cpio

PackageParser

application/x-cpio

zip

PackageParser

application/zip

rar

RarParser

application/x-rar-compressed

txt

TXTParser

text/plain

Parser Controls

Parsers can be turned on or off by changing the related value to true or false via the zxsuite config CLI command.

Attribute Parsers

pdfParsingEnabled

PDFParser

odfParsingEnabled

OpenDocumentParser

archivesParsingEnabled

CompressorParser, PackageParser, RarParser

microsoftParsingEnabled

OfficeParser, OOXMLParser, OldExcelParser

rtfParsingEnabled

RTFParser

e.g. to disable PDF parsing run:

zxsuite config server set server.domain.com attribute pdfParsingEnabled value false

By default, all parsers are active.

Zextras Powerstore CLI

zxsuite powerstore

Table 1. zxsuite powerstore
Command Description Usage

+setHsmPolicy

Add an additional policy to HSM.

zxsuite powerstore +setHsmPolicy {hsm_policy}

doCheckBlobs

Start the doCheckBlobs operation.

zxsuite powerstore doCheckBlobs {start} [attr1 value1 [attr2 value2…​]]

doCreateVolume

Create a volume on the server

zxsuite powerstore doCreateVolume

doDeduplicate

Start a deduplication operation.

zxsuite powerstore doDeduplicate {volume_name} [attr1 value1 [attr2 value2…​]]

doDeleteVolume

Delete a specific volume on the server

zxsuite powerstore doDeleteVolume {volume_name}

doMailboxMove

Perform a Mailbox Move on a single account or a domain. In order to remove orphaned mailboxes you should run the command 'zxsuite powerstore doPurgeMailboxes all ignore_retention true'.

zxsuite powerstore doMailboxMove {destinationHost} [attr1 value1 [attr2 value2…​]]

doMoveBlobs

Start the doMoveBlob operation.

zxsuite powerstore doMoveBlobs [attr1 value1 [attr2 value2…​]]

doPurgeMailboxes

Delete all leftover data of a mailbox that has been moved to another server

zxsuite powerstore doPurgeMailboxes {Mailbox id or "all" to delete all mailboxes moved more than 30 days ago} [attr1 value1 [attr2 value2…​]]

doRemoveHsmPolicy

Remove a policy from HSM.

zxsuite powerstore doRemoveHsmPolicy {hsm_policy}

doRemoveOrphanedBlobs

remove orphaned blobs

zxsuite powerstore doRemoveOrphanedBlobs [attr1 value1 [attr2 value2…​]]

doRestartService

restart a given service

zxsuite powerstore doRestartService {service_name}

doStartService

start a given service

zxsuite powerstore doStartService {service_name}

doStopAllOperations

Stops all running operations and empties the operations queue

zxsuite powerstore doStopAllOperations

doStopOperation

Stop a single running operation

zxsuite powerstore doStopOperation {operation_uuid}

doStopService

stop a given service

zxsuite powerstore doStopService {service_name}

doUpdateVolume

Update a specific volume on the server

zxsuite powerstore doUpdateVolume

doVolumeToVolumeMove

Start the doMoveVolumeBlobs operation.

zxsuite powerstore doVolumeToVolumeMove {source_volume_name} {destination_volume_name} [attr1 value1 [attr2 value2…​]]

getAllOperations

Displays all running and queued operations

zxsuite powerstore getAllOperations [attr1 value1 [attr2 value2…​]]

getAllVolumes

Prints all the volumes present in the server

zxsuite powerstore getAllVolumes

getHsmPolicy

Prints all the policies

zxsuite powerstore getHsmPolicy

getMovedMailboxes

Get list of mailboxes moved and no more used

zxsuite powerstore getMovedMailboxes

getProperty

Get configuration properties

zxsuite powerstore getProperty [attr1 value1 [attr2 value2…​]]

getServices

show current status of all services for this module

zxsuite powerstore getServices

getVolumeStats

Show space occupation, number of items and blobs contained in a volume

zxsuite powerstore getVolumeStats {volume_name} [attr1 value1 [attr2 value2…​]]

monitor

Monitor a running operation

zxsuite powerstore monitor {operation_uuid} [attr1 value1 [attr2 value2…​]]

runBulkDelete

Run bulk delete requests

zxsuite powerstore runBulkDelete [attr1 value1 [attr2 value2…​]]

setHSMPolicy

Set the default HSM policy

zxsuite powerstore setHSMPolicy {hsm_policy}

setProperty

Set configuration property

zxsuite powerstore setProperty {property_name} {property_value}

testS3Connection

Test the connection to an S3 bucket.

zxsuite powerstore testS3Connection {s3BucketConfigurationUuid}