Hierarchical Storage Management

Hierarchical Storage Management#

The HSM feature requires a separate license (called Storages HSM in the Carbonio Admin Panel' s Subscriptions section), therefore it may not be available on your Carbonio installation.

HSM is a data storage technique that moves data between different stores according to a defined policy.

The most common use of the HSM technique is the move of old data from a faster-but-expensive storage device (the Primary Volume) to a slower-but-cheaper one (the Secondary Volume) based on the following premises:

  • Fast storage is more expensive than slow storage

  • Old data are on average much less frequently accessed than new data

Therefore, storing old data on slower storage devices is acceptable, as users can afford to wait more time to retrieve a (very) old item (e.g., a mail or a document stored in Files), while they want that recent items be always available quickly. The advantages of HSM are clear: on the one side, lowering the overall storage cost since only a small part of your data needs to be on costly storage, and on the other side improving the overall user experience.

Defining Policies#

An HSM policy is a string that describes what should be moved, for example:

all:before:-20days

When this policy is applied, all the messages arrived before 20 day ago are moved: if today is March 21st, this means all items whose date is before March 1st will be moved.

A policy can consist of a single rule that is valid for all item types (Simple policy) or multiple rules valid for one or more item types (Composite policy).

Accounts and Domains

Contrary to the policies that can be defined in the Carbonio Admin Panel (see section Create a New Policy), from the CLI you can create a policy that is applied only on given accounts or domain, for example the policy:

all:before:-10days domain:example.com

will move all items older than 10 days that are in the domain example.com; policy:

all:before:-30days account:john.doe@example.com

will move all items older than 30 days that belong to the account john.doe@example.com.

Source and Destination Volumes

It is possible to choose the source volume(s) and the destination volume for a policy. By default, if neither of them is specified in a policy, items are moved from the Primary Volume to the Current Secondary Volume.

Note

In a policy can appear multiple comma-separated Source Volumes but only one Destination Volume.

If you define only Source Volume(s), the items will be moved to the Current Secondary Volume, while if you specify only the Destination Volume, all the items from the Primary Volume. In the policy you need to used the **ID of the volume, that you can retrieve using command carbonio powerstore getallvolumes. Here are some examples:

all:before:-10 source:1,2 destination:3

All items older than 10 days from Volumes 1 and 2 will be moved to Volume 3.

all:before:-10 source:1

All items older than 10 days from Volume 1 will be moved to the Current Secondary Volume, regardless of its id.

all:before:-10 destination:3

All items older than 10 days from the Primary Volume will be moved to Volume 3.

Policies Management#

Carbonio defines three CLI commands for the management of policies:

  1. carbonio powerstore getHsmPolicy

    This command takes no parameter and lists all the policies defined.

  2. carbonio powerstore setHsmPolicy

    This command takes one parameter, which is the string containing the policy and replaces the new policies to all the existent policies, so use it with care!

  3. carbonio powerstore +setHsmPolicy

    This command takes one parameter, which is the string containing the policy and adds it after all the other existent policies, if any.

The policies are evaluated in the order in which they are displayed. For example, suppose you have one Primary Volume (with id 1), two Destination Volumes (with id 3 and 5), and a few domains, one of which with high traffic (example.com). If you define the following policies:

all:before:-10days domain:example.com destination:3
all:before:-30days destination:5

The result is that all items older that 10 days from the high-traffic domain will be moved to Destination Volume 3, and all the other items older than 30 will be moved to the other Destination Volume, 5.

Running Policies#

From the CLI, you can manually run the policies using command doMoveBlobs.

Hint

Policies can be scheduled to run periodically from the Carbonio Admin Panel, see HSM Settings.

Once the move is started, the following operations are performed:

  • Carbonio Storage scans through the Primary Volume to see which items comply with the defined policy

  • All the Blobs of the items found in the first step are copied to the Secondary Volume

  • The database entries related to the copied items are updated to reflect the move

  • If the second and the third steps are completed successfully (and only in this case), the old Blobs are deleted from the Primary Volume

The Move operation is stateful - each step is executed only if the previous step has been completed successfully - so there is no risk of data loss during a doMoveBlobs operation. The syntax of the command is

zextras$ carbonio powerstore doMoveBlobs start

Optionally, you can add a policy on the command line to run it only once, for example