Carbonio Mesh

Carbonio Mesh is Zextras solution for service discovery and service mesh, a mechanism that secures communication of registered applications, implementing access control to on-premises or external resources with a single solution, including the management of SSL encryption certificates

Carbonio Mesh is used by Carbonio to add health checking/fault detection, dynamic and secure routing between its components (excluding faulty instances). Moreover, it also works as an application level firewall allowing the exchange of only the information necessary for the functioning of Carbonio as it was designed.

The setup of Carbonio Mesh depends on the Carbonio infrastructure, if a Single-Server or a Multi-Server.

Single-Server Setup

Run setup

The configuration is automatically generated by

# service-discover setup $(hostname -i) --password=MY_SECURE_PASSWORD


Replace MY_SECURE_PASSWORD with a strong enough password.

This command will:

  • find the hostname IP address (hostname -i)

  • set the cluster credential password to MY_SECURE_PASSWORD, which is used for setups, management, and to access the administration GUI. See section Carbonio Mesh Administration Interface for more information.

  • store the setup in file /etc/zextras/service-discover/cluster-credentials.tar.gpg


    Make sure to store the password in a safe place (like e.g., a password manager). In case the password is lost or the credential file becomes corrupted and unusable, you can Regenerate the Credentials.

Multi-Server Setup

The Multi-Server setup is slight more complex. as it requires to run commands on all the nodes.

Run setup

The setup is the same as in Single-Server Setup, except that the command must be run on the Directory-Server node.

# service-discover setup $(hostname -i) --password=MY_SECURE_PASSWORD

Copy Credentials

The outcome of the previous command is a GPG key that you need to copy to all other nodes.

Assuming that you have nodes proxy, mta, store, and logger (see the Multi-Server example installation scenario, use the following commands, provided you use the correct hostname or IP address of the nodes.

# scp /etc/zextras/service-discover/cluster-credentials.tar.gpg proxy:/etc/zextras/service-discover/cluster-credentials.tar.gpg

# scp /etc/zextras/service-discover/cluster-credentials.tar.gpg mta:/etc/zextras/service-discover/cluster-credentials.tar.gpg

# scp /etc/zextras/service-discover/cluster-credentials.tar.gpg store:/etc/zextras/service-discover/cluster-credentials.tar.gpg

# scp /etc/zextras/service-discover/cluster-credentials.tar.gpg logger:/etc/zextras/service-discover/cluster-credentials.tar.gpg

Complete setup on all nodes

Log in to each nodes and run the command, making sure to use the same password used in the first step.

# service-discover setup $(hostname -i) --password=MY_SECURE_PASSWORD

Regenerate the Credentials

Whenever the cluster credential password of file /etc/zextras/service-discover/cluster-credentials.tar.gpg are unaccessible, it is possible to generate a new file password and credentials file.

In those cases, the command consul acl bootstrap will terminate with an error message similar to:

Failed ACL bootstrapping: Unexpected response code: 403 (Permission denied: ACL bootstrap no longer allowed (reset index: 908))

Before attempting the recover, be prepared for a downtime of the Carbonio Mesh service for the whole duration of the procedure.

The procedure is the same for Single-Server and Multi-Server, but on the Multi-Server there are a few more steps to carry out.

Preliminary Tasks

In case of a Single-Server node, log in to it and skip to the next step.

On a Multi-Server, you need to identify the Carbonio Mesh leader node node and log into it. Most of the times, this is the Directory-Server node, whose IP address is retrieved using the command below.

# zmprov gas service-discover

To make sure you are on the leader, use the following command.

# wget -qO -

The output will be an IP address and a port, for example If this IP is different from the Directory Server’s, log in to the latter on (


All the commands must be run on the leader node, unless differently specified.

Step 1. Wipe Old Credentials

The first task, to be executed as the service-discover user, is to write a reset index, to allow a new ACL token to be generated.

# sudo -u service-discover bash -c "echo 908 > /var/lib/service-discover/data/acl-bootstrap-reset"

Then stop the service discover service.

# systemctl stop service-discover

Finally, remove all certificates related to service discover.

# rm /var/lib/service-discover/*.pem

Step 2. Generate New Credentials

Run the setup as a first instance.

# service-discover setup --first-instance --password=MY_SECURE_PASSWORD

This is essentially the same command as the one used during the Single-Server Setup of Carbonio Mesh, the only difference being that in this case we use the explicit IP address and run it as first instance.

Optionally, verify the ACL token using the commands

# export CONSUL_HTTP_TOKEN=$(gpg -qdo - /etc/zextras/service-discover/cluster-credentials.tar.gpg | tar xOf - consul-acl-secret.json | jq .SecretID -r)
# consul members
       Node              Address              Status  Type    Build  Protocol  DC   Segment  alive   server  1.9.3  2

On a Single-Server the procedure has been completed. Make sure to store the new credentials in a safe place!

Multi-Server Final Task

On a Multi-Server, you need to copy the credentials file on all other nodes, for example using scp, like explained in section Multi-Server Setup.

Finally, log in to all other nodes and repeat on each of them the setup using the following commands

# rm /var/lib/service-discover/*pem
# service-discover setup $(hostname -i) --password=MY_SECURE_PASSWORD