Troubleshooting Carbonio Mesh#

Carbonio Mesh is one of the main components of Carbonio CE, and is based on HashiCorp’s Consul. This page is meant to provide some of the most used CLI commands to inspect and fix any issues that may arise with the use of Consul.

It is possible to interact with Consul on any node of a cluster but remember that the consul operates by default on the current node. To operate on a different node, you need to explicitly specify it, for example this command show all services running on node with #ID 7ea9631e

# consul catalog services -node 7ea9631e


Some of the commands listed on this page can be used to or modify significantly or remove a service or a node from Consul, thus potentially disrupting Carbonio Mesh. These commands are marked with an icon: Use them with care!

Retrieve Token#

Whenever you want to use Consul, the first operation is to retrieve the bootstrap-token, to allow connection and interaction with the service.

# service-discover bootstrap-token


You need to provide the cluster credential password, which is stored in /var/lib/service-discover/password.

Export the token, which is a string similar to e5a4966f-a83e-689d-618d-08a0fe7e695b

# export CONSUL_HTTP_TOKEN=e5a4966f-a83e-689d-618d-08a0fe7e695b

You can automate the export process by using the following one-liner

# export CONSUL_HTTP_TOKEN=$(gpg -qdo - /etc/zextras/service-discover/cluster-credentials.tar.gpg | tar xOf - consul-acl-secret.json | jq .SecretID -r)

Common Cluster Operations#

The following commands are used to inspect a cluster:

Show members of a cluster
# consul members
Node                Address           Status  Type    Build    Protocol  DC   Segment
server-example-com  alive   server  1.10.12  2         dc1  <all>
agent1-example-com  alive   client  1.10.12  2         dc1  <default>

A similar command is consul catalog nodes -detailed, which outputs the node’s ID, used to run Consul on a given node.

Remove a member of a cluster
# consul force-leave agent1-example-com

Common Service Operations#

These commands allow to retrieve a list of services registered to a Consul cluster and to manipulate them.

Show services on a cluster
# consul catalog services

To query all services running on a given node, use consul catalog services -node 7ea9631e. As argument to -node you can provide either the node’s ID or name.

Register a service to the cluster
# consul services register /etc/zextras/service-discover/carbonio-mta.hcl

The /etc/zextras/service-discover/carbonio-mta.hcl is a JSON file that contains all information required to set up a service. The file must be a valid JSON file and must conform to Cunsul’s specification. Discussion about the format and syntax of that file is outside the scope of this page, please refer to the official upstream documentation for more information.

Deregister a service to the cluster
# consul services deregister -id=my-service-id

This command assumes that there is a registered service namend my-service

To deregister a service which has a configuration file (which is the case for Carbonio CE), simply delete the file and reload the agent on all nodes.

Other Commands#

Reload agent

To reload the consul agent on a node, execute

# consul reload

This is useful when you modify any configuration; the changes are picked up by the agent running on the node and propagated to the server.

Carbonio Mesh log files

To check log files of Carbonio Mesh, use the command

# journalctl -u service-discover


To follow log files in real time while they are emitted, add the -f option at the end of the command.

See also

Full CLI reference for Consul is available on the official documentation.