Troubleshooting Carbonio Mesh#
Carbonio Mesh is one of the main components of Carbonio, and is based on HashiCorp’s Consul. This page is meant to provide some of the most used CLI commands to inspect and fix any issues that may arise with the use of Consul.
It is possible to interact with Consul on any node of a cluster but remember that the consul operates by default on the current node. To operate on a different node, you need to explicitly specify it, for example this command show all services running on node with #ID 7ea9631e
# consul catalog services -node 7ea9631e
Warning
Some of the commands listed on this page can be used to or modify significantly or remove a service or a node from Consul, thus potentially disrupting Carbonio Mesh. These commands are marked with an icon: Use them with care!
Retrieve Token#
Whenever you want to use Consul, the first operation is to retrieve the bootstrap-token, to allow connection and interaction with the service.
# service-discover bootstrap-token
Hint
You need to provide the cluster credential password, which
is stored in /var/lib/service-discover/password.
Export the token, which is a string similar to e5a4966f-a83e-689d-618d-08a0fe7e695b
# export CONSUL_HTTP_TOKEN=e5a4966f-a83e-689d-618d-08a0fe7e695b
You can automate the export process by using the following one-liner
# export CONSUL_HTTP_TOKEN=$(gpg -qdo - /etc/zextras/service-discover/cluster-credentials.tar.gpg | tar xOf - consul-acl-secret.json | jq .SecretID -r)
Common Cluster Operations#
The following commands are used to inspect a cluster:
# consul members
Node Address Status Type Build Protocol DC Segment
server-example-com 10.10.10.12:8301 alive server 1.10.12 2 dc1 <all>
agent1-example-com 10.10.10.24:8301 alive client 1.10.12 2 dc1 <default>
A similar command is consul catalog nodes -detailed, which outputs the node’s ID, used to run Consul on a given node.
# consul force-leave agent1-example-com
Common Service Operations#
These commands allow to retrieve a list of services registered to a Consul cluster and to manipulate them.
# consul catalog services
carbonio-address-book
carbonio-address-book-sidecar-proxy
carbonio-auth
[...]
To query all services running on a given node, use
consul catalog services -node 7ea9631e. As argument
to -node you can provide either the node’s ID or name.
# consul services register /etc/zextras/service-discover/carbonio-mta.hcl
The /etc/zextras/service-discover/carbonio-mta.hcl is a
JSON file that contains all information required to set up a
service. The file must be a valid JSON file and must conform to
Cunsul’s specification. Discussion about the format and syntax
of that file is outside the scope of this page, please refer to
the official upstream documentation
for more information.
# consul services deregister -id=my-service-id
This command assumes that there is a registered service namend my-service
To deregister a service which has a configuration file (which is the case for Carbonio), simply delete the file and reload the agent on all nodes.
Other Commands#
To reload the consul agent on a node, execute
# consul reload
This is useful when you modify any configuration; the changes are picked up by the agent running on the node and propagated to the server.
To check log files of Carbonio Mesh, use the command
# journalctl -u service-discover
Hint
To follow log files in real time while they are emitted, add
the -f option at the end of the command.
See also
Full CLI reference for Consul is available on the official documentation.