Troubleshooting Carbonio

This page collect various commands and advices for problems that can arise during everyday use of Carbonio and are related to any of the third-party software installed.

Verify nginx Configuration

The configuration of nginx is split across a number of files so in case of errors while it is running, it may not be immediate to track down the source of the problem.

There are two main problems for which nginx may not run correctly or start:

File Ownership

nginx runs with the zextras user, therefor all its configuration file must be owned by that user, otherwise it will not start correctly.

nginx configuration files are stored in the following directories: /opt/zextras/conf/ and /opt/zextras/common/conf. To verify that all files have correct ownership, use the following commands as the zextras user

zextras$ ls -lR /opt/zextras/conf/nginx*
zextras$ ls -l /opt/zextras/common/conf/nginx.conf

Hint

For better readability, you can pipe the first command above using less

zextras$ ls -lR /opt/zextras/conf/nginx* | less

If any of the listed files has owner different from zextras zextras, you need to fix it by using as the root user the command

# chown zextras:zextras -R /opt/zextras/conf/nginx*
# chown zextras:zextras /opt/zextras/common/conf/nginx.conf

The first command will make sure that all the files in the directory are recursively processed.

Syntax Errors

Any syntax error in a configuration file will prevent nginx to start. To verify that the configuration is correct, run as the zextras user the command

zextras$ nginx -tc /opt/zextras/conf/nginx.conf

Any syntactic error, including the configuration file containing it, will be mentioned in the output, for example:

nginx: [emerg] unexpected end of file, expecting ";" or "}" in /opt/zextras/conf/nginx/includes/nginx.conf.main:37
nginx: configuration file /opt/zextras/conf/nginx.conf test failed

You can now edit the file and make any necessary correction.

In case of ownership errors, the output will be for example:

nginx: [emerg] open() "/opt/zextras/conf/nginx/includes/nginx.conf.main" failed (13: Permission denied) in /opt/zextras/conf/nginx.conf:5
nginx: configuration file /opt/zextras/conf/nginx.conf test failed

Use the already mentioned chown command to fix the problem.

PostgreSQL Database Issues

There are some corner cases in which a database becomes inaccessible, a situation that is marked by a message in the log file similar to:

... ERROR [main] [] extensions - SQL [ CREATE SCHEMA IF NOT EXISTS CORE]; ERROR: permission denied for database core

Here the database named core can not be read, causing SQL commands to fail. This kind of error happens whenever a database has the wrong owner. On Carbonio, the name of the user owning one of the databases listed in Table 6 must coincide with the database’s name.

Table 6 Excerpt of PostgreSQL databases list and their owner.

Name	Owner
abq	abq
activesync	activesync
auth	auth
carbonio-docs-connector-db	carbonio-docs-connector-db
carbonio-files-db	carbonio-files-db
carbonio_adm	carbonio_adm
core	core
ha	ha
powerstore	powerstore

Whenever the owner of any of these database is wrong, you can fix the error by executing as the postgres user the command

$ psql -c "ALTER DATABASE core OWNER TO core;"

replace core with the name of database that caused the error.

After you issued the command, you can check that the database has the correct owner using command

$ psql -c "\l core"

Carbonio Mesh

Troubleshooting for Carbonio Mesh can be found in the dedicated page Troubleshooting Carbonio Mesh.

Let’s Encrypt useful Commands

There are a number of commands that prove useful when dealing with certificates generated by Let’s Encrypt.

Check Status

This command outputs a number of useful information about the current certificate, including how many days the certificate is still valid.

zextras$ certbot certificates

Example output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: example.com
Serial Number: serial number
Key Type: ECDSA
Domains: demo.zextras.io
Expiry Date: 2024-01-31 12:50:33+00:00 (VALID: 14 days)
Certificate Path: certificate path /fullchain.pem
Private Key Path: private key path /privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Manual Renew

This command renews the certificates issued by Let’s Encrypt.

zextras$ certbot renew

certbot Timer

The certbot timer triggers a check of the currently installed certificates, which runs twice a day (i.e., every 12 hours). To check the status of the certbot timer, use command

# systemctl status carbonio-certbot.timer

You can check the log of the timer using command

# journalctl -u carbonio-certbot.timer

This proves useful If the output of the previous command reports the status as different from Active: active, like in the following sample:

● carbonio-certbot.timer - Run Carbonio Certbot twice daily
  Loaded: loaded (/lib/systemd/system/carbonio-certbot.timer; disabled; vendor preset: enabled)
  Active: inactive (dead)
 Trigger: n/a
Triggers: ● carbonio-certbot.service

To start an inactive timer, use command

# systemctl start carbonio-certbot.timer

carbonio-certbot.service Systemd Unit

This is the service that actually carries out all the tasks required for renewing the certificates installed on the Carbonio infrastructure. It is triggered by the certbot timer

You can check its log files using

zextras$ journalctl -u carbonio-certbot.service

A sample output of this command is:

-- Logs begin at Wed 2024-71-01 10:15:25 CET, end at Thu 2023-11-09 10:38:56 CET. --
Jan 17 10:08:46 demo.zextras.io systemd[1]: Starting Renew certificates acquired via Carbonio Certbot...
Jan 17 10:08:55 demo.zextras.io systemd[1]: carbonio-certbot.service: Succeeded.
Jan 17 10:08:55 demo.zextras.io systemd[1]: Finished Renew certificates acquired via Carbonio Certbot.
Jan 17 10:11:31 demo.zextras.io systemd[1]: Starting Renew certificates acquired via Carbonio Certbot...
Jan 17 10:11:32 demo.zextras.io systemd[1]: carbonio-certbot.service: Succeeded.
Jan 17 10:11:32 demo.zextras.io systemd[1]: Finished Renew certificates acquired via Carbonio Certbot.

certbot Log File

The operations carried out by certbot are logged to file /var/log/carbonio/letsencrypt/letsencrypt.log. To see it, use either of the following commands: less, more, tail, cat

zextras$ less /var/log/carbonio/letsencrypt/letsencrypt.log

System Troubleshooting

The carbonio service integrates seamlessly with the various system tools like systemctl and journalctl, therefore allowing quicker analysis of the situation and check for any problem should arise.

The syntax is the standard used by systemctl, hence you can use this command to start or stop all the carbonio units

# systemctl start | stop carbonio*

To verify the status of all the carbonio services run command

# systemctl list-unit-files "carbonio*"

More information about each unit you can run

# systemctl list-units "carbonio*"

If any service is in failed status, restart it. For example, if the nginx Carbonio Monitoring exporter has not started you will need to run

# systemctl restart carbonio-prometheus-nginx-exporter.service

# journalctl -u carbonio*

to access the logs produced by all the units.