Carbonio Upgrade from 24.5.0 to 24.7.0#
Carbonio does not have any installer: whenever new versions are released, the Zextras repositories are updated and packages are available for installation along with the other system updates. Therefore, the upgrade procedure is usually a very quick activity, carried out by means of a few commands to be executed on each Node.
The upgrade procedure consists of a preliminary task, followed by the proper upgrade, which is the same for each Node. Regardless of the version currently installed, the successful upgrade will always bring Carbonio to the latest version available, currently 24.7.0.
Depending on the packages upgraded, and the version from which you are upgrading, some manual step may be required. Please read carefully section Upgrade Checklist before starting the upgrade, to see if you need to carry out some additional task.
Note
Only the three previous versions are supported for upgrade. If you install from an older version, you may need to execute further tasks to ensure proper functioning of Carbonio.
Moreover, in some cases, incompatibilities may seldom arise in the upgrade of third-party software, which may lead to some additional manual steps to be carried out. Section Additional Manual Steps below contains information to prevent or fix these issues.
Release 24.3.0 of Carbonio was a major milestones that introduced:
support for PostgreSQL, version 16. If you did not yet upgraded, we strongly suggest to do so. Moreover, if you plan to upgrade to Ubuntu 22.04 (see next point), it is mandatory to upgrade to PostgreSQL 16.
support for Ubuntu 22.04 (Jammy Jellyfish).
Upgrade Checklist#
The upgrade to 24.7.0 impacts the following Roles, packages, or software, which require some manual interaction before, during, or after the procedure.
Carbonio 24.7.0 can be installed on Ubuntu 22.04 and RHEL 9 (BETA support), so you can proceed with the upgrade from Ubuntu 22.04 or RHEL 9 following the usual upgrade procedures: please refer to the respective upstream documentation: Ubuntu | RHEL.
If you plan to upgrade to RHEL 9, make sure to satisfy the specific requirements. Carbonio 24.7.0 supports also Ubuntu 22.04, so you can proceed with the upgrade from Ubuntu 20.04, according to the procedure described in section Upgrade Carbonio and Operating System.
PostgreSQL can be upgraded from version 12 to version 16. This step is mandatory if you also plan to upgrade from Ubuntu 20.04 to Ubuntu 22.04, because version 12 is no longer available on Ubuntu 22.04. Please follow the instructions in section Upgrade to PostgreSQL 16.
Due to a new package installed, you will see a conflict between the
installed package config-generator and the new
service-discover-template package on both RHEL and Ubuntu. To
fix this conflict, before upgrading, you need to remove the
existing package and install the new one. You can do this with the
following commands
# apt install service-discover-template
# rpm -e --nodeps config-generator
Right after the command completes, proceed with the upgrade.
During the upgrade of Prometheus, you will be notified of a conflict in a file and asked for a solution. Please refer to Section Carbonio Monitoring below for directions.
If you have any Delegated administrators in your Carbonio infrastructure, you need to carry out a task after the upgrade has successfully been completed. Please refer to Section Delegated Administrators below for directions.
Starting from Carbonio 24.7.0, the Backup uses an Account attribute, inherited from the CoS, to define if the account must be included on the backup or not. The upgrade procedure takes care of migrating the accounts and apply the attribute.
However, if some problem happens, the global administrator may receive an e-mail notification. If this happens, please refer to Section Backup below for directions.
Hint
For improved security, to prevent any data loss, it is suggested to make a backup or take a snapshot (if you are using an hypervisor) of each Node before upgrading.
Preliminary Tasks#
Note
You do not need to execute the following tasks if you
upgrade using the Ansible procedure. Indeed, in that case, the zmslapcat
command is run by Ansible, which will save the LDAP dump in the
/bck directory.
As a preliminary task, we backup the LDAP data of the Directory Server. On a Multi-Server installation, execute the following commands on the Node with the Directory Server Role installed.
-
Make a dump of the LDAP Database, especially if the if the upgrade includes the Directory Server. This can be done using the command (as the
zextrasuser)zextras$ /opt/zextras/libexec/zmslapcat /tmpNote
The dump will be saved in the
/tmp/directory, so make sure to copy it to a safe location. Make a backup copy of file
/opt/zextras/conf/localconfig.xmland store it in a safe place
Upgrade to PostgreSQL 16#
Please read carefully the whole section and the instructions for your Operating System.
Warning
The upgrade to PostgreSQL 16 is mandatory if you plan to upgrade Ubuntu 20.04 to 22.04, because in the newest release the support for PostgreSQL 12 has been dropped.
The upgrade to PostreSQL requires to execute a couple of CLI commands and should require little downtime. However, you should take into account that the time to migrate is proportional to all the data stored, so you should consider PostgreSQL to be offline for the time needed to carry out the migration.
The procedure differs slightly between Ubuntu 20.04, Ubuntu 22.04 (in the installation of the repository), while in RHEL 8 it is quite different. For this reason, we separate the directions of RHEL from Ubuntu.
In case after the upgrade you find in Postgres’ log files some error messages like the following one, please refer to Section Postgresql Upgrade Issue:
2024-03-19 12:28:14.209 UTC [909825] HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE activesync REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
2024-03-19 12:28:19.669 UTC [909915] WARNING: database "abq" has a collation version mismatch
Ubuntu 20.04 and 22.04#
The commands in this section must be executed as the root user
and, except for the Repository Configuration, are the same on both
versions of Ubuntu.
Repository Configuration
The following two commands work on both versions of Ubuntu, but the second one is deprecated in Ubuntu 22.04 and will raise a warning. Hence, to make sure the process is flawless, refer to the Ubuntu 22.04 tab for the command if you are installing on that version.
# sh -c 'echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# sh -c 'echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# wget -O- "https://www.postgresql.org/media/keys/ACCC4CF8.asc" | \
gpg --dearmor | sudo tee /usr/share/keyrings/postgres.gpg > \
/dev/null
# chmod 644 /usr/share/keyrings/postgres.gpg
# sed -i 's/deb/deb [signed-by=\/usr\/share\/keyrings\/postgres.gpg] /' /etc/apt/sources.list.d/pgdg.list
Install Packages
# apt update
# apt -y install postgresql-16
Prepare for migration
In this step some checks and preparatory tasks are executed.
-
Check installed PG clusters and version
# pg_lsclusters -
Stop the running service
# systemctl stop postgresql
-
Rename the clean 16 DB
# pg_renamecluster 16 main main_pristine
Upgrade and restart service
-
Upgrade the old DB to the new “main” DB
# pg_upgradecluster 12 main
-
Start the service
# systemctl start postgresql
-
Check that everything is working properly
# pg_lsclusters
(Optional) remove unused DB
At this point, all data have been migrated to DB 16, so, if you verify that Carbonio operates properly and all the data are present, you can remove the unused Databases.
Hint
You can keep the old Database, back it up, and remove it at a later point.
-
Drop the old DB
# pg_dropcluster 12 main --stop
-
Drop the pristine DB
#pg_dropcluster 16 main_pristine --stop
RHEL 8 and 9#
Repository Configuration
The PostgreSQL repositories are different in RHEL 8 and RHEL9; the rest of the procedure is exactly the same.
# dnf -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
# dnf -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
Package installation and checks
-
Install PostgreSQL16 packages
# dnf install postgresql16 postgresql16-server
-
Init the DB
# /usr/pgsql-16/bin/postgresql-16-setup initdb
-
Check that everything is correct and clusters are compatible. This command does not alter any existing data.
# su - postgres -c '/usr/pgsql-16/bin/pg_upgrade -b \ /usr/pgsql-12/bin/ -B /usr/pgsql-16/bin/ -d \ /var/lib/pgsql/12/data/ -D /var/lib/pgsql/16/data/ -c'
Upgrade and migrate
-
Stop the PostgreSQL 12 service
# systemctl stop postgresql-12.service
-
Run the upgrade
# su - postgres -c '/usr/pgsql-16/bin/pg_upgrade -b \ /usr/pgsql-12/bin/ -B /usr/pgsql-16/bin/ -d \ /var/lib/pgsql/12/data/ -D /var/lib/pgsql/16/data/'
-
Migrate the pg_hba.conf file from the previous version
# mv /var/lib/pgsql/16/data/pg_hba.conf /var/lib/pgsql/16/data/pg_hba.conf_orig # cp /var/lib/pgsql/12/data/pg_hba.conf /var/lib/pgsql/16/data/pg_hba.conf # chown postgres:postgres /var/lib/pgsql/16/data/pg_hba.conf
-
Start the new DB
# systemctl start postgresql-16.service
-
Disable the old DB and enable the new one
# systemctl mask postgresql-12.service # systemctl enable postgresql-16.service
(Optional) remove unused DB
At this point, all data have been migrated to DB 16, so, if you verify that Carbonio operates properly and all the data are present, you can remove the unused Databases.
-
Check the new Database
# su - postgres -c '/usr/pgsql-16/bin/vacuumdb --all --analyze-in-stages'
-
Delete the old cluster’s data files
# /var/lib/pgsql/delete_old_cluster.sh
Update pgpool-II
You need to update also the pgpool-II package so it matches
PostgresQL’s correct version, 16.
-
First, remove the installed package (and repository definition if installed)
# dnf remove pgpool-II pgdg-redhat-repo
-
Install the correct pgpool version
# dnf install https://www.pgpool.net/yum/rpms/4.5/redhat/rhel-8-x86_64/pgpool-II-pg16-4.5.1-1pgdg.rhel8.x86_64.rpm
# dnf install https://www.pgpool.net/yum/rpms/4.5/redhat/rhel-9-x86_64/pgpool-II-pg16-4.5.1-1pgdg.rhel9.x86_64.rpm
Upgrade Nodes#
There are two equivalent methods to upgrade a Carbonio infrastructure: automatically using Ansible, or manually, node by node. While we suggest to use the former procedure, the latter allows you to follow closely, node by node, the upgrade process.
Upgrade with Ansible#
This method works if you have a working Ansible installation.
In order to carry out the upgrade, you need to access the Control Node (see Section Ansible Setup) and issue the following command
$ ansible-galaxy collection install zxbot.carbonio_upgrade
Hint
This command can be run by any user, preferably by the one who installed the playbook.
The above command will install the necessary infrastructure to use for
the Carbonio upgrade. Next, you need to move to the directory where
you have the inventory file of your installation (it is called
data in the Ansible Setup, the issue the following
command
$ ansible-playbook zxbot.carbonio_upgrade.carbonio_upgrade \
-u root -i carbonio_inventory
In case the procedure does not complete successfully, you can run the command a second time, during which the tasks that successfully run will be skipped. For details, refer to Section Troubleshooting Ansible Installation.
Manual Upgrade#
Remember to start the upgrade from the node featuring the Directory Server, then all the other in the same order of installation.
As the zextras user, run:
zextras$ zmcontrol stop
# apt clean
# dnf clean all
# apt update && apt dist-upgrade
# dnf upgrade --best --allowerasing
# pending-setups -a
Once the upgrade has completed successfully, run command:
# reboot
Additional Manual Steps#
In the upgrade to 24.7.0, the following manual steps are required.
GPG Key of Grafana in Ubuntu 22.04#
If you migrate from Ubuntu 20.04 to 22.04 and use Monitoring Role’s Grafana, you need to update the location on which its GPG key is stored to prevent warning during upgrades. Log in to the Node on which the Carbonio Monitoring Role is installed and execute the following commands.
-
Remove existent repository
# add-apt-repository -r "deb https://packages.grafana.com/oss/deb stable main"
-
Retrieve GPG key
# wget -q -O - https://packages.grafana.com/gpg.key | \ gpg --dearmor | sudo tee /usr/share/keyrings/grafana.gpg > \ /dev/null
-
Add repository
# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] \ https://packages.grafana.com/oss/deb stable main" | sudo tee -a \ /etc/apt/sources.list.d/grafana.list
New Package Installation#
A new package must be installed on the Node featuring the Database Connector Role. Log in to that Node and execute command
# apt install carbonio-storages
# dnf install carbonio-storages
Carbonio Monitoring#
While upgrading the carbonio-prometheus package, you will be
prompted with the following text:
Configuration file '/etc/carbonio/carbonio-prometheus/prometheus.yml'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
You must replace the existing file with the new one, therefore answer Y or I.
Delegated Administrators#
In this release, you need to initialise again the domain(s) for Delegation: for each domain that has active Delegations, in the Carbonio Admin Panel go to and click the INIT DOMAIN button.
Backup#
If the Global Administrator receives an e-mail notification about some
accounts not migrated to use the new backupEnabled attribute, run
the following commands as the zextras user to fix the issue.
First, stop the backup migration service
zextras$ carbonio backup dostopservice migrate-backup-enabled-setting
Then, start the service again to trigger the migration
zextras$ carbonio backup dostartservice migrate-backup-enabled-setting