To help reduce the amount of disk space used by hosts on a Server, Journald is used to centralize the logs on a system with multiple hosts.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our customers to deal with issues on their Servers.

Now, let us look into how to centralize Ubuntu logs with journald.

Benefits of centralizing logs on a Server using Journald?

In a standard Linux Server, logs are stored locally on the same system. For a single system / Server, it is manageable but for multiple Systems logs use up a great amount of disk space.

The idea is to create a centralized logging server where hosts sends its logs to a dedicated log management server in real time. This will help to optimize the Server.

A centralized logging solution offers several benefits compared with storing logs on each host:

i. It reduces the amount of disk space needed on each host to store log files.

ii. Logs can be retained for longer as the dedicated log server can be configured with more storage capacity.

iii. Advanced log analysis can be carried out that requires logs from multiple systems and also more compute resources than may be available on the hosts.

iv. Systems administrators can access the logs for all their systems that they may not be able to log in to directly for security reasons.

What to do to centralize logs with Journald on Ubuntu Server?

To enable you centralize logs in an Ubuntu or Debian Server, we will implement some configurations on both the Server and the Client System and follow the steps below;

1. Systemd-journal-remote Installation.

Here, we will install the systemd-journal-remote package on the client and the server. This package ships all the required components which the client and server will use to work with log files.

Start with updating the system on both the client and Server to ensure that the packages and system is up to date. To do this, run the command below;

sudo apt update && sudo apt upgrade

Next, proceed with the installation of the systemd-journal-remote package on both client and the server by running the command below;

sudo apt install systemd-journal-remote

To receive log messages, enable that the systemd components on the Server is enabled and started by running the commands below;

sudo systemctl enable --now systemd-journal-remote.socket
sudo systemctl enable systemd-journal-remote.service

From the above command, the "-now" option tells the system to start the service immediately after installation.

To send the log messages to the server, enable the systemd component on the client system by running the command below;

sudo systemctl enable systemd-journal-upload.service

Next, open ports 19532 and 80 in the UFW firewall of the Server. This is to permit the server to start receiving log messages from the client machine. For Certbot to work, port 80 must be opened which is required in generating the TLS Certificate.

Now run the following commands to allow firewall;

sudo ufw allow in 19532/tcp
sudo ufw allow in 80/tcp

To open port 80 on the client machine, run the command below;

sudo ufw allow in 80/tcp

2. How to install Certbot and generate Certificates.

Here, you need to install the Certbot utility which will help you generate the certificates. You can also set the cron for Certbot to renew certificates automatically before expiry.

On the Server and Client machine, run the following commands to enable the universe repository which houses the Certbot utility;

sudo apt install software-properties-common
sudo add-apt-repository universe
sudo apt update

Now, you can install Certbot on both hosts by running the command below;

sudo apt install certbot

To register the certificates on both the server and client system, run the command below;

sudo certbot certonly --standalone --agree-tos --email ibmimedia@your_domain -d your_domain

From the above certbot command;

"certonly" helps to register the certificate and implements changes on the system.

"standalone" helps to verify the certificate request using the in-built web server of Certbot.

"agree-tos" means that you agree with Let's Encrypt Terms of Service automatically.

"email your_email" represents the email address which Let's Encrypt will use to notify you when the certificate is about to expire.

"-d your_domain" represents the hostname which the certificate will be assigned with as per your system's hostname.

After the certificate generation, the certificate files and keys will be placed in the directory "/etc/letsencrypt/live/your_domain/".

Now, Let's Encrypt CA and intermediate  needs to be put into the same file which Journald will use to verify the authenticity of the certificates on both the client and Server when they communicate with each other.

To download the two certificates from Let's Encrypt website which will be placed into a file "letsencrypt-combined-certs.pem" in the home directory of the Server user, run the command below on both the client and server;

curl -s https://letsencrypt.org/certs/{isrgrootx1.pem.txt,letsencryptauthorityx3.pem.txt} > ~/letsencrypt-combined-certs.pem

Next, move the created file into the Let's Encrypt directory containing the certificates and keys. To do this, run the command below;

sudo cp ~/letsencrypt-combined-certs.pem /etc/letsencrypt/live/your_domain/

3. How to configure the Server

Now, we will look into how to configure the server to use the certificate and key files which was previously generated.

You can implement some configurations in the systemd-journal-remote file "/etc/systemd/journal-remote.conf".

Edit this file with an editor and then uncomment all the lines under the "[Remote]" section while setting the paths to point to the TLS files already generated as shown below;

[Remote]
Seal=false
SplitMode=host
ServerKeyFile=/etc/letsencrypt/live/server.your_domain/privkey.pem
ServerCertificateFile=/etc/letsencrypt/live/server.your_domain/fullchain.pem
TrustedCertificateFile=/etc/letsencrypt/live/server.your_domain/letsencrypt-combined-certs.pem</code

From the above attributes;

"Seal=false" helps to Sign the log data in the journal. Enable this if we need maximum security; otherwise, leave it as false.

"SplitMode=host" tells the system that the logs from the remote clients will be split by host in /var/log/journal/remote. If we would prefer all the logs to be added to a single file set this to SplitMode=false.

"ServerKeyFile" represents the server's private key file.

"ServerCertificateFile"  is the server's certificate file.

"TrustedCertificateFile" is the file containing the Let's Encrypt CA certificates.

Now modify the permissions on the Let's Encrypt directories which contain the certificates and key so that the systemd-journal-remote can read and use them.

Start by changing the permission to make the certificate and private key readable by running the commands below;

sudo chmod 0755 /etc/letsencrypt/{live,archive}
sudo chmod 0640 /etc/letsencrypt/live/server.your_domain/privkey.pem

Next, make changes to the group ownership of the private key to the systemd-journal-remote's group with the command below;

sudo chgrp systemd-journal-remote /etc/letsencrypt/live/server.your_domain/privkey.pem

Now, the systemd-journal-remote service should be started with the command below;

sudo systemctl start systemd-journal-remote.service

You will see that the log collection server is up and running. It will be ready to accept log messages from the client.

4. How to configure the client for logs centralization using Journald

The component which handles log messages to the log collection server is named "systemd-journal-upload".

Start by creating the new user on the client "systemd-journal-upload", by running the adduser command below;

sudo adduser --system --home /run/systemd --no-create-home --disabled-login --group systemd-journal-upload

From the command above;

"system" helps to create a new user as a system user. This gives the user a UID (User Identifier) number under 1000. UID’s over 1000 are usually given to user accounts that a human will use to log in with.

"home /run/systemd" helps to Set /run/systemd as the home directory for this user.

"no-create-home" tells the system not to create the home directory set, as it already exists.

"disabled-login" means that the user cannot log in to the server via, for example, SSH.

"group" helps to create a group with the same name as the user.

Now, set the permissions and ownership of the Let's Encrypt certificate files by running the commands below;

sudo chmod 0755 /etc/letsencrypt/{live,archive}
sudo chmod 0640 /etc/letsencrypt/live/client.your_domain/privkey.pem
sudo chgrp systemd-journal-upload /etc/letsencrypt/live/client.your_domain/privkey.pem

Now, modify "systemd-journal-upload" configuration at the "/etc/systemd/journal-upload.conf" file. Edit this file via an editor to make it look like this;

[Upload]
URL=https://server.your_domain:19532
ServerKeyFile=/etc/letsencrypt/live/client.your_domain/privkey.pem
ServerCertificateFile=/etc/letsencrypt/live/client.your_domain/fullchain.pem
TrustedCertificateFile=/etc/letsencrypt/live/client.your_domain/letsencrypt-combined-certs.pem

Now, you can restart the "systemd-journal-upload" service to assume the new configuration by running the command below;

sudo systemctl restart systemd-journal-upload.service

5. How to test the Server and Client system?

Now, let us test the log messages relaying between the Client and the Server to ensure that the logs is relayed correctly.

The log collection server stores logs from the clients to a directory "/var/log/journal/remote/".

To check that the client's log file is available on the server, run the ls command below;

sudo ls -la /var/log/journal/remote/

The directory contents showing the log file will be printed as shown below;

total 16620
drwxr-xr-x 2 systemd-journal-remote systemd-journal-remote 4096 May 30 16:17 .
drwxr-sr-x+ 4 root systemd-journal 4096 Jun 30 15:55 ..
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 June 1 10:46 'remote-CN=client.your_domain'

Now with the logger utility, create an custom log message on the client and write a log message to check that the server is receiving the client's log messages as expected. Run the logger command below to do this;

sudo logger -p syslog.debug "### TEST MESSAGE from client.your_domain ###"

Here, the "-p syslog.debug" in this command sets the facility and severity of the message. Setting this to syslog.debug will make clear it is a test message.

Next, read the client’s journal file on the server to check that the log messages are arriving from the client. 

This file is a binary log file, so read the file using journalctl with the –file= option that allows us to specify a custom journal file by running the command below;

sudo journalctl --file=/var/log/journal/remote/remote-CN=client.your_domain.journal

You will see the log messages as shown below;

Test log message
. . .
May 28 13:10:09 client root[3576]: ### TEST MESSAGE from client.your_domain ###

Need support in centralizing logs on Ubuntu or Debian Server using Journald? We are available to help you today.