How To Troubleshoot Common HAProxy Errors

Server Management Service

Need support with troubleshooting Common HAProxy Errors?

This guide is for you.

HAProxy (High Availability Proxy) is used for TCP and HTTP-based applications.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to fix HAProxy related errors.

In this context, we shall look into the fixes to common High Availability Proxy errors.

How to Troubleshoot Common HAProxy Errors ?

There are three main commands and a common log location that we use to start troubleshooting HAProxy errors.

Basically, when we troubleshoot HAProxy, we use these commands in the given order. Then we examine the log file for specific diagnostic data.

i. systemctl – Used to control and interact with Linux services via the systemd service manager.

ii. journalctl – To query and view the logs that are generated by systemd.

iii. haproxy – Used to check HAProxy’s configuration.

iv. /var/log/haproxy.log – This file contains log entries from HAProxy itself detailing TCP and HTTP traffic handled by the server.

Now let us now see how to use them and where we can find additional information about errors.

i. systemctl Commands for HAProxy

To troubleshoot common HAProxy errors using the systemd service manager, the first step is to inspect the state of the HAProxy processes on the system.

The following systemctl commands will query systemd for the state of HAProxy’s processes on most Linux distributions.

$ sudo systemctl status haproxy.service -l –no-pager

The -l a flag will ensure that output is not truncated. The --no-pager flag will make sure that output will go directly to our terminal.

If we omit the --no-pager flag we will be able to scroll through the output using arrow keys. To quit from the pager use the q key.

We will receive output like this:

● haproxy.service – HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 488 (haproxy)
Tasks: 2 (limit: 2344)
. . .
Aug 19 21:31:46 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

The output may be slightly different depending on the Linux distribution we use. But in any case, make a note of the Active line in the output.

If the HAProxy server does not show active, there may be an error. Typically if there is a problem, we will have a line like the following in the output:

Active: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago

If there is a problem with the HAProxy configuration, we suggest using the journalctl command.

ii. journalctl Commands for HAProxy

To inspect the systemd logs for HAProxy, we can use the journalctl command. It will usually indicate whether there is a problem with starting or managing the HAProxy process.

These logs are separate from HAProxy’s request and error logs. journalctl displays logs from systemd that describe the HAProxy service itself, from startup to shutdown. It will also describe any process errors that may encounter along the way.

$ sudo journalctl -u haproxy.service –since today –no-pager

The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day. Using this option will help restrict the volume of log entries that we need to examine when checking for errors.

We should receive output like the following (there may be a few extra lines between the Starting and Started lines depending on the Linux distribution):

Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Starting HAProxy Load Balancer…
. . .
Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

If there is an error, we will have a line in the output that is similar to the following:

Aug 20 19:32:25 yourhostname systemd[1]: Failed to start HAProxy Load Balancer.

If our HAProxy server has errors in the journalctl logs, the next step is to investigate HAProxy's configuration using the haproxy command-line tool.

iii. Troubleshooting with haproxy

Initially, to troubleshoot HAProxy configuration issues, we use the haproxy -c command. The tool will parse HAProxy files and detect errors or missing settings before attempting to start the server.

We run the command like this on Ubuntu, Debian, CentOS, and Fedora-based distributions. We make sure to change the path to the configuration file if we use a different filename or location:

$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg

A working HAProxy configuration will result in an output like the following:

Configuration file is valid

If there is an error in the HAProxy configuration, haproxy -c will detect it and attempt to notify us about the problem.

For instance, attempting to use the bind directive in haproxy.cfg in the wrong location will result in:

[ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword ‘bind’ in ‘global’ section
[ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 232/194354 (199) : Fatal errors found in configuration.

In this example, the bind directive is inside a global configuration section. Hence the HAProxy unknown keyword error.

The message also mentions line 13. We can edit the file and fix or remove the erroneous line without having to search through the file.

Learning how to use haproxy -c to detect and fix errors is useful while troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.

[Failed to track HAProxy errors? We are available 24*7. ]

More about HAProxy Log Files ?

HAProxy log files are a very helpful resource for troubleshooting.

Generally, any error that we receive in a browser or HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.

On Ubuntu and Debian based Linux distributions, the haproxy package includes scripts that configure log output in /var/log/haproxy.log.

Similarly, on CentOS, Fedora, and other RedHat-derived Linux distributions, haproxy does not output to a log file by default.

When we troubleshoot HAProxy using its log file, we examine /var/log/haproxy.log for errors using a tool.

For example, to view the last two lines of the log using tail, we run:

$ sudo tail -n 2 /var/log/haproxy.log

An example error will resemble the following lines, regardless of which Linux distribution we are using to run your HAProxy server:

Aug 20 19:36:21 d6cdd0c71489 haproxy[19202]: [ALERT] 258/134605 (19202) : Proxy ‘app’, server ‘app1’ [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you’re running on a LAN where you’re certain to trust the server’s certificate, please set an explicit ‘verify none’ statement on the ‘server’ line, or use ‘ssl-server-verify none’ in the global section to disable server-side verifications by default.
Aug 20 19:36:22 d6cdd0c71489 haproxy[4451]: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 – – SC– 1/1/0/0/0 0/0 “GET / HTTP/1.1”

These example lines are just for illustration purposes. If we diagnose errors with our HAProxy server, chances are the lines in logs have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.

Regardless of our Linux distribution, the format of the lines in the HAProxy logs will include HTTP status codes, along with requesting IPs and the status of backend servers.

Once we have an idea of what might be causing problems with the HAProxy server we can continue researching and troubleshooting the issue.

The HTTP status code and text description are useful since they give us explicit and specific terms. We can use them to narrow down the range of possible causes of a problem.