[Ref: OpenBSD 5.7, 5.8, HAProxy 1.5.14 Configuration Manual ]
Why another document on configuring HAProxy? Because like so many, I also scan the Internet for HOWTOs instead of reading the project documentation.
I had a particular goal in mind when I first installed HaProxy, and since I’d got most of it working using something else Enginx how difficult could it be to rehash it in HaProxy? Why do I need to read documentation, why can’t i just get a recipe and have it up and running in minutes instead of days?
Table of Contents
It really isn’t that difficult to go from
After many missteps I find that new deployments work better following simple steps like the above. Get the basic environment up and running to make sure all the bits and pieces are actually working as expected before debugging the more difficult ‘beautiful’ design I had in mind 8-)
The Project documentation has a nice Section 2 configuring HAProxy
2.0. Configuring HAProxy 2.1. Configuration file format 2.2. Time format 2.3. Examples
It didn’t make sense to me when I first read it, but I know you could do better (then you can come back here to be confused.)
HAProxy's configuration process involves 3 major sources of parameters :
- the arguments from the command-line, which always take precedence
- the "global" section, which sets process-wide parameters
- the proxies sections which can take form of "defaults", "listen",
"frontend" and "backend".
Of course, after having a working system the documentation makes a whole lot more sense?
If you’re trying HAProxy for the first time, I recommend you go through at least the examples in the above reference.
The basic installation for OpenBSD is through the ports/package system.
Once up and installed, and we’re ready to test our installation the process I generally use re-emphasises the first configuration process:
HAProxy's configuration process involves ... : - the arguments from the command-line, which always take precedence
$ sudo haproxy -c -f /etc/haproxy/haproxy.conf$ sudo haproxy -d -f /etc/haproxy/haproxy.confThe above spits out a lot of debug information useful for visualising what the proxy is doing.
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
....
A simple adaptation of the first example from the documentation highlights how simple (flexible?) HAProxy can be.
TO help you familiarise with the syntax, and using the manual we’ll look at how/what the directives in the sample configuration.
File: /etc/haproxy/haproxy.conf
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen http-in
bind *:80
server websvr1 192.168.0.5:80 maxconn 32
server websvr2 192.168.0.6:80 maxconn 32Presuming that 192.168.0.5 and 192.168.0.6 are web servers with a simple web page “This is HTTP Server websvrX”, then you could expect to have a result such as:
$ sudo haproxy -c -f /etc/haproxy/haproxy.conf
$ sudo haproxy -f /etc/haproxy/haproxy.conf
$ while true; do curl http://localhost; sleep 1; doneThis is HTTP server websvr1 (192.168.0.5). This is HTTP server websvr2 (192.168.0.6). This is HTTP server websvr1 (192.168.0.5). This is HTTP server websvr2 (192.168.0.6). ...
The general settings are configured in the (line 1) “global” and (line 5) “defaults” section, and the real work is configured in the (line 11)“listen” section labelled “http-in”
The ‘global’ parameters are categorised into three separate groups.
For Process Management and Security we specify daemon
daemon Makes the process fork into background. This is the recommended mode of operation. It is equivalent to the command line "-D" argument. It can be disabled by the command line "-db" argument.
For Performance tuning we set the maxconn rate
maxconn <number> Sets the maximum per-process number of concurrent connections to <number>. It is equivalent to the command-line argument "-n". Proxies will stop accepting connections when this limit is reached. The "ulimit-n" parameter is automatically adjusted according to this value. See also "ulimit-n". Note: the "select" poller cannot reliably use more than 1024 file descriptors on some platforms. If your platform only supports select and reports "select FAILED" on startup, you need to reduce maxconn until it works (slightly below 500 in general).
There is no debugging specified in the configuration file, and we’ve shown above and below that we can specify this on the command-line.
(you can look those up at: debug or quiet)
The Proxy configuration can be located in a set of sections :
Most examples you’ll find involve the ‘frontend’ and ‘backend’, but in our simplified example, the required services are provided in a ’listen’ section.
Right now, two major proxy modes are supported : "tcp", also known as layer 4, and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the protocol, and can interact with it by allowing, blocking, switching, adding, modifying, or removing arbitrary contents in requests or responses, based on arbitrary criteria.
For this example’s defaults, we are going to be dealing with http (layer 7) traffic, with some preset timeouts related to http traffic.
A "listen" section defines a complete proxy with its frontend and backend parts combined in one section. It is generally useful for TCP-only traffic.
bind [<address>]:<port_range> [, ...] [param*] bind /<path> [, ...] [param*] Define one or several listening addresses and/or ports in a frontend.
HAProxy will listen (bind) on all interfaces (".") at port 80. All traffic for that port will be load balanced to two web servers:
Below we show how we can start the HaProxy server to review the above configuration.
Two ports/packages are important for using HAProxy on OpenBSD are:
Socat isn’t really required on OpenBSD, as Netcat nc(1) is in ‘base’ and supports ‘-U’ UNIX-domain socket connections. But you will find the manual and majority of online references using socat.
The basic command-line option is to parse/check the configuration file.
$ sudo haproxy -c -f /etc/haproxy/haproxy.confCommand-line options:
-d Start in foreground with debugging mode enabled. When the proxy
runs in this mode, it dumps every connections, disconnections,
timestamps, and HTTP headers to stdout. ...
$ sudo haproxy -d -f /etc/haproxy/haproxy.conf
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
[WARNING] ..
[ALERT] .. Starting frontend absdefg: cannot bind socket [0.0.0.0:80]
[ALERT] .. Starting frontend abcdefghij: cannot bind socket [0.0.0.0:443]
The errors are well categorised, [ALERTS] highlight sections that will definitely cause problems with your installation (such as in the above example, it is not going to work as you expect.) [WARNING]s are likely to cause problems and you should probably investigate and make a determination whether you need to fix or it can be safely ignored.
[Ref: HAProxy hot-reconfiguration, HAProxy reloading your config with minimal service impact]
HAProxy supports ‘soft-reconfiguration,’ a mechanism for pausing or disabling haproxy without effecting existing connections.
-sfSend FINISH signal to the pids in pidlist after startup. The processes which receive the signal will wait for all sessions to finish before exiting. This option must be specified last, followed by any number of PIDs. Technically speaking, SIGTTOU and SIGUSR1 are sent. -st Send TERMINATE signal to pids in pidlist after startup. The processes which receive this signal will wait immediately terminate, closing all active sessions. This option must be specified last, followed by any number of PIDs. Technically speaking, SIGTTOU and SIGTERM are sent.
$ sudo haproxy -f /etc/haproxy/haproxy.conf -sf$ sudo haproxy -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.conf)[Ref: Unix Socket commands, HAProxy stats from the command line]
The command-line, live query is based on HAProxy’s ‘stats’ features which are configured using something like the below (more details later)
[ Ref: stats socket]
stats socket [<address:port>|<path>] [param*]
Binds a UNIX socket to <path> or a TCPv4/v6 address to <address:port>.
Connections to this socket will return various statistics outputs and even
allow some commands to be issued to change some runtime settings. Please
consult section 9.2 "Unix Socket commands" for more details.
All parameters supported by "bind" lines are supported, for instance to
restrict access to some users or their access rights. Please consult
section 5.1 for more information.
stats timeout <timeout, in milliseconds>
The default timeout on the stats socket is set to 10 seconds. It is possible
to change this value with "stats timeout". The value must be passed in
milliseconds, or be suffixed by a time unit among { us, ms, s, m, h, d }.
To get a list of available options, send a ‘blank’ to the socket, such as:
$ echo "" | sudo nc -U /var/run/haproxy/admin.sockUnknown command. Please enter one of the following commands only : clear counters : clear max statistics counters (add 'all' for all counters) clear table : remove an entry from a table help : this message prompt : toggle interactive mode with prompt quit : disconnect show info : report information about the running process show pools : report information about the memory pools usage show stat : report counters for each proxy and server show errors : report last request and response errors for each proxy show sess [id] : report the list of current sessions or dump this session show table [id]: report table usage stats or dump this table's contents get weight : report a server's current weight set weight : change a server's weight set server : change a server's state or weight set table [id] : update or create a table entry's data set timeout : change a timeout setting set maxconn : change a maxconn setting set rate-limit : change a rate limiting value disable : put a server or frontend in maintenance mode enable : re-enable a server or frontend which is in maintenance mode shutdown : kill a session or a frontend (eg:to release listening ports) show acl [id] : report avalaible acls or dump an acl's contents get acl : reports the patterns matching a sample for an ACL add acl : add acl entry del acl : delete acl entry clear acl: clear the content of this acl show map [id] : report avalaible maps or dump a map's contents get map : reports the keys and values matching a sample for a map set map : modify map entry add map : add map entry del map : delete map entry clear map : clear the content of this map set ssl : set statement for ssl
$ echo "show info" | sudo nc -U /var/run/haproxy/admin.sockName: HAProxy Version: 1.5.14 Release_date: ... ...
Using ‘socat’ is similar
$ echo "show errors" | sudo socat /var/run/haproxy/admin.sock[Ref: Global parameters]
Parameters in the "global" section are process-wide and often OS-specific. They are generally set once for all and do not need being changed once correct. Some of them have command-line equivalents.
As mentioned earlier, HAProxy’s global section can be logically grouped into three groups:
We have covered some of the configuration settings earlier, and the more interesting parts from our production environment are the “log” specifications, and the “statistics” directives.
[Ref: log]
When we enable logging using log <address> <facility> [max level] we are sending startup, exit notifications.
log <address> [len <length>] <facility> [max level [min level]] Adds a global syslog server. Up to two global servers can be defined. They will receive logs for startups and exits, as well as all logs from proxies configured with "log global".
By default, configuring a log address sends all events to the log host. We can separate log entries from HAProxy with a configuration such as in the above.
As shown in the earliest example, you can specify the statistical view in the Global section. There are options, one described further below.
[Ref: Proxies]
Right now, two major proxy modes are supported : "tcp", also known as layer 4, and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the protocol, and can interact with it by allowing, blocking, switching, adding, modifying, or removing arbitrary contents in requests or responses, based on arbitrary criteria.
All proxy names must be formed from upper and lower case letters, digits, '-' (dash), '_' (underscore) , '.' (dot) and ':' (colon). ACL names are case-sensitive, which means that "www" and "WWW" are two different proxies.
[Ref: Proxies]
A "defaults" section sets default parameters for all other sections following its declaration. Those default parameters are reset by the next "defaults" section. See below for the list of parameters which can be set in a "defaults" section. The name is optional but its use is encouraged for better readability.
[Ref: Proxies]
A "frontend" section describes a set of listening sockets accepting client connections.
[Ref: Proxies]
A "backend" section describes a set of servers to which the proxy will connect to forward incoming connections.
[Ref: Proxies]
A "listen" section defines a complete proxy with its frontend and backend parts combined in one section. It is generally useful for TCP-only traffic.
The example ’listen’ section we have, highlights the general feature, but is better described below.
[Ref: Proxies]
[Ref: Statistics and monitoring]
It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket.
[Ref: Logging
[Ref: syslog.conf(5)]
You should have something like the below in your configuration file:
File extract: /etc/syslog.conf
The format is explained in the manpage, with some of the relevant extracts below:
Line 1: Blocks of rules(?) are separated with a tag line beginning with an exclamation mark.
manpage extracts: syslog.conf(5)
Each block of lines is separated from the previous block by a tag. The tag is a line beginning with !prog and each block will be associated with calls to syslog from that specific program. When a message matches multiple blocks, the action of each matching block is taken. If no tag is specified at the beginning of the file, every line is checked for a match and acted upon (at least until a tag is found).
The selectors are encoded as a facility, a period ('.'), and a
level, with no intervening whitespace. Both the facility and the
level are case insensitive.
The action field is /var/log/haproxy/info.log and /var/log/haproxy.notice.log
The action field of each line specifies the action to be taken when the selector field selects a message. There are six forms: • A pathname (beginning with a leading slash). Selected messages are appended to the file.
Note: the ‘air-gap’ between the ‘selector’ and ‘action’ are separated by ’tabs’ not invisible blank spaces.
You must use ’tabs’ between the ‘selector’ field and the ‘action’ field. You can guarantee yourself a disfunctioning logging environmnet if you ignore this and blindly presume the ‘whitespaces’ are correct (even if it is correct for /etc/newsyslog.conf)
manpage extract: syslog.conf(5)
The selector field is separated from the action field by one or more tab characters
The simplest way to view whether you used ’tabs’ or spaces, is to open the file in ‘vim’ and expose tabs using the command “:set list”