Hot Failover using CARP

[Ref: FAQ: PF CARP]

OpenBSD supports a Hot Failover on network failure using carp(4) (Common Access Redundancy Protocol.) The CARP Protocol provides a pseudo-device to share a single IP between multiple hosts.

The FAQ entry, and manual page is very descriptive. If you don’t understand that, these notes just may confuse you further. Hopefully these notes can assist you, but always refer back to the official documentation.

Table of Contents

  • Sample Configuration
  • Configuring CARP
    • External NICs
    • Internal NICs
    • CARP Pseudo Interface
  • Configuring PF
    • Allow CARP
    • PF State Synchronisation
      • PFSync NICs
      • Create the Interface
      • Configure It
  • Failing Over
    • Forced Physical Failure
    • Increase ADVSKEW
    • Increase demote count
  • Related Links

Basic CARP Diagram

KernelTRAP Interview: Ryan McBride

Essentially, CARP provides the ability for one host to assume the network identity of an other, and a mechanism to decide when that’s necessary.

– Ryan McBride (dev lead)

From the manpage carp(4)

carp allows multiple hosts on the same local network to
share a set of IP addresses.  Its primary purpose is to ensure that these
addresses are always available, but in some configurations carp can also
provide load balancing functionality.

Applications (or services) that can share their state data between separate hosts can now use multiple hosts to service clients over a single IP address.

Data synchronisation is not part of the CARP protocol. Synchronisation can occur over the same ‘wire’ as the CARP protocol, and in our practise we use a dedicated cross-over cable for configurations requiring two hosts. A dedicated switch can also be used for services requiring 3+ hosts.

The OpenBSD base installation provides a state, data synchronisation toolset for:

  • PF Packet Filter using pfsync(4) for firewall state synchronisation.
  • IPSec using sasyncd(8) for IPsec SA, SPD synhronization

For more detailed discussions of Redundant Firewalls, and configurations using CARP, refer to Ryan McBride’s Firewall Failover with pfsync and CARP the Related Links below and the [man(ual)] pages.

In practise? We use carp on our firewalls to provide high-availability and:

  1. Augment high-end servers with commodity desktops
  2. Replace expensive equipment (closed/hugely expensive solutions) with (affordable) commodity desktops

Always refer to the [man(ual)] pages and official FAQ: PF CARP.

Sample Configuration

To review using OpenBSD’s CARP solution, we’re going to look at a mythical configuration of an organisation with a redundant gateway separating the external network from the internal network.

Basic CARP Diagram - IP Addresses included

The external router is connected to our Gateways through carp0 which is using IP Address 10.0.0.1.

The internal network is connected to our gateways through carp1 which is using IP Address 192.168.0.1.

There are two parts to configuring hot failover:

  • Configuring CARP
    • Physical Network Interface
    • Pseudo Interface
  • Configuring PF Firewall
    • Allow CARP, PFSYNC
    • PF State Synchronisation

Configuring CARP involves configuring the interfaces on either end of the gateway/firewall (eth0 with carp0, and eth1 with carp1.) The 3rd interface to configure (eth2 with pfsync0) is used for synchronising PF firewall status information.

Basic CARP Diagram - IP Addresses included

Configuring CARP

[Ref: carp(4)]

carp is a pseudo-device.

Before configuring the pseudo-device, we are going to configure the physical devices.

Physical Network Interface

For our example configuration, we have 3 physical network interfaces for each host, but only two will implement carp.

  1. External
  2. Internal

External Interface

[Ref: Correct netmask on carp interfaces]

Basic CARP Diagram - External Link

It is usually sufficient to have an “up” command in the hostname.interface configuration, but where possible: assigning an IP address to the firewall will allow direct connection to the ‘backup’ firewall without needing to jump through the ‘master’ firewall.

Gateway A:/etc/hostname.eth0

inet 10.0.0.11 255.255.255.0 10.0.0.255

Gateway B:/etc/hostname.eth0

inet 10.0.0.12 255.255.255.0 10.0.0.255

Other important items that may be relevant within the Interface configuration are whether you need to specify metrics or media options.

Internal Interface

[Ref: Correct netmask on carp interfaces]

Shared CARP IP through vhid 2: 192.168.100.1

Basic CARP Diagram - Internal Link

Gateway A:/etc/hostname.eth1

inet 192.168.100.11 255.255.255.0 192.168.100.255

Gateway B:/etc/hostname.eth1

inet 192.168.100.12 255.255.255.0 192.168.100.255

CARP Pseudo Interface

To set carp(8) up, we

  • enable receipt of CARP packets,
  • create the interface, and
  • configure it.
NominalPurpose
vhid _Virtual Host ID_ is a unique number used to identify the group to other nodes on the network (1 ~ 255)
pass The password is unique for each group
carpdev Optional paremeter specifying the network interface that belongs to the above group?
advbase Optional parameter specifying how often , in seconds, to advertise that we're a member of the group
advskew Lowest number becomes the master. Failover (with net.inet.carp.preempt=1) forces value to 240
IP Address The unique shared ip address for the group.
balancing _mode_ optional load balancing modes: [ arp | ip | ip-stealth | ip-unicast ]
ifconfig(8), carp(4), FAQ - Firewall Redundancy with CARP and pfsync.

enable CARP packets

sudo sysctl -w net.inet.carp.allow=1
sudo sysctl -w net.inet.carp.preempt=1
sudo sysctl -w net.inet.carp.log=1

To make sure this is set between system restarts.

File: /etc/sysctl.conf

net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=1
NominalPurpose
net.inet.carp.allow accept incoming CARP packets or not. Default is yes, and not in /etc/sysctl.conf
net.inet.carp.preempt Allow hosts within group to preempt the master. Sets failover of all CARP interfaces on the failure of one interface. Disabled by default.
net.inet.carp.log Log bad CARP packets.
net.inet.carp.arpbalance Load balance traffic across group hosts. Default is disabled.

Remember, we’ve allowed the carp packets into the system, but when we turn on PF, we normally block everything by default and need to explicitly allow carp packets in.

create the interface

sudo ifconfig carp0 create

To make the change permanent, create an /etc/hostname.carp0 file with the appropriate details.

configure the interface

[Ref: Correct netmask on carp interfaces]

A simple configuration is such as the below from the command-line

sudo ifconfig carp0 vhid 1 pass secretpassword carpdev eth0 advskew 100 10.0.0.1

To make sure this is set between system restarts.

File: /etc/hostname.carp0

inet 10.0.0.1 255.255.255.0 vhid1 pass secretpassword carpdev eth0 advskew 100

Note: from Correct netmask on carp interfaces, if the carpdev will not have an ip-address in the same subnet, then we want to change the netmask on the carp interface to /32 (instead of the sample /24 shown above)

Basic CARP Diagram - External Link

Shared CARP:

  • vhid: 1
  • IP Address: 10.0.0.1

Gateway A:/etc/hostname.carp0 (text should be a single line, it is formatted below for legibility)

inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 
    pass secretpassword carpdev eth0 advskew 50 state 
    MASTER description "OUTSIDE"

Gateway B:/etc/hostname.carp0 (text should be a single line, it is formatted below for legibility)

inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 
    pass secretpassword carpdev bge0 advskew 100 state 
    BACKUP description "OUTSIDE

Basic CARP Diagram - Internal Link

Shared CARP:

  • vhid: 2
  • IP Address: 192.168.100.1

Note: from Correct netmask on carp interfaces, if the carpdev will not have an ip-address in the same subnet, then we want to change the netmask on the carp interface to /32 (instead of the sample /24 shown above)

Gateway A:/etc/hostname.carp1 (text should be a single line, it is formatted below for legibility)

inet 192.168.100.1 255.255.255.0 192.168.100.255 vhid 2 
    pass anothersecretpassword carpdev eth1 advskew 50 state 
    MASTER description "INSIDE"

Gateway B:/etc/hostname.carp1 (text should be a single line, it is formatted below for legibility)

inet 192.168.100.1 255.255.255.0 192.168.100.255 vhid 2 
    pass anothersecretpassword carpdev eth1 advskew 100 state 
    BACKUP description "INSIDE"

Configuring PF Firewall

To effectively use carp, we have configured the interfaces above, but we also need to specify in the Packet Filter:

  1. allow carp protocol in/out of firewalls
  2. allow synchronisation between the firewalls
  3. synchronise PF state between redundant hosts
    • Physical Interface
    • Pseudo-device

Allow CARP/PFSYNC

The PF ruleset to allow carp and synchronisation is as simple as the below:

pass on { eth0 eth1 } proto carp keep state (no-sync)
pass quick on eth2 proto pfsync keep state (no-sync)

pfsync is the OpenBSD protocol used for synchronising Packet Filter state.

Synchronise PF State

[Ref: pfsync(4)]

Remember, CARP only allows for the high availability of the IP Address, it still the responsibility of your application/service to synchronise required data between the carp hosts to ensure that failover is seemless.

PF supports synchronising the state table between hosts using pfsync(4).

The pfsync interface is a pseudo-device which exposes certain changes to
the state table used by pf(4).  ...

If configured with a physical
synchronisation interface, pfsync will also send state changes out on
that interface, and insert state changes received on that interface from
other systems into the state table.
...
pfsync and carp(4) can be used together to provide automatic failover of
a pair of firewalls configured in parallel.  One firewall will handle all
traffic until it dies, is shut down, or is manually demoted, at which
point the second firewall will take over automatically.

In our example, we use the third network interface.

Basic CARP Diagram - PF Sync Interface

We need to configure:

  1. Physical Interface
  2. Pseudo Interface

Physical Interface

We configure the physical interfaces with IP Addresses relevant to how we want/need to manage services. The below chosen IP Addresses are arbitrary:

Gateway A:/etc/hostname.eth2

inet 172.16.1.1 255.255.255.0 172.16.1.255

Gateway B:/etc/hostname.eth2

inet 172.16.1.2 255.255.255.0 172.16.1.255

Pseudo Interface

pfsync0 is the first pseudo-device to use, create the interface using ifconfig ‘create’.

sudo ifconfig pfsync0 create

Configure the pfsync interface as your normally do with other network interfaces (in /etc/hostname.pfsync0) with additional commands relevant to pf synchronisation.

sudo ifconfig pfsync0 syncdev eth2

From our current sample configuration:

NominalPurpose
pfsyncN The name of the pfsync(4) interface used to send pfsync updates out
syncdev The name of hte physical interface used to send pfsync updates out.
syncpeer Optional IP address of a host to exchange pfsync updates with. By default, pfsync updates multicast on the local network. This option changes that behaviour to unicast the update to the specified syncpeer.

To make sure this is set between system restarts.

Gateway A:/etc/hostname.pfsync0

up syncdev eth2 syncpeer 172.16.1.2

Gateway B:/etc/hostname.pfsync0

up syncdev eth2 syncpeer 172.16.1.1

The use of syncpeer is relevant in a two host CARP configuration, and especially when the synchronisation wire is shared with other devices (i.e. insecure.)

Failing Over

The carp system monitors at least three mechanisms to determine whether the CARP interface should be in the MASTER or BACKUP state.

Exploring these, is a good way of gaining better understanding of how and why your hosts will fail between each other.

  • Forced Physical Failure
  • Advertising SKEW
  • DEMOTE Counter

Forced Physical Failure

[Ref: carp, ifconfig]

Disabling a CARP Interface can be achieved through physically disconnecting the interface, or by using the ifconfig utility to disable the interface.

ifconfig carp0 down

Increase ADVSKEW

[Ref: carp, ifconfig]

Another method for failing an carp interface, is to change the frequency of carp advertisements (the most frequent gaining MASTER status.) Increasing the time between carp advertisements allows another CARP host to take over MASTER status.

Use something like:

ifconfig carp0 advskew 254

Obviously, if you want to return the node back to MASTER, then you can reset the advertising skew advskew to a number lower than the advskew on the other server.

Advantage: Interface is still functional.

Increase demote count

[Ref: From ifconfig]

ifconfig -g group-name [[-]carpdemote [number]]

-g group-name
		Specify the group.
		
carpdemote [number]
		Increase carp(4) demote count for given interface group by
		number. If number is omitted, it is increased by 1.

-carpdemote [number]
		Decrease carp(4) demote count for given interface group by
		number. If number is omitted, it is decreased by 1.

Demote works on carp groups, so to sample our carp0 interface we need it in a group, with something like the below:

ifconfig carp0 group test

From which we can now confirm the carp demote count using

ifconfig -g test

test: carp demote count 0

We can increase the demote count, using the above sample as in:

ifconfig -g test carpdemote 50
ifconfig -g test

test: carp demote count 50

Advantage: Interface is still functional.

Related Links

There are various links, some listed below, with more indepth discussion of the CARP protocol, and underpinnings.