Example Redundant Configuration

This section describes a simple three interface HA configuration. The three interfaces are LAN, WAN, and Sync. This is functionally equivalent to a two interface LAN and WAN deployment, with the pfsync interface being used solely to synchronize configuration and firewall states between the primary and secondary firewalls.

Note

This example only covers an IPv4 configuration. High Availability is compatible with IPv6, but it requires static addressing on the firewall interfaces. When preparing to configure HA, if static IPv6 assignments are not available, set IPv6 to None on all interfaces.

Determine IP Address Assignments

The first task is to plan IP address assignments. A good strategy is to use the lowest usable IP address in the subnet as the CARP VIP, the next subsequent IP address as the primary firewall interface IP address, and the next IP address as the secondary firewall interface IP address. This design is optional, any scheme may be used, but we strongly recommend a consistent and logical scheme to make design and administration simpler.

WAN Addressing

The WAN addresses will be selected from those assigned by the ISP. For the example in Table WAN IP Address Assignments, the WAN of the HA pair is 198.51.100.0/24, and the addresses 198.51.100.200 through 198.51.100.202 will be used as the WAN IP addresses.

WAN IP Address Assignments
IP Address Usage
198.51.100.200/24 CARP shared IP address
198.51.100.201/24 Primary node WAN IP address
198.51.100.202/24 Secondary node WAN IP address

LAN Addressing

The LAN subnet is 192.168.1.0/24. For this example, the LAN IP addresses will be assigned as shown in Table LAN IP Address Assignments.

LAN IP Address Assignments
IP Address Usage
192.168.1.1/24 CARP shared IP address
192.168.1.2/24 Primary node LAN IP address
192.168.1.3/24 Secondary node LAN IP address

Sync Interface Addressing

There is no shared CARP VIP on this interface because there is no need for one. These IP addresses are used only for communication between the firewalls. For this example, 172.16.1.0/24 is used as the Sync subnet. Only two IP addresses will be used, but a /24 is used to be consistent with the other internal interface (LAN). For the last octet of the IP addresses, use the same last octet as that firewall’s LAN IP address for consistency.

Sync IP Address Assignments
IP Address Usage
172.16.1.2/24 Primary node Sync IP address
172.16.1.3/24 Secondary node Sync IP address

Figure Example HA Network Diagram shows the layout of this example HA pair. The primary and secondary each have identical connections to the WAN and LAN, and a crossover cable between them to connect the Sync interfaces. In this basic example, the WAN switch and LAN switch are still potential single points of failure. Switching redundancy is covered later in this chapter in Layer 2 Redundancy.

../_images/diagrams-example-carp.png

Example HA Network Diagram

Cluster Configuration Basics

Each node requires some basic configuration outside of the actual HA setup. Do not connect both nodes into the same LAN before both nodes have a non- conflicting LAN setup.

Installation, interface assignment and basic configuration

Install the OS on the firewalls as usual and assign the interfaces identically on both nodes. Interfaces must be assigned in the same order on all nodes exactly. If the interfaces are not aligned, configuration synchronization and other tasks will not behave correctly. If any adjustments have been made to the interface assignments, they must be replicated identically on both nodes.

Then, connect to the GUI and use the Setup Wizard to configure each firewall with a unique hostname and non-conflicting static IP addresses. Refer back to Setup Wizard if needed.

For example, one node could be “firewall-a.example.com” and the other “firewall- b.example.com”, or a more personalized pair of names.

Note

Avoid naming the nodes “master” or “backup” since those are states and not roles, instead they could be named “primary” and “secondary”.

The default LAN IP address is 192.168.1.1. Each node must be moved to its own address, such as 192.168.1.2 for the primary and 192.168.1.3 for the secondary. This layout is shown in LAN IP Address Assignments. Once each node has a unique LAN IP address, then both nodes may be plugged into the same LAN switch.

Setup Sync Interface

Before proceeding, the Sync interfaces on the cluster nodes must be configured. Sync IP Address Assignments lists the addresses to use for the Sync interfaces on each node. Once that has been completed on the primary node, perform it again on the secondary node with the appropriate IPv4 address value.

To complete the Sync interface configuration, firewall rules must be added to both nodes to allow synchronization.

At a minimum, the firewall rules must pass the configuration synchronization traffic (by default, HTTPS on port 443) and pfsync traffic. In most cases, a simple “allow all” style rule is used.

When complete, the rules will look like the example in figure Example Sync Interface Firewall Rules, which also includes a rule to allow ICMP echo (ping) for diagnostic purposes.

../_images/ha-sync-rules.png

Example Sync Interface Firewall Rules

The secondary does not need those rules initially, only a rule to allow traffic to the GUI for XML-RPC to function. The full set of rules will synchronize once XML-RPC has been configured.

Configure pfsync

State synchronization using pfsync must be configured on both the primary and secondary nodes to function.

First on the primary node and then on the secondary, perform the following:

  • Navigate to System > High Avail Sync
  • Check Synchronize States
  • Set Synchronize Interface to SYNC
  • Set pfsync Synchronize Peer IP to the other node. Set this to 172.16.1.3 when configuring the primary node, or 172.16.1.2 when configuring the secondary node
  • Click Save

Configure Configuration Synchronization (XML-RPC)

Warning

Configuration synchronization must only be configured on the primary node. Never activate options in this section on the secondary node of a two-member cluster.

On the primary node only, perform the following:

  • Navigate to System > High Avail Sync

  • Set Synchronize Config to IP to the Sync interface IP address on the secondary node, 172.16.1.3

  • Set Remote System Username to admin.

    Note

    This must always be admin, no other user will work!

  • Set Remote System Password to the admin user account password, and repeat the value in the confirmation box.

  • Check the boxes for each area to synchronize to the secondary node. For this guide, as with most configurations, all boxes are checked. The Toggle All button may be used to select all of the options at once, rather than selecting them individually.

  • Click Save

As a quick confirmation that the synchronization worked, on the secondary node navigate to Firewall > Rules on the SYNC tab. The rules entered on the primary are now there, and the temporary rule is gone.

The two nodes are now linked for configuration synchronization! Changes made to the primary node in supported areas will be synchronized to the secondary whenever a change is made.

Warning

Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization.

Configuring the CARP Virtual IPs

With configuration synchronization in place, the CARP Virtual IP addresses need only be added to the primary node and they will be automatically copied to the secondary.

  • Navigate to Firewall > Virtual IPs on the primary node to manage CARP VIPs

  • Click fa-plus Add at the top of the list to create a new VIP.

    Note

    A VIP must be added for each interface handling user traffic, in this case WAN and LAN.

Type:

Defines the type of VIP, in this case CARP.

Interface:

Defines the interface upon which the VIP will reside, such as WAN

Address(es):

The Address box is where the IP address values are entered for the VIP. A subnet mask must also be selected and it must match the subnet mask on the interface IP address. For this example, enter 198.51.100.200 and 24 (See WAN IP Address Assignments).

Virtual IP Password:
 

Sets the password for the CARP VIP. This need only match between the two nodes, which will be handled by synchronization. The password and confirm password box must both be filled in and they must match.

VHID Group:

Defines the ID for the CARP VIP A common tactic is to make the VHID match the last octet of the IP address, so in this case choose 200

Advertising Frequency:
 

determines how often CARP heartbeats are sent.

Base:Controls how many whole seconds elapse between Heartbeats, typically 1. This should match between cluster nodes.
Skew:Controls fractions of a second (1/256th increments). A primary node is typically set to 0 or 1, secondary nodes will be 100 or higher. This adjustment is handled automatically by XML-RPC synchronization.
Description:

Some text to identify the VIP, such as WAN CARP VIP.

Note

If CARP appears to be too sensitive to latency on a given network, adjusting the Base by adding one second at a time is recommended until stability is achieved.

The above description used the WAN VIP as an example. The LAN VIP would be configured similarly except it will be on the LAN interface and the address will be 192.168.1.1 (See LAN IP Address Assignments).

If there are any additional IP addresses in the WAN subnet that will be used for purposes such as 1:1 NAT, port forwards, VPNs, etc, they may be added now as well.

Click Apply Changes after making any edits to the VIPs.

After adding VIPs, check Firewall > Virtual IPs on the secondary node to ensure that the VIPs synchronized as expected.

The Virtual IP addresses on both nodes will look like CARP Virtual IP Address List if the process was successful.

../_images/ha-carp-vips.png

CARP Virtual IP Address List

Configure Outbound NAT for CARP

The next step will be to configure NAT so that clients on the LAN will use the shared WAN IP as the address.

  • Navigate to Firewall > NAT, Outbound tab
  • Click to select Manual Outbound NAT rule generation
  • Click Save

A set of rules will appear that are the equivalent rules to those in place for Automatic Outbound NAT. Adjust the rules for internal subnet sources to work with the CARP IP address instead.

  • Click fa-pencil to the right of the rule to edit
  • Locate the Translation section of the page
  • Select the WAN CARP VIP address from the Address drop-down
  • Change the Description to mention that this rule will NAT LAN to the WAN CARP VIP address

Warning

If additional local interfaces are added later, such as a second LAN, DMZ, etc, and that interface uses private IP addresses, then additional manual outbound NAT rules must be added at that time.

When complete, the rule changes will look like those found in Outbound NAT Rules for LAN with CARP VIP

../_images/ha-out-nat.png

Outbound NAT Rules for LAN with CARP VIP

Modifying the DHCP Server

The DHCP server daemons on the cluster nodes need adjustments so that they can work together. The changes will synchronize from the primary to the secondary, so as with the VIPs and Outbound NAT, these changes need only be made on the primary node.

  • Navigate to Services > DHCP Server, LAN* tab.
  • Set the DNS Server to the LAN CARP VIP, here 192.168.1.1
  • Set the Gateway to the LAN CARP VIP, here 192.168.1.1
  • Set the Failover Peer IP to the actual LAN IP address of the secondary node, here 192.168.1.3
  • Click Save

Setting the DNS Server and Gateway to a CARP VIP ensures that the local clients are talking to the failover address and not directly to either node. This way if the primary fails, the local clients will continue talking to the secondary node.

The Failover Peer IP allows the daemon to communicate with the peer directly in this subnet to exchange data such as lease information. When the settings synchronize to the secondary, this value is adjusted automatically so the secondary points back to the primary.