Route Troubleshooting

When diagnosing traffic flow issues, one of the first things to check is the routes known to pfSense.

Viewing Routes

There are two ways to view the routes: Via the WebGUI, and via the command line.

To view the routes in the WebGUI, navigate to Diagnostics > Routes and output is shown similar to Figure Route Display.

../_images/routes.png

Route Display

The output from the command line is similar to that seen in the WebGUI:

# netstat -rWn
Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
default            198.51.100.1       UGS        1822   1500       igb1
10.2.0.0/24        link#2             U             0   1500       igb0
10.2.0.1           link#2             UHS           0  16384        lo0
127.0.0.1          link#11            UH          204  16384        lo0
198.51.100.0/24    link#3             U          1181   1500       igb1
198.51.100.1       00:08:a2:09:95:b6  UHS        2789   1500       igb1
198.51.100.2       link#3             UHS           0  16384        lo0

The columns shown on these screens indicate various properties of the routes, and are explained later in this section.

Destination

This column contains the destination host or network. The default route for the system is simply listed as default. Otherwise, hosts are listed as by IP address, and networks are listed with an IP address and CIDR subnet mask.

Gateway

A gateway is the router through which packets going to a specific destination are sent. If this column shows a link, such as link#1, then that network is directly reachable by that interface and no special routing is necessary. If a host is visible with a MAC address, then it is a locally reachable host with an entry in the ARP table, and packets are sent there directly.

Flags

There are quite a few flags, all of which are covered in the FreeBSD man page for netstat(1), reproduced in Table Route Table Flags and Meanings with some modifications.

Route Table Flags and Meanings
Letter Flag Meaning
1 RTF_PROTO1 Protocol specific routing flag #1
2 RTF_PROTO2 Protocol specific routing flag #2
3 RTF_PROTO3 Protocol specific routing flag #3
B RTF_BLACKHOLE Discard packets during updates
b RTF_BROADCAST Represents a broadcast address
D RTF_DYNAMIC Created dynamically by redirect
G RTF_GATEWAY Destination requires forwarding by intermediary
H RTF_HOST Host entry (net otherwise)
L RTF_LLINFO Valid protocol to link address translation
M RTF_MODIFIED Modified dynamically (by redirect)
R RTF_REJECT Host or net unreachable
S RTF_STATIC Manually added
U RTF_UP Route usable
X RTF_XRESOLVE External daemon translates proto to link address

For example, a route flagged as UGS is a usable route, packets are sent via the gateway listed, and it is a static route.

Refs

This column counts the current number of active uses of a given route.

Use

This counter is the total number of packets sent via this route. This is helpful for determining if a route is actually being used, as it will continually increment as packets utilize the route.

Netif

The network interface used for this route.

Expire

For dynamic entries, this field shows how long until this route expires if it is not used again.

Using traceroute

Traceroute is a useful tool for testing and verifying routes and multi-WAN functionality, among other uses. It shows each “hop” along the path a packet travels from one end to the other, along with the latency encountered in reaching that intermediate point. On pfSense, a traceroute can be performed by navigating to Diagnostics > Traceroute, or by using traceroute at the command line. From clients running Windows, the program is available under the name tracert.

Every IP packet contains a time-to-live (TTL) value. When a router passes a packet, it decrements the TTL by one. When a router receives a packet with a TTL of 1 and the destination is not a locally attached network, the router returns an ICMP error message “Time-to-live exceeded” and drops the packet. This is to limit the impact of routing loops, which otherwise would cause each packet to loop indefinitely.

Traceroute uses this TTL to its advantage to map the path to a specific network destination. It starts by sending the first packet with a TTL of 1. The first router (usually the default gateway) will send back an ICMP time-to-live exceeded error. The time between sending the packet and receiving the ICMP error is the time displayed, listed along with the IP address that sent the error and its reverse DNS, if any. After sending three packets with a TTL of 1 and displaying their response times, it will increment the TTL to 2 and send three more packets, noting the same information for the second hop. Traceroute increments the TTL and repeats the process until it reaches the specified destination, or exceeds the maximum number of hops.

Traceroute functions slightly differently on Windows and Unix-like operating systems (BSD, Linux, Mac OS X, Unix, etc.). Windows uses ICMP echo request packets (pings) while Unix-like systems use UDP packets by default. ICMP and UDP are layer 4 protocols, and traceroute is done at layer 3, so the protocol used is largely irrelevant except when considering a policy routing configuration. Traceroute from Windows clients will be policy routed based on which rule permits ICMP echo requests, while Unix-like clients will be routed by the rule matching the UDP ports in use.

In this example, traceroute is used to view the route to www.google.com:

# traceroute www.google.com
traceroute: Warning: www.google.com has multiple addresses; using 74.125.95.99
traceroute to www.l.google.com (74.125.95.99), 64 hops max, 40 byte packets
 1  core (172.17.23.1)  1.450 ms  1.901 ms  2.213 ms
 2  172.17.25.21 (172.17.25.21)  4.852 ms  3.698 ms  3.120 ms
 3  bb1-g4-0-2.ipltin.ameritech.net (151.164.42.156)  3.275 ms  3.210 ms  3.215 ms
 4  151.164.93.49 (151.164.93.49)  8.791 ms  8.593 ms  8.891 ms
 5  74.125.48.117 (74.125.48.117)  8.460 ms  39.941 ms  8.551 ms
 6  209.85.254.120 (209.85.254.120)  10.376 ms  8.904 ms  8.765 ms
 7  209.85.241.22 (209.85.241.22)  19.479 ms  20.058 ms  19.550 ms
 8  209.85.241.29 (209.85.241.29)  20.547 ms  19.761 ms
    209.85.241.27 (209.85.241.27)  20.131 ms
 9  209.85.240.49 (209.85.240.49)  30.184 ms
    72.14.239.189 (72.14.239.189)  21.337 ms  21.756 ms
10  iw-in-f99.google.com (74.125.95.99)  19.793 ms  19.665 ms  20.603 ms

The output shows that it took 10 hops to get there, and the latency generally increased with each hop, which is expected.

Note

When utilizing policy routing, such as with Multi-WAN, the firewall itself may not appear as a hop in traceroute. When policy routing is employed, pf does not decrement the TTL when forwarding packets, so traceroute cannot detect it as an intermediate router.

Routes and VPNs

Depending on the VPN being used, a route may not display in the table for the far side. IPsec does not use the routing table, it is instead handled internally in the kernel using IPsec security policy database (SPD) entries. Static routes will never cause traffic to be directed across an IPsec connection. OpenVPN uses the system routing table and as such entries are present for networks reachable via an OpenVPN tunnel, as in the following example:

#netstat -rWn
Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
default            198.51.100.1       UGS       92421   1500        em0
10.6.0.0/16        10.6.203.1         UGS           0   1500     ovpnc2
10.6.203.0/24      10.6.203.2         UGS           0   1500     ovpnc2
10.6.203.1         link#9             UH            0   1500     ovpnc2
10.6.203.2         link#9             UHS           0  16384        lo0
10.7.0.0/24        link#2             U       1260771   1500        em1
10.7.0.1           link#2             UHS           0  16384        lo0
127.0.0.1          link#7             UH          866  16384        lo0
198.51.100.0/24    link#1             U       1251477   1500        em0
198.51.100.7       link#1             UHS           0  16384        lo0

The OpenVPN interface is 10.6.203.2, with a gateway of 10.6.203.1 and the interface is ovpnc2. The network reachable using OpenVPN in this example is 10.6.0.0/16.

With IPsec, traceroute is not as useful as with routed setups like OpenVPN, because the IPsec tunnel itself does not have IP addresses. When running traceroute to a destination across IPsec, a timeout will be shown for the hop that is the IPsec tunnel.