Using tcpdump from the command line

The tcpdump program is a command line packet capture utility provided with most UNIX and UNIX-like operating system distributions, including FreeBSD. It is also included with pfSense, and usable from a shell on the console or over SSH. It is an exceptionally powerful tool, but that also makes it daunting to the uninitiated user. The tcpdump binary in FreeBSD 10.3 supports 50 different command line flags, limitless possibilities with filter expressions, and its man page, providing only a brief overview of all its options, is nearly 1200 lines long and 67k. After learning to use it, knowledge of how to interpret the data it provides is also necessary, which can require an in-depth understanding of networking protocols.

A comprehensive review of packet capturing and interpretation of the results is outside the scope of this book. Indeed, entire books have been written on this subject alone. For those with a thirst for more than basic knowledge in this area, some recommendations for additional reading are provided at the end of this chapter. This section is intended to provide an introduction to this topic, and leave the reader with enough knowledge for basic troubleshooting.

tcpdump command line flags

The following table shows the most commonly used command line flags with tcpdump. Each option will be discussed in further detail in this section.

Commonly Used tcpdump Flags
Flag Description
-i <interface> Listen on <interface>, .e.g. “-i igb0”
-n Do not perform reverse DNS resolution on IP addresses
-w <filename> Save capture in pcap format to <filename>, e.g. “-w /tmp/wan.pcap”
-s Snap length: Amount of data to be captured from each frame
-c <packets> Exit after receiving a specific number of packets
-p Do not put the interface in promiscuous mode
-v Verbose output
-e Print link-layer header on each line

-i flag

The -i flag specifies the interface on which tcpdump will listen. Use FreeBSD interface names here, such as igb0, em0, vmx0, etc.

-n flag

Do not resolve IP addresses using reverse DNS. When this option is not specified, tcpdump will perform a reverse DNS (PTR) lookup for each IP address. This generates a significant amount of DNS traffic in captures displaying large volumes of traffic. Disable this to avoid adding load to DNS servers. We recommend always using -n because it eliminates the delay between a packet’s capture and its display that is caused by performing the reverse lookup. Also IP addresses tend to be easier to read and understand than their PTR records. That is a matter of personal preference though, and in familiar environments where the PTR records are known to provide the actual host names of the devices, captures may be run without -n to show the hostnames.

Another reason to use -n, is to be “sneaky.” One means of detecting packet capturing is looking for spikes and patterns in DNS PTR lookups. Skipping the DNS lookup will not cause any extra traffic to be generated in the process.

-w flag

tcpdump allows capture files to be saved in pcap format for later analysis or analysis on another system. This is commonly done from command line only devices like pfSense so the file can be copied to a host running Wireshark or another graphical network protocol analyzer and reviewed there. When saving to a file using -w, the frames will not be displayed in the terminal as they otherwise are.

See also

See Using Wireshark with pfSense for more information about using Wireshark with pfSense.

-s flag

By default tcpdump will only save the first 64 bytes of each frame when capturing to a file. This is enough to get the IP and protocol header for most protocols, but limits the usability of capture files. By using the -s flag, tcpdump can be told how much of the frame to capture, in bytes. This is called the snap length.

Example Uses of tcdump -s
Flag Description
-s 500 Capture the first 500 bytes of each frame
-s 0 Capture each frame in its entirety

In most cases, using -s 0 is the best practice when capturing to a file for analysis on another system. The only exception to this is scenarios where a significant amount of traffic must be captured over a longer period of time. If the information being sought is known to be in the header, the default 64 bytes of each frame may be used to get the required information while significantly reducing the size of the resulting capture file.

-c flag

To capture a certain number of frames and then exit, use the -c flag. Example usage: tcpdump will exit after capturing 100 frames by specifying -c 100.

-p flag

Normally when capturing traffic with tcpdump, it puts the network interface into promiscuous mode. When not running in promiscuous mode, the NIC only receives frames destined for its own MAC address as well as broadcast and multicast addresses. When switched into promiscuous mode, the interface shows every frame on the wire that arrives at the network interface. In a switched network, this generally has little impact on the capture. In networks where the device is connected to a vswitch also in promiscuous mode, or a hub, using -p can significantly limit noise in the capture when the only traffic of interest is to and from the system performing the capture.

-v flag

The -v flag controls the detail, or verbosity, of the output. Using more “v” options yields more detail, so use -v, -vv, or -vvv to view even more detail in the output printed to the console. This option does not affect the detail stored in a capture file when using the -w switch, but will instead cause the process to report the number of packets captured every 10 seconds.

-e flag

Normally tcpdump does not show any link layer information. Specify -e to display the source and destination MAC addresses, and VLAN tag information for any traffic tagged with 802.1q VLANs.

Example capture without -e

This capture shows the default output, containing no link layer information:

# tcpdump -ni igb1 -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
23:18:15.830706 IP 10.0.64.210.22 > 10.0.64.15.1395: P 2023587125:2023587241(116) ack 2091089207 win 65535
23:18:15.830851 IP 10.0.64.210.22 > 10.0.64.15.1395: P 116:232(116) ack 1 win 65535
23:18:15.831256 IP 10.0.64.15.1395 > 10.0.64.210.22: . ack 116 win 65299
23:18:15.839834 IP 10.0.64.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 4, prio 0, authtype none, intvl 1s, length 36
23:18:16.006407 IP 10.0.64.15.1395 > 10.0.64.210.22: . ack 232 win 65183
5 packets captured

Example capture using -e

Here the link layer information is included by using -e. Note the source and destination MAC addresses in addition to the source and destination IP addresses:

# tcpdump -ni igb1 -e -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
23:30:05.914958 00:0c:29:0b:c3:ed > 00:13:d4:f7:73:d2, ethertype IPv4 (0x0800), length 170: 10.0.64.210.22 > 10.0.64.15.1395: P 2023592509:2023592625(116) ack 2091091355 win 65535
23:30:05.915110 00:0c:29:0b:c3:ed > 00:13:d4:f7:73:d2, ethertype IPv4 (0x0800), length 170: 10.0.64.210.22 > 10.0.64.15.1395: P 116:232(116) ack 1 win 65535
23:30:05.915396 00:13:d4:f7:73:d2 > 00:0c:29:0b:c3:ed, ethertype IPv4 (0x0800), length 60: 10.0.64.15.1395 > 10.0.64.210.22: . ack 116 win 65299
23:30:05.973359 00:00:5e:00:01:04 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 10.0.64.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 4, prio 0, authtype none, intvl 1s, length 36
23:30:06.065200 00:13:d4:f7:73:d2 > 00:0c:29:0b:c3:ed, ethertype IPv4 (0x0800), length 60: 10.0.64.15.1395 > 10.0.64.210.22: . ack 232 win 65183
5 packets captured

tcpdump Filters

On most firewalls running tcpdump with no filters will produce so much output that it will prove very difficult to find traffic of interest. There are numerous filtering expressions available that limit the traffic displayed or captured.

Host filters

To filter for a specific host, append host and the IP address to the tcpdump command. To filter for host 192.168.1.100 use the following command:

# tcpdump -ni igb1 host 192.168.1.100

That will capture all traffic to and from that host. To only capture traffic being initiated by that host, use the src directive:

# tcpdump -ni igb1 src host 192.168.1.100

Similarly, filtering for traffic destined to that IP address is possible by specifying dst:

# tcpdump -ni igb1 dst host 192.168.1.100

Network filters

Network filters narrow the capture to a specific subnet using the net expression. Following net, specify a dotted quad ( 192.168.1.1), dotted triple ( 192.168.1), dotted pair ( 192.168) or simply a number ( 192). A dotted quad is equivalent to specifying host, a dotted triple uses a subnet mask of 255.255.255.0, a dotted pair uses 255.255.0.0, and a number alone uses 255.0.0.0.

The following command displays traffic to or from any host with a 192.168.1.x IP address:

# tcpdump -ni igb1 net 192.168.1

The next command will capture traffic to or from any host with a 10.x.x.x IP address:

# tcpdump -ni igb1 net 10

Those examples will capture all traffic to or from the specified network. The src or dst keywords may be used the same as with host filters to capture only traffic initiated by or destined to the specified network:

# tcpdump -ni igb1 src net 10

A CIDR mask can also be passed as an argument to net:

# tcpdump -ni igb1 src net 172.16.0.0/12

Protocol and port filters

Narrowing down by host or network frequently isn’t adequate to eliminate unnecessary traffic from a capture. Or the source or destination of traffic may not be significant, and all traffic of a certain type should be captured. In other cases, filtering out all traffic of a specific type can reduce noise.

TCP and UDP port filters

To filter on TCP and UDP ports, use the port directive. This captures both TCP and UDP traffic using the specified port either as a source or destination port. It can be combined with tcp or udp to specify the protocol, and src or dst to specify a source or destination port.

Capture all HTTP traffic
# tcpdump -ni igb1 tcp port 80
Capture all DNS traffic

Capture all DNS traffic (usually UDP, but some queries use TCP):

# tcpdump -ni igb1 port 53

Protocol filters

Specific protocols can be filtered using the proto directive or by using the protocol name directly. Parameters passed to the proto directive can be specified using the IP protocol number or one of the names icmp, igmp, igrp, pim, ah, esp, carp, vrrp, udp, or tcp. Because the normal protocol names are reserved words, they must be escaped with one or two backslashes when used with the proto directive, depending on the shell. The shell available in pfSense requires two backslashes to escape these protocol names. If a syntax error is returned, check that the protocol name is properly escaped.

The following capture will show all ICMP traffic on the igb1 interface:

# tcpdump -ni igb1 proto \\icmp

Specifying carp for the protocol will capture CARP traffic but it also needs -T carp in order to interpret the CARP packets correctly when viewing the output using tcpdump. The GUI makes this adjustment automatically when capturing CARP.

The following capture will show all CARP traffic on the igb1 interface, which can be useful to ensure CARP traffic is being sent and received on the specified interface. It also omits the proto keyword, showing that it works on its own:

# tcpdump -i igb1 -T carp carp

Negating a filter match

In addition to matching specific parameters, a filter match can be negated by specifying not in front of the filter expression. When troubleshooting something other than CARP, and its multicast heartbeats are cluttering the capture output, exclude it as follows:

# tcpdump -ni igb1 not proto \\carp

Combining filters

Any of the aforementioned filters can be combined using and or or. The following sections provide some examples.

Display all HTTP traffic to and from a host

Display all HTTP traffic to or from 192.168.1.11:

# tcpdump -ni igb1 host 192.168.1.11 and tcp port 80

Display all HTTP traffic to and from multiple hosts

Display all HTTP traffic from either 192.168.1.11 or 192.168.1.15:

# tcpdump -ni igb1 host 192.168.1.11 or host 192.168.1.15 and tcp port 80

Filter expression usage

Filter expressions must come after every command line flag used. Adding any flags after a filter expression will result in a syntax error.

Incorrect ordering

# tcpdump -ni igb1 -T carp carp -c 2
tcpdump: syntax error

Correct ordering

# tcpdump -ni igb1 -T carp -c 2 carp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 65535 bytes
14:50:07.426993 IP 198.51.100.12 > 224.0.0.18: CARPv2-advertise 36: vhid=11 advbase=1 advskew=0 authlen=7 counter=5449924379588860810
14:50:08.436849 IP 198.51.100.12 > 224.0.0.18: CARPv2-advertise 36: vhid=11 advbase=1 advskew=0 authlen=7 counter=5449924379588860810
2 packets captured
78 packets received by filter
0 packets dropped by kernel

More on Filters

This section covered the most commonly used tcpdump filter expressions, and probably covers all the syntax most users will need. However this barely scratches the surface of the possibilities. There are many documents on the web that cover tcpdump in general and filtering specifically. See Additional References at the end of this chapter for links to more resources.

Practical Troubleshooting Examples

This section details an approach we prefer for troubleshooting a few specific problems. There are multiple ways to approach any problem, but packet capturing can rarely be beat for its effectiveness. Examining the traffic on the wire provides a level of visibility into what is actually happening on the network

Port forward not working

In this example, a new port forward is failing to respond to a request from a host on the Internet. The troubleshooting steps outlined in Port Forward Troubleshooting offers one way to approach this, but sometimes packet capturing is the only or easiest way to find the source of the problem.

Start from WAN

First, make sure the traffic is getting to the WAN interface. Start a tcpdump session on the WAN interface, and watch for the traffic:

# tcpdump -ni igb1 tcp port 5900
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:02.444006 IP 172.17.11.9.37219 > 10.0.73.5.5900: S 3863112259:3863112259(0) win 65535 <mss 1260,nop,nop,sackOK>

In this case, a packet comes in from the WAN, so it is making it that far. Note that the first part of the TCP handshake, a packet with only SYN set (the S shown), is reaching the firewall. If the port forward was working, a SYN ACK (S.) packet would be shown in reply to the SYN. With no return traffic visible, it could be a firewall rule or the target system may be unreachable – turned off, not listening on the specified port, host firewall blocking the traffic, etc.

Check Internal Interface

The next step would be to run a tcpdump session on the internal interface associated with the port forward:

# tcpdump -ni igb0 tcp port 5900`
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:38.339926 IP 172.17.11.9.2302 > 192.168.30.5.5900: S 1481321921:1481321921(0) win 65535 <mss 1260,nop,nop,sackOK>

Looking at the internal traffic, the connection left the inside interface and the local IP address was translated correctly. If this local address matches what was expected, then both the port forward and the firewall rule are working properly, and connectivity to the local PC must be confirmed by other means. If no output was displayed, then there is a problem with the firewall rule or the port forward may have been incorrectly defined. For this example, the target system was unplugged.

IPsec tunnel will not connect

tcpdump has some awareness of the protocols being used, which can be very helpful in figuring out problems with IPsec tunnels. The next few examples will show how certain error conditions may present themselves when monitoring with tcpdump. The IPsec logs are usually more helpful, but this can confirm what is actually being seen by the firewall. For encrypted traffic such as IPsec, packet capturing of the traffic is of less value as the payload of the captured packets cannot be examined without additional parameters, but it is helpful to determine if traffic from the remote end is reaching the firewall and which phases complete.

This first tunnel has an unreachable peer:

# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes

19:11:11.542976 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:11:21.544644 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg

This tunnel attempt has a mismatched PSK, notice how it attempts to move to phase 2, but then stops:

# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
19:15:05.566352 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:15:05.623288 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
19:15:05.653504 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]

Now Phase 1 is OK but there is a mismatch in the Phase 2 information. It will repeatedly attempt phase 2 traffic but there will not be any traffic in the tunnel:

# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
19:17:18.447952 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:17:18.490278 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
19:17:18.520149 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:17:18.520761 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R inf[E]
19:17:18.525474 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]
19:17:19.527962 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]

Finally, a fully working tunnel with two-way traffic after Phase 1 and Phase 2 have completed!:

#  tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
21:50:11.238263 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
21:50:11.713364 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
21:50:11.799162 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
21:50:11.801706 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]
21:50:11.812809 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R inf[E]
21:50:12.820191 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]
21:50:12.836478 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R oakley-quick[E]
21:50:12.838499 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]
21:50:13.168425 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x1), length 132
21:50:13.171227 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x1), length 132
21:50:14.178820 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x2), length 132
21:50:14.181210 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x2), length 132
21:50:15.189349 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x3), length 132
21:50:15.191756 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x3), length 132

Traffic traversing an IPsec tunnel

Traffic can also be observed traversing IPsec tunnels by capturing on the enc0 interface. This can help determine if traffic is attempting to reach the far end by using the tunnel. All traffic for all IPsec tunnels appears on the enc0 interface.

In the following example, a host on one side of the tunnel is successfully sending an ICMP echo request (ping) to the far side, and receiving replies:

# tcpdump -ni enc0
tcpdump: WARNING: enc0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 65535 bytes
15:52:46.151098 (authentic,confidential): SPI 0xcd77e085: IP 10.3.0.1 > 10.7.0.1: ICMP echo request, id 44640, seq 0, length 64
15:52:46.151814 (authentic,confidential): SPI 0xc0afb14d: IP 10.7.0.1 > 10.3.0.1: ICMP echo reply, id 44640, seq 0, length 64
15:52:47.154243 (authentic,confidential): SPI 0xcd77e085: IP 10.3.0.1 > 10.7.0.1: ICMP echo request, id 44640, seq 1, length 64
15:52:47.154843 (authentic,confidential): SPI 0xc0afb14d: IP 10.7.0.1 > 10.3.0.1: ICMP echo reply, id 44640, seq 1, length 64

If traffic was not properly entering the tunnel, no output would be shown. If there is a firewall or internal routing issue on the far side, traffic will appear leaving but nothing will show returning.

Troubleshooting Outbound NAT

For complex environments where Manual Outbound NAT is needed, tcpdump can be of great assistance in troubleshooting the Outbound NAT configuration. One good capture to use is to look for traffic with private IP addresses on the WAN interface, as everything on WAN should be have NAT applied and appear to be a public IP address. The following capture will display any traffic with RFC 1918 IP addresses as the source or destination. This will show any traffic that is not matching one of the outbound NAT rules, providing information to help review the Outbound NAT configuration to find the problem:

# tcpdump -ni igb1 net 10 or net 192.168 or net 172.16.0.0/12