OpenVPN Maintenance

[ OpenBSD, Stuart Henderson’s Dropping UDP Packets ]

Visibility of what is happening with your network traffic is difficult. Performance issues with the network could be in several places, let alone the VPN tunnel itself. The following are gleaned notes that have helped me.

Review the network behaviour by checking whether there are visible problems with UDP packets using netstat

The netstat command symbolically displays the contents of various
network-related data structures. There are a number of output formats,
depending on the options for the information presented.

A useful, simple(?), incantation of the command to show us what active connections we have coming through the net can be:

Show all statistics for the ‘address family’ inet.

netstat -an -f inet
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp          0      0  Internal.9999          OtherIPAd.27161        TIME_WAIT
tcp          0    336  Internal.22            MyIPAddres.1595        ESTABLISHED
tcp          0      0  Internal.22            MyIPAddres.1161        ESTABLISHED
tcp          0      0  *.80                   *.*                    LISTEN

Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
udp          0      0  External-IPAddr.24874  RemoteTmeSvr1.123
udp          0      0  External-IPAddr.24257  RemoteTmeSvr2.123
udp          0      0  External-IPAddr.42424  RemoteTmeSvr3.123
udp          0      0  External-IPAddr.1807   RemoteTmeSvr4.123
udp          0      0  *.1194                 *.*

The above is a quick summary of what’s happening on the wire at the moment.

OK, so I’m connecting to the box on TCP port 22 (ssh) and there’s a dormant server on TCP port 80.

UDP has something listening on port 1194, as well as a bunch of outgoing connections to UDP port 123 (NTP)

The number of connections is not necessary good or bad, although if you have ‘screens’ full of connections then you will want to diagnose further and assess whether your system is capable.

As our concern here is with OpenVPN and UDP traffic, netstat can give us a summary:

netstat -sp udp
udp:
    432883673 datagrams received
    0 with incomplete header
    0 with bad data length field
    0 with bad checksum
    949750 with no checksum
    431894815 input packets hardware-checksummed
    0 output packets hardware-checksummed
    60472 dropped due to no socket
    432556938 broadcast/multicast datagrams dropped due to no socket
    0 dropped due to missing IPsec protection
    0 dropped due to full socket buffers
    266263 delivered
    321743 datagrams output
    288968 missed PCB cache

The summary screen gives a good historical snapshot, whoa, that doesn’t look good at all.

Our sample server has a huge number of “dropped due to no socket” errors. As historical data, it could have all been yesterday or today but we have seen dropped packets.

The next step of information gathering is to monitor/review which sockets have queues (the sendq/recvq counters)

netstat -an 

which you can further isolate with a similar invocation (but removing output for interfaces without any queueing)

netstat -an | grep -v ' 0      0'

[ Ref: Michel Blais’ Re: net.inet.ip.ifq.maxlen ]

You can monitor the packet drops on interface queues by looking at net.inet.ip.ifq.drops:

sysctl net.inet.ip.ifq.drops

If you have rapidly incrementing ‘drops’, then the next step is to determine what is the reason for that queue growth.

  • Do we have queueing
  • How is fragmentation handled
  • Fragmentation strategies in place

Use the dropped packets/second to value the performance loss/gain to expect.

Basic optimisation of the Packet Filtering/queueing can include the following:

  • remove duplicate rules
  • remove rules that are a subset of another rule
  • combine multiple rules into a table when advantageous

For firewall ruleset optimisation refer dhartmei’s OpenBSD Journal article

A general review of PF ‘information’ may give you some indicators.

sudo pfctl -si

Are packets blocked, as well as queued?

Where possible, review the queueing issues at both ends of the UDP service to better determine whether the dropped UDP packets lean more towards the receiving or ending, and if you must, adjust accordingly

  • net.inet.udp.recvspace=41600
  • net.inet.udp.sendspace=9216
sysctl net.inet.udp.sendspace=????