WAN in the Sky (keeps on routing)

Here's an interesting case I encountered recently that required some creative network engineering.

The customer is a web marketing company with three small offices and a number of mobile workers. They had existing Internet connectivity and a firewall at each location, and they wanted to establish office-to-office connectivity and remote access to server resources for their mobile staff. This all seemed like a very vanilla Internet VPN configuration until we got to the one major caveat:

All of the offices connect to the Internet via 4G LTE wireless routers. None have static public IPs - or any public IPs at all. Each location's firewall receives an RFC 1918 private IP and is NAT'd by the carrier.

The customer had been told by another consultant that what they wanted was not possible with their current connectivity. He told them they would need to install some sort of landline Internet connection in each office, something that would provide a public IP (and preferably a static public IP that would not change). Honestly, my initial reaction was similar.

However, due to a business partnership with their mobile carrier the customer had what amounted to free, unlimited 4G LTE data service for each of their offices and their remote staff. Installing new business-class landlines with comparable speeds (roughly 40Mbps synchronous) in each of their offices would represent a significant increase in their current connectivity costs (which were zero). Obviously, the preferred solution was to use what they had, even if there were some architectural compromises that needed to be made.

The customer is not hosting public content from the office networks, but they do have company-use server resources for file shares and other services. Due to the nature of the data they handle for business, the preference is to host those resources internally rather than moving them to a cloud-hosted service (which was my next suggestion).

The solution I ultimately devised costs the company just US$12 per month, but it meets all of their requirements:

  • Secure, encrypted office-to-office connectivity.
  • Secure, encrypted remote user access to office servers and resources.
  • Works with their NAT'd 4G LTE connections.
  • Self-administered (to prevent vendor dependence and lock-in).
  • Based on open source technology (to minimize cost).

The design does come with the following caveats:

  • It's not a true full mesh VPN design, but rather a hub-and-spoke topology.
  • There is the potential for network throughput bottleneck at the hub.
  • The hub represents a single point of failure for the virtual WAN.

The customer found these design compromises acceptable for their use case.

Problem: Everyone's a client

Each of the offices and the remote workers have Internet access, so the obvious solution to providing connectivity would be Virtual Private Networking (VPN). VPN encapsulates network traffic in an encrypted tunnel, protecting the privacy of the transported data even when traversing public networks. VPN allows us to use public networks (such as the Internet) for secure, private communication.

VPN topologies generally follow one of two models:

In the peer-to-peer model, both endpoints of the connection are reachable via the public network (i.e., they both have static public IP addresses) and the VPN tunnel between the endpoints can be initiated on-demand by either peer.

In the client-server model, one endpoint (the server) has a static public IP address and the other endpoint (the client) typically does not. The VPN tunnel is only initiated by the client, because the server does not know the address of the client endpoint.

In this scenario, we have a problem. All of the office firewalls and remote worker devices have private IP addresses and are NAT'd behind the carrier's public IP space. There is no way for any endpoint to connect to any other endpoint. They are all clients.

We Need A Server

In order to resolve this, we needed to introduce a VPN server into the topology. This VPN server would have a static public IP address and would be reachable by every VPN client. This VPN server would also serve as the hub of a hub-and-spoke topology, and route traffic between clients to allow for the office-to-office and remote access connectivity.

If one of the office connections had a static public IP address, we could have designated its firewall as the VPN server. Since none of the locations have even a public IP much less a static public IP, we instead had to utilize a cloud-hosted virtual private server (VPS) to serve as the VPN server and meet-me point for all of the VPN clients.

I typically utilize DigitalOcean for my VPS hosting as I've found them to be very reliable and cost-effective. For this solution, I created a VPS for the customer with 1GB of memory, 30GB of SSD storage, a static IPv4 address, and 2TB/month of data transfer (which was estimated to be sufficient for their data requirements). The VPS will cost the customer US$10 per month, plus US$2 per month for automatic weekly snapshot backups (which the customer opted for).

The memory and disk space were not a factor, and the VPN server could have run on less. Data throughput was the biggest consideration, and there was some concern about going with a less costly VPS option that only included 1TB of monthly transfer. The customer will watch the monthly transfer usage and if it's later determined that 2TB/month is overkill, it will be easy for them to downsize the VPS later and cut their monthly recurring costs. Likewise, if they need more transfer it's easy to scale up with only a brief downtime.

The VPS is running Ubuntu Server 16.04.1 LTS.

For VPN software, we selected OpenVPN. I have used OpenVPN many times in the past and always found it to be very effective, flexible, and reliable. Further, it has a number of benefits:

  • Ease of configuration.
  • Widely used and supported.
  • Cross-platform client support.
  • Tunnels use a single UDP port, so NAT/firewall traversal is not generally a problem.
  • It is free and open source.

The customer's existing firewalls are pfSense appliances, and pfSense includes support for OpenVPN as both a server and a client. The remote users' mobile devices are a blend of Linux, Windows, and Mac laptops, as well as Android and Apple phones and tablets. OpenVPN clients exist for all of these platforms and unlike other VPN protocols such as IPsec and L2TP, cross-platform configuration and interoperability is rarely an issue.

The Topology

Here is what the topology looks like for this VPN.

                  VPS |  vpnserver  |
                      | [public IP] |
                            | tun0
  +----------------+        |
  | mobile devices |        |
  +----------------+        |
           \                |
            \               |
             \              |
              \             |
          (     \           |  public Internet )
          (     ~~~~~~~~~~~~~~~~~~~~~~~~~      )
          (    (           VPN           )     )
          (    (      )     )
          (     ~~~~~~~~~~~~~~~~~~~~~~~~~      )
          (           /     |     \            )
                    /       |       \
                   /        |        \
          +---------+  +---------+  +---------+
          |    LTE  |  |   LTE   |  |   LTE   |
          |  router |  | router  |  | router  |
          +---------+  +---------+  +---------+
              /             |             \
       tun0  /              |  tun0        \  tun0 /               |  \
 +--------------+    +--------------+    +--------------+
 |  denver_fw   |    |  chicago_fw  |    |  stlouis_fw  |
 | [private IP] |    | [private IP] |    | [private IP] |
 +--------------+    +--------------+    +--------------+
    pfSense FW          pfSense FW          pfSense FW
 ~~~~~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~
(     Denver     )  (     Chicago    )  (   St. Louis    )
( )  ( )  ( )
 ~~~~~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~

Fig. 1: The VPN topology.

The "vpnserver" VPS is the hub of a hub-and-spoke topology, and all VPN clients connect to that VPS and establish a VPN tunnel to the server. You'll see in the configuration below that the VPN server then passes traffic between each VPN clients' discrete tunnel, performing "client-to-client" routing.

OpenVPN Configuration Basics

For the sake of brevity, I'm not going to go into depth about the basic installation and setup of OpenVPN. That process is well documented in an excellent tutorial written by Justin Ellingwood and hosted on the DigitalOcean community site. The tutorial covers OpenVPN server installation, certificate generation, and client configuration for all of the above-mentioned platforms, except pfSense. For pfSense configuration, see this page on the pfSense documentation site, omitting the Server Setup section.

In brief, the following steps are required:

  • Install the OpenVPN software.
  • Create CA and server SSL certificates.
  • Create client SSL certificates for each of your VPN client devices.
  • Enable the OpenVPN daemon to run the server configuration at boot.
  • Distribute the client SSL certificates to your users and configure the client devices.

Justin's tutorial also includes documentation about forwarding Internet traffic from the VPN clients through the OpenVPN server, if that is desired. It was not a part of this customer's solution though, so I won't be delving into that. That said it is a useful add-on configuration if you want to protect user Internet traffic from deep packet inspection by the mobile carrier, protect device Internet traffic from packet capture on shared public WiFi networks, or perform content filtering on end user Internet traffic, to cite a few example use cases. Just note that the caveats with doing so would be:

  • Increased latency for user Internet traffic.
  • Increased use of VPS data transfer (which may require purchasing a more expensive VPS option for a higher monthly data cap).

Building a WAN in the Cloud

Once you've performed the basic OpenVPN configuration outlined in the tutorial linked above, we need to perform some additional configuration on the VPN server. All of the options we will be configuring are on the server side, so other than the basic client setup in the aforementioned tutorials, no further client-side configuration should be required.

This configuration is based on the following assumptions:

The three pfSense firewalls hosted in the office locations have client SSL certificates that were generated with the following Common Names (CN):

Denver office - CN: denver.example.com

Chicago office - CN: chicago.example.com

St. Louis office - CN: stlouis.example.com

You can generate client certificates with whatever CNs you prefer, just make a note of what you use as it will be important later.

The Denver office LAN is (

The Chicago office LAN is

The St. Louis office LAN is

The VPN interface of the pfSense firewall in each office will be assigned an IP address by the VPN server, and that address will be the same each time the pfSense firewall connects. This is necessary in order for the VPN server to know where to route the traffic for each office LAN.

The VPN client IPs will be assigned from the network The VPN server will take the first IP (, and will assign the following IPs to each of the pfSense firewalls:

Denver firewall -

Chicago firewall -

St. Louis firewall -

The remaining mobile devices will be dynamically assigned an available IP address from the network.

The customer's internal network uses the domain 'example.com'. There is a DNS server in the Denver office and another in the Chicago office that resolve hostnames in the example.com domain. We will configure VPN clients to use these DNS servers, and to use the search domain 'example.com'.

Denver DNS -

Chicago DNS -

On the VPN server, we have the following SSL certificates and keys:

The CA certificate:


The server certificate and key:


The generated Diffie-Hellman keys:


The generated TLS authentication key:


Note: the files have all been named with a server- prefix purely for organization purposes. I occasionally have situations where multiple OpenVPN configurations exist on the same host, so I name all of the files associated with a particular configuration using a common prefix. You may name them however you like, just update the lines in the configuration below accordingly.

The main configuration for the OpenVPN server will be stored in:


The server configuration will be as follows (with comments inline):

port 1194
proto udp
dev tun
ca server-ca.crt
cert server.crt
key server.key
dh server-dh2048.pem
# the network to use for VPN clients
# assign client IPs as a common subnet, not discrete nets
topology subnet
# define the networks to be routed
# push routes to the VPN clients
push "route"
push "route"
push "route"
# push DNS servers and search domain
push "dhcp-option DNS"
push "dhcp-option DNS"
push "dhcp-option DOMAIN example.com"
# allow client to client communication
# directory for special client configurations
client-config-dir /etc/openvpn/ccd
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth server-ta.key 0
# enable network compression (optional)
# limit the number of connected clients (optional)
max-clients 50
user nobody
group nogroup
status /var/log/openvpn-status.log
log /var/log/openvpn.log
verb 3

Now create the client configuration directory:

mkdir /etc/openvpn/ccd

We will need to create three configuration files, one for each of the office firewalls, and named to match their client SSL certificate CN (important!):







These configuration files will tell the server to assign a reserved VPN client IP to each firewall (ifconfig-push), and will tell the server the network to route to each remote endpoint (iroute).

Once the configuration changes are made, (re)start the OpenVPN server daemon:

systemctl restart openvpn@server

Be sure to also enable the VPN server to start automatically at boot time:

systemctl enable openvpn@server

Restart the firewall OpenVPN client connection on each of the configured pfSense firewalls, and then test. Site-to-site connectivity should now be possible. Configured mobile devices should now be able to connect and reach all three office locations.

Additional Security Measures

There are a few additional security measures to consider:

Control your CA key

Don't leave your CA key on the VPN server. It is not necessary that it be there, and leaving the CA key on your server exposes the key to potential compromise. Back up your CA and server certificates and keys, and remove the CA key from the server.

If you will need additional SSL certificates later (for new mobile devices, etc.), my recommendation would be to move your Easy-RSA keys directory off of the VPN server and put it on a secure certificate-signing workstation hosted on a protected network. Don't leave your key on a VPS in the cloud where it might be used to sign unauthorized certificates.

Secure your VPN server

Make sure access to your VPN server is well-secured. Use key-based authentication for SSH, and disable password authentication. Change the SSH port to a high, non-standard port. It can still be scanned by a dedicated attacker, but having SSH listen on port 31498 instead of port 22 adds some additional protection from automated drive-by login attacks. Install fail2ban and configure it to block attackers with too many failed SSH authentication attempts.

You may change your OpenVPN listening port to something other than 1194. Again, using something like UDP port 32756 makes it just a little harder for casual attackers, and it's difficult even for dedicated attackers to reliably portscan open UDP ports. If you do this, be sure to update the configuration on your VPN clients to use the new port.

[Security through obscurity should not be one's only protection, but obscurity does have value in a layered, defense-in-depth model. If you're harder to find, you're harder to hit.]

Enable the UFW firewall, and only allow inbound access to your VPN port and your SSH port.

Do not use your VPN server for other purposes (such as hosting a web or mail server). Let this be a dedicated host. The only ports listening should be your SSH and OpenVPN ports.

One certificate per device

Don't use the same certificate for every client device and just allow multiple simultaneous connections using the same certificate. Issue a unique certificate for each device that will connect to your VPN. Yes, it is an added inconvenience to do so, but it is far less inconvenient than having to replace the client certificate on every one of your devices if that used-everywhere certificate is compromised by someone misplacing their iPhone at the pub. With unique client certificates, it's a relatively straightforward matter to revoke the compromised cert and re-issue a new certificate for the replacement device (once you setup the CRL config - Easy-RSA management is beyond the scope of this post). Having to replace a compromised common certificate on every user device would likely make for a Bad Day.

Add authentication

For a second factor of authentication for VPN clients (the first factor being the possession of the SSL certificates) you can choose to require password authentication in addition to certificate authentication. This 2FA approach requires something you have (the certificate and key) and something you know (the username and password) in order to gain access to the VPN.

Create the following script in /etc/openvpn/ucn.pl:

#!/usr/bin/perl -t

$username = $ENV{'username'};
$common_name = $ENV{'common_name'};

exit !(length($username) > 0 && length($common_name) 
  > 0 && $username eq $common_name);

And set the script to be executable:

chmod 755 ucn.pl

Then add the following configuration option to your server.conf to enable password authentication via PAM and enforce that the username supplied must match the certificate CN supplied:

plugin /usr/lib/openvpn/openvpn-plugin-auth-pam.so system-auth
auth-user-pass-verify ./ucn.pl via-env

Create a username and password on the VPN server for each device CN (be sure to use the /usr/sbin/nologin shell). If the user device is compromised, revoke and re-issue the client certificate and key, and change the device's account password.

Note: Technically, this is three-factor authentication if you're also using TLS authentication (ta.key), which we are above. But if the device is compromised and they got the client cert, they also likely got the ta.key. Requiring a password also (assuming it's manually entered each time the user connects and is not saved on the device!) protects against unauthorized use of the compromised cert. This will buy you some time to revoke and re-issue if there's a gap between the time the device is lost and the time the loss is reported.

Firewall rules on the endpoints

The VPN server does not perform any kind of packet filtering on traffic routed over the VPN. It's basically an any-to-any configuration. For the office networks where servers are hosted, consider implementing inbound traffic rules on the OpenVPN interfaces of the connected firewalls. For example, if there are two servers in the office, one for SFTP and one for DNS, implement rules on the firewall to only allow traffic from VPN networks to those two specific servers on ports tcp/22 and udp/53, respectively. Block all other inbound connections by default. This will help limit potential damage in the event that someone were to compromise the VPN server or the VPN network.

For remote devices such as laptops, phones, and tablets, consider configuring host-based firewall software to protect against unauthorized connection attempts. Most current operating systems include some firewall software by default.

Conclusion (You Can't Always Get What You Want)

This solution is definitely not the ideal architecture, but rather a case study in taking what you have and making something that works. It certainly wouldn't be my first choice of WAN topology, but given the customer constraints it did solve the problem at hand. Time will tell how well this solution holds up. What do you think? Is this a clever use of creative routing, or something from the don't-try-this-at-home-dept? Send me email to let me know your opinion.