Beyond the Marketing: The Architecture of VPNs, Overlays, and IPsec

“VPN” has become one of the most overloaded terms in networking. In consumer contexts, it implies privacy or geolocation masking. In enterprise environments, it implies secure transport, identity enforcement, and controlled access.

From an engineering perspective, however, a VPN is rarely a single protocol. It is a stack of technologies combining encapsulation (tunneling) and cryptography (confidentiality). Whether dealing with a simple privacy app or a complex SD-WAN fabric, the architecture is defined by how the payload is wrapped, encrypted, and routed.

This article dissects the underlying mechanisms—IPsec, GRE, WireGuard, and Protocol Breaking—to clarify why Consumer VPNs, Enterprise VPNs, Identity Overlays, and OT access platforms are fundamentally different tools designed for different risk models.

1. Consumer VPNs vs. Enterprise VPNs: The Architectural Divergence

While both utilize encryption, the routing logic and trust models differ fundamentally.

Consumer VPNs (Encrypted Proxies)

• Primary Goal: Privacy, anonymity, and geolocation masking.
• The Mechanism: These function as a "black box" tunnel. The provider creates a single encrypted pipe from the user device to a Point of Presence (POP).
• Packet Flow:

Crucially, the provider performs PAT (Port Address Translation) at the egress point.

• Why they are not Enterprise tools:
  • No Internal Routing: There is no extension of the corporate routing table.
  • No Segmentation: It is a single-hop architecture.
  • No Device Trust: They do not validate the security posture of the endpoint.
  • Zero Visibility: From an enterprise perspective, these are simply opaque, encrypted proxies.

Enterprise VPNs (Routed Access)

• Primary Goal: Secure extension of the private network.
• The Mechanism: These rely on Authenticated Peers and Routed Access. When a tunnel is established, the remote device or site effectively becomes a node on the internal network.
• Key Differentiator: Traffic is identified by Traffic Selectors (ACLs) or routed via Virtual Tunnel Interfaces (VTI). If the cryptographic check passes, the packet is trusted and routed to internal resources.

2. The Foundation: IPsec and Traditional Tunnels

To understand enterprise connectivity, one must understand IPsec (Internet Protocol Security). Unlike SSL/TLS which operates at the application layer, IPsec operates at Layer 3, securing the IP packet itself.

The IPsec Protocol Suite

IPsec is not a single protocol, but a concert of three mechanisms:

1. IKE (Internet Key Exchange): The negotiation plane. Peers authenticate and use Diffie-Hellman to derive session keys.
2. ESP (Encapsulating Security Payload): The data carrier. It provides encryption (confidentiality) and anti-replay protection.
3. Tunnel Mode: The standard for Site-to-Site VPNs. It encapsulates the entire original IP packet inside a new outer IP header.

Packet = New IP + ESP Header + Encrypted (Original IP + Payload)

Traditional Implementation 1: Site-to-Site IPsec VPN

• Use Case: Branch-to-HQ, DC-to-Cloud.
• The Risk: It relies on Network-Based Trust. Once the tunnel is up, the routing table dictates access. In mesh designs, managing these static tunnels becomes operationally heavy.

Traditional Implementation 2: Remote Access VPN

• Use Case: Employee access (e.g., Cisco AnyConnect, GlobalProtect).
• The Risk: While it adds user authentication (incl. MFA), it still operates as a Network Extension. The user’s device traffic is "hairpinned" into the corporate network, introducing lateral movement risks if the device is compromised.

3. Overlay Networks: Solving the IPsec Limitation

Overlays solve a specific limitation of pure IPsec: IPsec does not natively support multicast traffic. Multicast is required for dynamic routing protocols (OSPF, EIGRP, BGP) to function. To get dynamic routing over a VPN, we must "overlay" a new protocol.

The Mechanics of an Overlay

An overlay creates a logical topology on top of an existing underlay.

• GRE (Generic Routing Encapsulation): A tunneling protocol that can carry multicast routing updates, but offers no encryption.
• The Stack: To secure an overlay, we wrap GRE inside IPsec.

Physical IP > IPsec ESP > GRE Header > Original IP > Payload

Cisco DMVPN (Dynamic Multipoint VPN)

DMVPN cleanly separates the Topology (handled by mGRE and NHRP) from the Security (handled by IPsec). It remains the canonical example of scalable WAN design, allowing branches to speak directly to each other (dynamic mesh) without hairpinning through HQ.

SD-WAN (Automated Overlays)

SD-WAN is effectively DMVPN operationalized.

• The Evolution: Instead of manually configuring IKE and GRE, a centralized controller pushes policy and keys to edge devices.
• Capabilities: It adds application-aware routing to the encrypted overlay, allowing the network to route traffic based on performance (jitter/latency) rather than just IP subnets.

4. Identity-Based Overlays (The Modern Shift)

Newer platforms like Tailscale and NetBird are rejecting the complexity of IPsec (specifically the chatty IKE negotiation) in favor of WireGuard.

The Cryptographic Shift

• Protocol: WireGuard (using the Noise protocol framework built based on DH Diffie-Hellman Key Exchange).
• Cryptokey Routing: Unlike IPsec, which creates a tunnel and then checks ACLs, WireGuard binds Identity to the IP address.
  • The Public Key is the Identity.
  • A peer will only decrypt a packet if it originates from the specific IP tied to that key.
• Outcome: This creates a peer-to-peer mesh that is far lighter and easier to manage than traditional IPsec/IKE stacks, making it ideal for DevOps, Cloud, and IT access.

5. OT Environments: Why Tunnels Fail the Risk Model

Operational Technology (OT) prioritizes safety and determinism. Traditional VPNs whether IPsec or WireGuard introduce a fatal flaw for OT: Layer 3 Reachability.

The Lateral Movement Risk

Because VPNs function as "long Ethernet cables," they extend the network. If a remote engineer’s laptop is infected with malware, that malware can scan the OT network across the IPsec tunnel.

The Solution: Protocol Breaking (ZTNA)

OT-specific platforms (Cyolo, BeyondTrust, Claroty) reject tunneling entirely. They function as Reverse Proxies.

1. Session Brokerage: The user connects to a portal (HTML5/Client).
2. Protocol Break: The TCP session terminates at the proxy. The proxy validates the request and initiates a new, separate session to the PLC or SCADA system.
3. Outcome: There is no routed IP connectivity between the user and the asset. Malware cannot "scan" across a gap where no IP packets can flow. This meets IEC 62443 compliance standards for zone segregation.

6. Architectural Summary: Choosing the Right Tool

A strong network architecture does not rely on a single "VPN" solution. It layers connectivity models based on threat profiles.

Final Takeaway

Don't get caught up in the marketing jargon.

• A Tunnel (IPsec/GRE) implies network extension and routing.
• Encryption implies confidentiality, not authorization.
• Proxies (OT/ZTNA) imply session isolation.

Strong architectures emerge when connectivity models are selected based on the underlying packet mechanics and business risk, not just the "VPN" label.