How IPSec VPN Works

So far we have a overview of IPSec concept and how the Security Associations are established. Before a further discussion, here is the IPSec VPN’s operation broken down into five main steps:

1. The initial packet matching correspondent IPSec VPN policies and attempting to pass through the IPSec VPN gateway triggers the IKE processes to establish Security Associations.
2. During IKE Phase 1, IKE proposals are negotiated, secret keys are shared and the two IPSec endpoints are authenticated. The ISAKMP SA is established for IKE Phase 2.
3. IKE Phase 2 negotiates new parameters and calculates new secret keys. The IPSec SA is established for VPN communications.
4. Communications over the two IPSec VPN gateways are protected according on the security parameters and keys stored in Security Association database. Data packets are encapsulated with ESP header and new IP header,and transferred over the IPSec VPN tunnel.
5. IPSec SAs terminate by timing out.

Modes of IPSec VPN data transmission

IPSec transfers the encrypted or authenticated IP packets (ESP or AH encapsulated packets) in a host-to-host transport mode, as well as in a tunneling mode. Packet exchanges during IKE Phase 1 and Phase 2 are nothing about the two modes.

Tunnel mode

IPSec Tunnel mode is commonly used for site-to-site communications by tunneling through incompatible networks. For example, it delivers protected communications between two private networks through Internet, which is a typical IPSec VPN. In IPSec tunnel mode, the original IP packet is entirely encrypted (not only the payload data but also the routing information are encrypted), and is encapsulated with a new IP header. With the new IP header encapsulation and decapsulation, two incompatible networks deliver encrypted packets to each other by tunneling through Internet.

Transport mode

IPSec Transport mode is used for communications between two end-stations (host-to-host). An end-station can be a IPSec gateway or just a host running IPSec server/client. Both are actually the destination to each other while communicating. The basic concept of IPsec Transport mode is that the original IP header is intact; the routing is neither modified nor encrypted. Transport mode only provides protection of the payload of the original IP packet by encryption. The two endpoints are supposed to be accessible to each other originally. Usually, Transport mode is applied to other tunneling protocols to provide protection of GRE/L2TP encapsulated IP data packets ( GRE/L2TP transmission over IPSec protection). FortiWAN IPSec Transport mode is only available for Tunnel Routing.