IPSec VPN overview

VPN Tunnels

Tunneling is a technique to perform data transmission for a foreign protocol over a incompatible network; such as running IPv6 over IPv4, and the transmission of data for use within a private, corporate network through a public network. Tunneling is done by encapsulating and decapsulating data and information of the particular protocol within the incompatible transmission units symmetrically. IPSec protocol sets define the processes, which is the Tunnel Mode we will introduce later (See Modes of IPSec VPN data transmission), to deliver encryption protected data between incompatible networks by tunneling through an intermediate network. IPSec offers another option to deliver protected data end-to-end without tunneling, which is called Transport Mode (See Modes of IPSec VPN data transmission). It provides the flexibility to integrate other tunneling protocols with IPSec to establish a VPN network.

Secure data transmission

IPSec employs encryption and authentication of data packets for VPN transmission to ensures that any third-party from public network who intercepts the packets can not access the data and impersonate each endpoint. It protects the communications between two endpoints against malicious attacks from intermediate, untrusted network, so that privacy and authenticity are guaranteed to the communications. However, it is concerned that how the two endpoints securely share the encryption and authentication methods, and the correspondent secret key without compromising them to others. This is the major object that IPSec functions for. Once these security parameters are shared securely between the two entities, which is called a establishment of Security Association (See IPSec key exchange), the privacy and authentication of data transmission are guaranteed.

Basic IPSec VPN scenario

To connect two incompatible networks within an IPSec VPN network over an intermediate network, an IPSec VPN device is required to be deployed in front of each the network. The IPSec VPN devices (the FortiWAN units) establish an IPSec VPN tunnel with each other. Each of the IPSec VPN devices performs the processes to encrypt and encapsulate, or decapsulate and decrypt the incoming packets (from the network behind it or the opposite IPSec VPN device), and then forwards the packets to the destination (the opposite IPSec VPN device or the network behind it). The two incompatible networks, therefore, have the secure access to each other through the two IPSec VPN devices (the IPSec VPN tunnel established between the two devices). A host in the network communicates with a opposite host (in the opposite network) without running any IPSec VPN software; what they do is like performing a communication in the same network as usual. All the processes and details for a IPSec VPN communication are taken by the two IPSec VPN devices; hosts are not aware of this. The IPSec VPN devices are so-called IPSec VPN gateways, and this is the typical site-to-site VPN.

VPN tunnel between two private networks

The above diagram shows an IPSec VPN connection between two private networks, which two FortiWAN units (two endpoints of the VPN tunnel) functions as the IPSec VPN gateways for. The IPSec VPN tunnel is established through public IP addresses (for example 1.1.1.1 and 2.2.2.2) of FortiWAN’s WAN interfaces. FortiWAN A receives packets from site A network (192.168.1.0/24) with source IP 192.168.1.10 and destination IP 192.168.2.10 (site B network), and then performs:

• encrypt packets with shared security parameters (algorithms and secret keys) l encapsulate packets with a new IP header that source IP is 1.1.1.1 and destination IP is 2.2.2.2. l forward packets to the site B network (FortiWAN B) FortiWAN B receives the packets and performs:
• recover the encrypted packets by decapsulation l recover the original data and IP header by decryption l forward packets to host 192.168.2.10

Processes for traffic in the opposite direction are the same. From the standpoint of FortiWAN A, FortiWAN A is local unit and FortiWAN B is the remote unit, vice versa.

IPSec key exchange

After the basic concept of IPSec VPN introduced above, here comes the details of IPSec’s key exchange processes which is the major part to configure an IPSec VPN. As the previous discussion, IPSec performs data encryption and authentication for the VPN communications. The way to securely distribute a common secret key to each endpoint is essential to make the secure data transmission complete. After all, a encrypted data is no longer secure if its secret key is not safe or compromised. Before we take look into IPSec’s key exchange, a basic concept of encryption and authentication is introduced first.

Encryption

Encryption mathematically transforms data to meaningless random numbers. The original data is called plaintext and the encrypted data is called ciphertext. The opposite process, called decryption, performs the inverse operation to recover the original plaintext from the ciphertext. The process by which the plaintext is transformed to ciphertext and back again is called an algorithm. All algorithms use a small piece of information, a key, in the arithmetic process of converted plaintext to ciphertext, or vice-versa. IPSec uses symmetrical algorithms, which the same key is used for both encrypt and decrypt the data. The length of the key is one of the factors determining the security of an encryption algorithm. FortiWAN IPsec VPNs offer the following encryption algorithms, in descending order of security:

AES256	A 128-bit block algorithm that uses a 256-bit key.
AES192	A 128-bit block algorithm that uses a 192-bit key.
AES128	A 128-bit block algorithm that uses a 128-bit key.
3DES	Triple-DES, in which plain text is DES-encrypted three times by three keys.
DES	Digital Encryption Standard, a 64-bit block algorithm that uses a 56-bit key.

Authentication

In Information Security (or Cryptography), Authentication is the process of determining whether someone or something is, in fact, who or what it is declared to be. In authentication, one has to prove its identity to the remote one, and the identity will be verified by the remote one. A typical providing proof can be a certificate or username and password. In cryptography, a message authentication code (MAC) is a short piece of information used to authenticate a message—in other words, to provide integrity and authenticity assurances on the message. Integrity assurances detect accidental and intentional message changes, while authenticity assurances affirm the message’s origin. A MAC algorithm, sometimes called a keyed (cryptographic) hash function (however, cryptographic hash function is only one of the possible ways to generate MACs), accepts as input a secret key and an arbitrary-length message to be authenticated, and outputs a MAC (sometimes known as a tag). The MAC value protects both a message’s data integrity as well as its authenticity, by allowing verifiers (who also possess the secret key) to detect any changes to the message content. As with any MAC, it may be used to simultaneously verify both the data integrity and the authentication of a message. FortiWAN IPsec VPNs offer the following MAC algorithms, in descending order of security:

hmac-sha512	A SHA512-based MAC algorithm with 512-bit hash output.
hmac-sha384	A SHA384-based MAC algorithm with 384-bit hash output.
hmac-sha256	A SHA256-based MAC algorithm with 256-bit hash output.
hmac-sha1	A SHA1-based MAC algorithm with 160-bit hash output.
hmac-md5	A MD5-based MAC algorithm with 128-bit hash output.

Security Association

To support secure communications (data encryption and authentication) between two VPN gateways, the common security attributes must be shared in advance, which are the cryptographic and authentication algorithms, encryption secret key and other necessary parameters. A common set of the security attributes maintained by two IPSec VPN gateways for an IPSec VPN tunnel is what called Security Association (SA), which is used to provide a secure channel and protect the communications between the two site networks. Each of the two IPSec VPN gateways encrypts/decrypts data according to the established Security Association. The process to establish a Security Association involves sharing and negotiation of the security attributes.

IKE key exchange

Internet Key Exchange (IKE) is the protocol used to establish a Security Association (SA), which is included in the IPSec protocol suite. The purposes of IKE are to l Negotiate an encrypt algorithm and an authentication algorithm l Generate a shared secret key to encrypt/decrypt IPSec VPN communications (data transmission).

Both are used by IPSec VPN to provide secure communications between two endpoints.

IKE consists of two phases, Phase 1 and Phase 2. The purpose of IKE Phase 1 is to establish a secure and authenticated channel, which is actually a Security Association (called ISAKMP SA as well), between two entities for further IKE Phase 2 negotiations. With the protection of ISAKMP SA, Phase 2 will then be performed to establish the final Security Association (called IPSec SA as well) used to protect the VPN communications (data transmission) between two sites. In other words, before users’ VPN communication starts (data packet being transferred to each other), the corresponding IKE Phase 1 and Phase 2 must be done to establish the SAs between the two VPN gateways. With the established SA between two VPN gateways, privacy and authenticity are so that guaranteed to the VPN communications (by encryption and authentication). Basically, IKE Phase 1 authenticates a remote peer and sets up a secure channel for going forward Phase 2 negotiations to establish the IPSec SA.

IKE Phase 1

Before we talk about the details of IKE Phase 1, let us have an overview on Phase 1’s Identity Verification (Authentication). The endpoint who begins the IKE Phase1 negotiation makes a declaration of who it is to the opposite endpoint, and the opposite endpoint verifies the identity. FortiWAN’s IPSec employs a pre-shared key to achieve the identity verification. The pre-shared key is a common key (similar to a password) pre-shared between the two entities who join in the Phase 1 negotiations. This pre-shared key is used for verification of the declared identity in a cryptographic system (MAC calculation of the identity). This mechanism is on the premise that the pre-shared key is never compromised to the third-party. Although it looks like a password, the pre-shared key, also known as a shared secret, is never sent by either endpoint during the processes of authentication. Actually, the pre-shared key is involved in the calculations of encryption keys, which is actually used for the authentication, at each endpoint.Unmatched pre-shared keys result in unmatched encryption keys, and indirectly cause the authentication in IKE Phase 1 failed.

Now back to the IKE Phase 1. Phase 1 achieves the following objectives to establish ISAKMP Security Association:

IKE Proposals negotiation

An IKE proposal is a set of necessary parameters for negotiations to establish a Security Association. The negotiation initiator offers opposite endpoint the proposals of the suggested encryption and authentication algorithms, the time-period that keys should remain active, and the strength of the keys used in Diffie-Hellman key exchange process. The opposite endpoint chooses an appropriate proposal and responds it to the initiator, so that the algorithms and other parameters used to protect data transmission between two endpoints are determined.

Generate the secret key for encryption

A secret key is necessary for the established ISAKMP Security Association to work with the determined encryption and authentication protocols. Therefore, except the negotiations of IKE proposals, a secret key must be determined and shared between the two entities during IKE Phase 1 negotiations. However, it is insecure to send a secret key directly to the opposite endpoint over the public network (no SA protection is offered during Phase 1 negotiations). Diffie-Hellman key exchange, which is a method used to securely exchange cryptographic keys over a public channel, is introduced to IKE to generate the secret key. The two entities running a Diffie-Hellman key exchange will start by exchanging key materials, which are public to third-party, via the public network. With the key materials, calculation of Diffie-Hellman key exchange performed on each of the endpoints derives a common value, which is a seed to generate the secret key we need. With the private and common seed, the two endpoints further calculate the common secret key, and so that the secret key is securely shared. Actually, the pre-shared key used for identity authentication is involved in the final calculations generating the secret key.

Authentication

Identity protection

The two endpoints running the Phase 1 processes declare its identity to each other. A pre-shared key between the two entities is used to verify the declared identity and thus prevent malicious attacks from counterfeit identity. With cryptographic method and the pre-shared key, one can prove its identity to the opposite end. Although it looks like a password, the pre-shared key, also known as a shared secret, is never sent by either gateway. Actually, it is involved in the generation of encryption secret key.

Message integrity

A message authentication code (MAC) not only verifies identity but also provides integrity and authenticity assurances on the exchanged messages. The MAC value protects both a message’s data integrity as well as its authenticity against man-in-the-middle attacks or tampering.

Main mode and Aggressive mode

Phase 1 parameters are exchanged in either Main mode or Aggressive mode:

In Main mode, the processes of IKE Phase 1 consists of six message exchanges. An IKE Phase 1 session begins with IKE proposals negotiations between initiator and responder (as the previous description). In the next two message exchanges, the necessary keying materials are exchanged to calculate the common secret key at both ends. For the last two exchanges, encrypted authentication information is exchanged to verify the identity and message integrity on each end.

In Aggressive mode, the processes of IKE Phase 1 is squeezed into three message exchanges. All data required for IKE proposal negotiation and Diffie-Hellman key exchange passed by the initiator and responder in the first two message exchanges. Unencrypted authentication information for sessions passed in the second and third message exchanges. Comparing with main mode, aggressive mode might not be such secure (weak identity protection and risk of pre-shared key crack), the advantage to aggressive mode is that it is faster than Main mode however. FortiWAN’s IPSec, however, does not support IKE Phase 1 in Aggressive mode, only Main mode is available.

The successful outcome of Phase 1 negotiations (either aggressive mode or main mode) establishes the ISAKMP Security Association, and the Phase 2 negotiation begins immediately. Phase 2 negotiations will be protected (encryption) within the ISAKMP Security Association.

IKE Phase 2

Under the protection of ISAKMP Security Association, IKE Phase 2 performs parameters negotiations to establish the IPsec Security Association which protects the subsequent IPSec VPN communications. IKE Phase 2 is processed in one mode called Quick Mode (New Group Mode is not supported by FortiWAN). Similar to Phase 1, in IKE Phase 2, another proposal of encryption and authentication algorithms is negotiated, shared secret keys are derived, and the negotiation sessions are authenticated. The negotiated encryption and authentication algorithms, derived secret keys and other necessary parameters, which are the successful outcome of IKE Phase 2, constitute the IPSec Security Association. So that the security association between two IPSec VPN gateways is established, and the VPN communications are so that protected.

Perfect Forward Secrecy, PFS

Perfect Forward Secrecy is a property of communication security that past session keys can not be compromised by the compromise of long-term keys if a session key is associated to the long-term key in some way. Actually, the shared secret key we introduced in IKE Phase 2 is derived by calculation with the secret key derived in IKE Phase 1 and some insecure (is public to any third-party) parameters (a Diffie-Hellman exchange is not involved in the calculation), if PFS is not enabled for IKE Phase 2. Once the secret key of IKE Phase 1 is compromised to an attacker, all the secret session keys derived in IKE Phase 2 might become compromised. With enabling PFS, the calculation of secret keys involves a new Diffie-Hellman exchange. The private key material of Diffie-Hellman exchange protects the session secret keys of IKE Phase 2 from the compromise of IKE Phase 1’s keys. However, system performance might be concerned if Diffie-Hellman exchange is performed twice (Phase 1 and Phase 2 individually) for a establishment of IPsec Security Association.