Primary unit selection and monitored interfaces

If you have configured interface monitoring the cluster unit with the highest number of monitored interfaces that are connected to networks becomes the primary unit. Put another way, the cluster unit with the highest number of failed or disconnected monitored interfaces cannot become the primary unit.

Normally, when a cluster starts up, all monitored interfaces of all cluster units are connected and functioning normally. So monitored interfaces do not usually affect primary unit selection when the cluster first starts.

A cluster always renegotiates when a monitored interface fails or is disconnected (called link failover). A cluster also always renegotiates when a failed or disconnected monitored interface is restored.

If a primary unit monitored interface fails or is disconnected, the cluster renegotiates and if this is the only failed or disconnected monitored interface the cluster selects a new primary unit.

If a subordinate unit monitored interface fails or is disconnected, the cluster also renegotiates but will not necessarily select a new primary unit. However, the subordinate unit with the failed or disconnected monitored interface cannot become the primary unit.

Multiple monitored interfaces can fail or become disconnected on more than one cluster unit. Each time a monitored interface is disconnected or fails, the cluster negotiates to select the cluster unit with the most connected and operating monitored interfaces to become the primary unit. In fact, the intent of the link failover feature is just this, to make sure that the primary unit is always the cluster unit with the most connected and operating monitored interfaces.

 

Primary unit selection and age

The cluster unit with the highest age value becomes the primary unit. The age of a cluster unit is the amount of time since a monitored interface failed or is disconnected. Age is also reset when a cluster unit starts (boots up). So, when all cluster units start up at about the same time, they all have the same age. Age does not affect primary unit selection when all cluster units start up at the same time. Age also takes precedence over priority for primary unit selection.

If a link failure of a monitored interface occurs, the age value for the cluster unit that experiences the link failure is reset. So, the cluster unit that experienced the link failure also has a lower age value than the other cluster units. This reduced age does not effect primary unit selection because the number of link failures takes precedence over the age.

If the failed monitored interface is restored the cluster unit that had the failed monitored interface cannot become the primary unit because its age is still lower than the age of the other cluster units.

In most cases, the way that age is handled by the cluster reduces the number of times the cluster selects a new primary unit, which results in a more stable cluster since selecting a new primary unit has the potential to disrupt traffic.

 

Cluster age difference margin (grace period)

In any cluster, some of the cluster units may take longer to start up than others. This startup time difference can happen as a result of a number of issues and does not affect the normal operation of the cluster. To make sure that cluster units that start slower can still become primary units, by default the FGCP ignores age differences of up to 5 minutes (300 seconds).

In most cases, during normal operation this age difference margin or grace period helps clusters function as expected. However, the age difference margin can result in some unexpected behavior in some cases:

• During a cluster firmware upgrade with uninterruptible-upgrade enabled (the default configuration) the cluster should not select a new primary unit after the firmware of all cluster units has been updated. But since the age difference of the cluster units is most likely less than 300 seconds, age is not used to affect primary unit selection and the cluster may select a new primary unit.
• During failover testing where cluster units are failed over repeatedly the age difference between the cluster units will most likely be less than 5 minutes. During normal operation, if a failover occurs, when the failed unit rejoins the cluster its age will be very different from the age of the still operating cluster units so the cluster will not select a new primary unit. However, if a unit fails and is restored in a very short time the age difference may be less than 5 minutes. As a result the cluster may select a new primary unit during some failover testing scenarios.

Changing the cluster age difference margin

You can change the cluster age difference margin using the following command:

config system ha

set ha-uptime-diff-margin 60 end

This command sets the cluster age difference margin to 60 seconds (1 minute). The age difference margin range

1 to 65535 seconds. The default is 300 seconds.

You may want to reduce the margin if during failover testing you don’t want to wait the default age difference margin of 5 minutes. You may also want to reduce the margin to allow uninterruptible upgrades to work. See Upgrading cluster firmware on page 1483.

You may want to increase the age margin if cluster unit startup time differences are larger than 5 minutes.

 

Displaying cluster unit age differences

You can use the CLI command diagnose sys ha dump-by all-vcluster to display the age difference of the units in a cluster. This command also displays information about a number of HA-related parameters for each cluster unit. You can enter the command from the primary unit CLI or you can enter the command from a subordinate unit after using execute ha manage to log into a subordinate unit CLI. The information displayed by the command is relative to the unit that you enter the command from.

For example, a cluster of two FortiGate-5001C units with no changes to the default HA configuration except to enable port monitoring for port1. Entering the diagnose sys ha dump-by all-vcluster command from the primary unit CLI displays information similar to the following:

diagnose sys ha dump-by all-vcluster

HA information.

vcluster id=1, nventry=2, state=work, digest=4.e8.62.17.7b.1d… ventry idx=0,id=1,FG-5KC3E13800084,prio=128,0,claimed=0,override=0,

flag=0x01,time=0,mon=0 mondev=port1,50

ventry idx=1,id=1,FG-5KC3E13800051,prio=128,0,claimed=0,override=0, flag=0x00,time=189,mon=0

The command displays one ventry line for each cluster unit. The first ventry in the example contains information for the cluster unit that you are logged into (usually the primary unit). The other ventry lines contain information for the other units in the cluster (in the example there is only one other cluster unit). The command also includes a mondev entry that displays the interface monitoring configuration.

The time field is always 0 for the unit that you are logged into. The time field for the other cluster unit is the age difference between the unit that you are logged into and the other cluster unit. The age difference is in the form seconds/10.

In the example, the age of the subordinate unit is 18.9 seconds more than the age of the primary unit. The age difference is less than 5 minutes (less than 300 seconds) so age has no affect on primary unit selection. The cluster selected the unit with the highest serial number to be the primary unit.

If you use execute ha manage 1 to log into the subordinate unit CLI and enter diagnose sys ha dump

you get results similar to the following:

 

diagnose sys ha dump-by all-vcluster

HA information.

vcluster id=1, nventry=2, state=standby, digest=4.e8.62.17.7b.1d… ventry idx=1,id=1,FG-5KC3E13800051,prio=128,0,claimed=0,override=0,

flag=0x01,time=0,mon=0 mondev=port1,50

ventry idx=0,id=1,FG-5KC3E13800084,prio=128,0,claimed=1,override=0, flag=0x00,time=-189,mon=0

The time for the primary unit is -189, indicating that age of the subordinate unit age is 18.9 seconds higher than the primary unit age.

If port1 (the monitored interface) of the primary unit is disconnected, the cluster renegotiates and the former subordinate unit becomes the primary unit. When you log into the new primary unit CLI and enter diagnose sys ha dump-by all-vcluster you could get results similar to the following:

diagnose sys ha dump-by all-vcluster

HA information.

vcluster id=1, nventry=2, state=work, digest=3.f8.d1.63.4d.d2… ventry idx=0,id=1,FG-5KC3E13800046,prio=128,0,claimed=0,

override=0,flag=1,time=0,mon=0

mondev=port1,50

ventry idx=1,id=1,FG-5KC3E13800084,prio=128,-50,claimed=0, override=0,flag=0,time=1362,mon=0

The command results show that the age of the new primary unit is 136.2 seconds higher than the age of the new subordinate unit.

 

If port1 of the former primary unit is reconnected the cluster will once again make this the primary unit because the age difference will still be less than 300 seconds. When you log into the primary unit CLI and enter diagnose sys ha dump-by all-vcluster you get results similar to the following:

diagnose sys ha dump-by all-vcluster

HA information.

vcluster id=1, nventry=2, state=work, digest=4.a5.60.11.cf.d4… ventry idx=0,id=1,FG-5KC3E13800084,prio=128,0,claimed=0,

override=0,flag=1,time=0,mon=0

mondev=port1,50

ventry idx=1,id=1,FG-5KC3E13800046,prio=128,0,claimed=0, override=0,flag=0,time=-1362,mon=0