Using Link Health Monitor and e-mail alerts
Another tool available to you on FortiGate units is the Link Health Monitor, useful for dead gateway detection. This feature allows the FortiGate unit to ping a gateway at regular intervals to ensure it is online and working. When the gateway is not accessible, that interface is marked as down.
To detect possible routing loops with Link Health Monitor
Use the following command to configure dead gateway detection:
config system link-monitor edit “test”
set srcintf “internal4” set server “8.8.8.8”
set interval 5 set failtime 1
end
Set the Interval (how often to send a ping) and failtime (how many lost pings is considered a failure). A smaller interval and smaller number of lost pings will result in faster detection, but will create more traffic on your network.
You may also want to log CPU and Memory usage as a network outage will cause your CPU activity to spike.
If you have VDOMs configured, you will have to enter the basic SMTP server inform- ation in the Global section, and the rest of the configuration within the VDOM that includes this interface.
After this configuration, when this interface on the FortiGate unit cannot connect to the next router, the FortiGate unit will bring down the interface and alert you with an email to the outage.
Looking at the packet flow
If you want to see what is happening on your network, look at the packets travelling on the network. In this situation, you are looking for routes that have metrics higher than 15 as that indicates they are unreachable. Ideally if you debug the flow of the packets, and record the routes that are unreachable, you can create an accurate picture of the network outage.
Action to take on discovering a routing loop
Once you have mapped the problem on your network, and determined it is in fact a routing loop there are a number of steps to take in correcting it.
1. Get any offline routers back online. This may be a simple reboot, or you may have to replace hardware. Often this first step will restore your network to its normal operation, once the routing tables finish being updated.
2. Change your routing configuration on the edges of the outage. Even if step 1 brought your network back online, you should consider making changes to improve your network before the next outage occurs. These changes can include configuring features like holddowns and triggers for updates, split horizon, and poison reverse updates.
Split horizon and Poison reverse updates
Split horizon is best explained with an example. You have three routers linked serially, let’s call them A, B, and C. A is only linked to B, C is only linked to B, and B is linked to both A and C. To get to C, A must go through B. If the link to C goes down, it is possible that B will try to use A’s route to get to C. This route is A-B-C, so it will not work. However, if B tries to use it this begins an endless loop.
This situation is called a split horizon because from B’s point of view the horizon stretches out in each direction, but in reality it only is on one side.
Poison reverse is the method used to prevent routes from running into split horizon problems. Poison reverse “poisons” routes away from the destination that use the current router in their route to the destination. This “poisoned” route is marked as unreachable for routers that cannot use it. In IS-IS this means that route is marked with a distance of 16.
Simple IS-IS example
This is an example of a typical medium-sized network configuration using IS-IS routing.
Imagine a company with four FortiGate devices connected to one another. A FortiGate at one end of the network connects to two routers, each with its own local subnet; one of these routers uses OSPF and the other uses RIP.
Your task is to configure the four FortiGates to route traffic and process network updates using IS-IS, such that the farthest FortiGate (see ‘FGT4’ in Network layout and assumptions on page 429) receives route updates for the two routers at the opposite end of the network. Furthermore, FGT4 has been given a loopback subnet that must be identified by the router running RIP.
Since the internal networks use OSPF and RIP, those protocols will need to be redistributed through the IS-IS network. To keep the example simple, there will be no authentication of router traffic.
With IS-IS properly configured in this example, if a router fails or temporarily goes offline, the route change will propagate throughout the system.
This section includes the following topics:
- Network layout and assumptions
- Expectations
- CLI configuration
- Verification
- Troubleshooting
Network layout and assumptions
Routing domains
IP scheme and interfaces
Local subnets
10.1.1.0/24
10.2.2.0/24
Local subnets
10.3.3.0/24
10.4.4.0/24
- It is assumed that each FortiGate is operating in NAT mode, running FortiOS 4.0MR2+.
- All interfaces have been previously assigned and no static routes are required.
- The AFI (Authority and Format Identifier) used is 49 : Locally administered (private).
- The Area identifiers are 0048 and 0049.
Expectations
- FGT4 must get the IS-IS route updates for RTR1 and RTR2 local subnets (10.1.1.0, 10.2.2.0, 10.3.3.0, 10.4.4.0).
- RTR1 must receive (via RIP2) the loopback subnet of FGT4 (10.60.60.1/32).
CLI configuration
The following CLI configuration occurs on each FortiGate (as identified), including only the relevant parts.
FGT1
config router isis config isis-interface
edit “port3”
set circuit-type level-1 set network-type broadcast set status enable
next end
config isis-net edit 1
set net 49.0048.1921.6818.2136.00 next
end
config redistribute “connected” end
config redistribute “rip” set status enable
set level level-1 end
config redistribute “ospf” set status enable
set level level-1 end
end
config router rip config interface
edit “port2”
set receive-version 2 set send-version 2
next end
config network edit 1
set prefix 10.10.10.0 255.255.255.0 next
end
config redistribute “isis” set status enable
end
end
FGT2
config router isis config isis-interface
edit “port3”
set circuit-type level-1 set network-type broadcast set status enable
next
edit “port2”
set network-type broadcast set status enable
next end
config isis-net edit 1
set net 49.0048.1221.6818.2110.00 next
end
set redistribute-l1 enable set redistribute-l2 enable
end
FGT3
config router isis
set is-type level-2-only config isis-interface
edit “wan1”
set network-type broadcast set status enable
next
edit “dmz1”
set network-type broadcast set status enable
next end
config isis-net edit 1
set net 49.0048.1921.6818.2108.00 next
edit 2
set net 49.0049.1921.6818.2108.00 next
end
end
FGT4
config router isis
set is-type level-2-only config isis-interface
edit “wan1”
set network-type broadcast set status enable
next end
config isis-net edit 1
set net 49.0049.1721.0160.1004.00 next
end
config redistribute “connected” set status enable
end
end
Verification
Once the network has been configured, you need to test that it works as expected. Use the following CLI commands on the devices indicated.
Verifying if RTR1 receives loopback subnet of FGT4
(RTR1) # get router info routing-table all
Result:
C 10.1.1.0/24 is directly connected, vlan1
C 10.2.2.0/24 is directly connected, vlan2
C 10.10.10.0/24 is directly connected, dmz1
R 10.40.40.0/24 [120/2] via 10.10.10.1, dmz1, 00:04:07
R 10.50.50.0/24 [120/2] via 10.10.10.1, dmz1, 00:04:07
R 10.60.60.1/32 [120/2] via 10.10.10.1, dmz1, 00:04:07
(*) If required, filtering out 10.50.50.0 and 10.40.40.0 from the routing table could be done with a route-map.
Verification on FGT2, which is the border between L1 and L2 routing levels; looking at IS-IS information
FGT2 # get router info isis interface
Result:
port2 is up, line protocol is up Routing Protocol: IS-IS ((null)) Network Type: Broadcast
Circuit Type: level-1-2
Local circuit ID: 0x01
Extended Local circuit ID: 0x00000003
Local SNPA: 0009.0f85.ad8c
IP interface address:
10.40.40.2/24
IPv4 interface address:
Level-1 Metric: 10/10, Priority: 64, Circuit ID: 1221.6818.2110.01
Number of active level-1 adjacencies: 0
Level-2 Metric: 10/10, Priority: 64, Circuit ID: 1221.6818.2110.01
Number of active level-2 adjacencies: 1
Next IS-IS LAN Level-1 Hello in 6 seconds Next IS-IS LAN Level-2 Hello in 1 seconds port3 is up, line protocol is up
Routing Protocol: IS-IS ((null)) Network Type: Broadcast
Circuit Type: level-1
Local circuit ID: 0x02
Extended Local circuit ID: 0x00000004
Local SNPA: 0009.0f85.ad8d
IP interface address:
10.30.30.2/24
IPv4 interface address:
Level-1 Metric: 10/10, Priority: 64, Circuit ID: 1221.6818.2110.02
Number of active level-1 adjacencies: 1
|
Next IS-IS LAN Level-1 Hello in 2 seconds FGT2 # get router info isis neighbor Result:
Verification on FGT3, which is border between 2 areas; looking at IS-IS information
IS-IS router CLI commands available:
FGT3 # get router info isis ?
Result:
interface show isis interfaces
neighbour show CLNS neighbor adjacencies is-neighbour show IS neighbor adjacencies database show IS-IS link state database route show IS-IS IP routing table topology show IS-IS paths
Example of interface status and neighbors:
FGT3 # get router info isis interface
Result:
wan1 is up, line protocol is up Routing Protocol: IS-IS ((null)) Network Type: Broadcast
Circuit Type: level-1-2
Local circuit ID: 0x01
Extended Local circuit ID: 0x00000003
Local SNPA: 0009.0f04.0794
IP interface address:
10.40.40.1/24
IPv4 interface address:
Level-2 Metric: 10/10, Priority: 64, Circuit ID: 1221.6818.2110.01
Number of active level-2 adjacencies: 1
Next IS-IS LAN Level-2 Hello in 3 seconds
dmz1 is up, line protocol is up Routing Protocol: IS-IS ((null)) Network Type: Broadcast
Circuit Type: level-1-2
Local circuit ID: 0x02
Extended Local circuit ID: 0x00000005
Local SNPA: 0009.0f04.0792
IP interface address:
10.50.50.1/24
IPv4 interface address:
Level-2 Metric: 10/10, Priority: 64, Circuit ID: 1721.0160.1004.01
Number of active level-2 adjacencies: 1
Next IS-IS LAN Level-2 Hello in 7 seconds
FGT3 # get router info isis neighbor
Result:
| System Id | Interface | SNPA | State | Holdtime | Type | Protocol |
|
1221.6818.2110 |
wan1 |
0009.0f85.ad8c |
Up |
8 |
L2 |
IS-IS |
|
1721.0160.1004 |
dmz1 |
0009.0f52.7704 |
Up |
8 |
L2 |
IS-IS |