Troubleshooting Virtual Domains
When you are configuring VDOMs you may run into some issues, with your VDOM configuration, your network configuration, or your device setup. This section addresses common problems and specific concerns that an administrator of a VDOM network may have.
This section includes:
- VDOM admin having problems gaining access
- FortiGate unit running very slowly
- General VDOM tips and troubleshooting
VDOM admin having problems gaining access
With VDOMs configured, administrators have an extra layer of permissions and may have problems accessing their information.
Confirm the admin’s VDOM
Each administrator account, other than the super_admin account, is tied to one specific VDOM. That administrator is not able to access any other VDOM. It may be possible they are trying to access the wrong VDOM.
Confirm the VDOM’s interfaces
An administrator can only access their VDOM through interfaces that are assigned to that VDOM. If interfaces on that VDOM are disabled or unavailable there will be no method of accessing that VDOM by its local administrator. The super_admin will be required to either bring up the interfaces, fix the interfaces, or move another interface to that VDOM to restore access.
Confirm the VDOMs admin access
As with all FortiGate units, administration access on the VDOM’s interfaces must be enabled for that VDOM’s administrators to gain access. For example if SSH is not enabled, that is not available to administrators.
To enable admin access, the super_admin will go to the Global > Network > Interfaces page, and for the interface in question enable the admin access.
FortiGate unit running very slowly
You may experience a number of problems resulting from your FortiGate unit being overloaded. These problems may appear as:
- CPU and memory threshold limits exceeded on a continual basis
- AV failopen happening on a regular basis
- dropped traffic or sessions due to lack of resources
These problems are caused by a lack of system resources. There are a number of possible reasons for this.
Too many VDOMs
If you have configured many VDOMs on your system, past the default ten VDOMs, this could easily be your problem.
Each VDOM you create on your FortiGate unit requires system resources to function – CPU cycles, memory, and disk space. When there are too many VDOMs configured there are not enough resources for operation. This may be a lack of memory in the session table, or no CPU cycles for processing incoming IPS traffic, or even a full disk drive.
Go to Global > System > VDOM and see the number of configured VDOMs on your system. If you are running 500 or more VDOMs, you must have a FortiGate 5000 chassis. Otherwise you need to reduce the number of VDOMs on your system to fix the problem. Even if you have the proper hardware, you may encounter noticeably slow throughput if you are using advanced features such as security profiles or deep content inspection with many configured VDOMs.
One or more VDOMs are consuming all the resources
If you have sufficient hardware to support the number of VDOMs you are running, check the global resources on your FortiGate unit. At a glance it will tell you if you are running out of a particular resource such as sessions, or users. If this is the case, you can then check your VDOMs to see if one particular VDOM is using more than its share of resources. If that is the case you can change the resource settings to allow that VDOM (or those VDOMs) fewer resources and in turn allow the other VDOMs access to those resources.
Too many Security Features in use
It is likely that reducing the Security Features in use regardless of number of VDOMs will greatly improve overall system performance and should be considered as an option.
Finally it is possible that your FortiGate unit configuration is incorrect in some other area, which is using up all your resources. For example, forgetting that you are running a network sniffer on an interface will create significant amounts of traffic that may prevent normal operation.
General VDOM tips and troubleshooting
Besides ping and traceroute, there are additional tools for troubleshooting your VDOM configurations. These include packet sniffing and debugging the packet flow.
Perform a sniffer trace
When troubleshooting networks, it helps to look inside the headers of packets to determine if they are traveling along the route you expect that they are. Packet sniffing can also be called a network tap, packet capture, or logic analyzing.
If your FortiGate unit has NP interfaces that are offloading traffic, this will change the sniffer trace. Before performing a trace on any NP interfaces, you should disable off- loading on those interfaces.
What sniffing packets can tell you
If you are running a constant traffic application such as ping, packet sniffing can tell you if the traffic is reaching the destination, what the port of entry is on the FortiGate unit, if the ARP resolution is correct, and if the traffic is being sent back to the source as expected.
Sniffing packets can also tell you if the Fortigate unit is silently dropping packets for reasons such as RPF (Reverse Path Forwarding), also called Anti Spoofing, which prevents an IP packet from being forwarded if its Source IP does not either belong to a locally attached subnet (local interface), or be part of the routing between the FortiGate and another source (static route, RIP, OSPF, BGP). Note that RPF can be disabled by turning on asymmetric routing in the CLI (config system setting, set asymmetric enable), however this will disable stateful inspection on the FortiGate unit and cause many features to be turned off.
If you configure virtual IP addresses on your Fortigate unit, it will use those addresses in preference to the physical IP addresses. You will notice this when you are sniffing packets because all the traffic will be using the virtual IP addresses. This is due to the ARP update that is sent out when the VIP address is configured.
How to sniff packets
When you are using VDOMs, you must be in a VDOM to access the diag sniffer command. At the global level, the command is not available. This is limit the packets only to the ones on your VDOM, and protects the privacy of other VDOM clients.
The general form of the internal FortiOS packet sniffer command is:
diag sniffer packet <interface_name> <‘filter’> <verbose> <count>
To stop the sniffer, type CTRL+C.
<interface_name> The name of the interface to sniff, such as “port1” or “internal”. This can also be “any” to sniff all interfaces.
<‘filter’>
What to look for in the information the sniffer reads. none indicates no fil- tering, and all packets will be displayed as the other arguments indicate.
The filter must be inside single quotes (‘).
<verbose> The level of verbosity as one of:
1 – print header of packets
2 – print header and data from IP of packets
3 – print header and data from Ethernet of packets
<count> The number of packets the sniffer reads before stopping. If you don’t put a number here, the sniffer will run forever unit you stop it with <CTRL C>.
For a simple sniffing example, enter the CLI command diag sniffer packet port1 none 1 3. This will display the next 3 packets on the port1 interface using no filtering, and using verbose level 1. At this verbosity level you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers.
In the output below, port 443 indicates these are HTTPS packets, and 172.20.120.17 is both sending and receiving traffic.
| Head_Office_620b # diag sniffer packet port1 none 1 | 3 | |||
| interfaces=[port1] | ||||
| filters=[none] | ||||
| 0.545306 172.20.120.17.52989 -> 172.20.120.141.443: | psh | 3177924955 | ack | 1854307757 |
|
0.545963 172.20.120.141.443 -> 172.20.120.17.52989: |
psh |
1854307757 |
ack |
3177925808 |
|
0.562409 172.20.120.17.52988 -> 172.20.120.141.443: |
psh |
4225311614 |
ack |
3314279933 |
For a more advanced example of packet sniffing, the following commands will report packets on any interface travelling between a computer with the host name of PC1 and the computer with the host name of PC2. With verbosity 4 and above, the sniffer trace will display the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C. Note that PC1 and PC2 may be VDOMs.
FGT# diagnose sniffer packet any “host <PC1> or host <PC2>” 4
or
FGT# diagnose sniffer packet any “(host <PC1> or host <PC2>) and icmp” 4
The following sniffer CLI command includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for instance PC2 may be down and not responding to the FortiGate ARP requests).
FGT# diagnose sniffer packet any “host <PC1> or host <PC2> or arp” 4
Debugging the packet flow
Traffic should come in and leave the VDOM. If you have determined that network traffic is not entering and leaving the VDOM as expected, debug the packet flow.
Debugging can only be performed using CLI commands. Debugging the packet flow requires a number of debug commands to be entered as each one configures part of the debug action, with the final command starting the debug.
If your FortiGate unit has NP interfaces that are offloading traffic, this will change the packet flow. Before performing the debug on any NP interfaces, you should disable off- loading on those interfaces.
The following configuration assumes that PC1 is connected to the internal interface of the FortiGate unit and has an IP address of 10.11.101.200. PC1 is the host name of the computer.
To debug the packet flow in the CLI, enter the following commands:
FGT# diag debug enable
FGT# diag debug flow filter add <PC1> FGT# diag debug flow show console enable FGT# diag debug flow trace start 100
FGT# diag debug enable
The start 100 argument in the above list of commands will limit the output to 100 packets from the flow. This is useful for looking at the flow without flooding your log or your display with too much information.
To stop all other debug activities, enter the command:
FGT# diag debug flow trace stop
The following is an example of debug flow output for traffic that has no matching Firewall Policy, and is in turn blocked by the FortiGate unit. The denied message indicates the traffic was blocked. Note that even with VDOMs not enabled, vd-root is still shown.
id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg=”vd-root received a packet(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3.”
id=20085 trace_id=319 func=resolve_ip_tuple line=2924 msg=”allocate a new session-013004ac”
id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg=”find a route: gw-192.168.150.129 via port1″
id=20085 trace_id=319 func=fw_forward_handler line=248 msg=” Denied by forward policy check”