Troubleshooting tools
FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup.
This section also contains information about troubleshooting FortiGuard issues. This section contains the following topics:
- FortiOS diagnostics
- FortiOS ports
- FortiAnalyzer/FortiManager ports
- FortiGuard troubleshooting
FortiOS diagnostics
A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring. Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands. Both commands display information about system resources, connections, and settings that enable you to locate and fix problems, or to monitor system performance.
This topic includes diagnostics commands to help with:
- Check date and time
- Resource usage
- Proxy operation
- Hardware NIC
- Traffic trace
- Session table
- Firewall session setup rate
- Finding object dependencies
- Flow trace
- Packet sniffing and packet capture
- NPU based interfaces
- Debug command
- The execute tac report command
- Other commands
Check date and time
The system date and time are important for FortiGuard services, when logging events, and when sending alerts. The wrong time will make the log entries confusing and difficult to use.
Use Network Time Protocol (NTP) to set the date and time if possible. This is an automatic method that does not require manual intervention. However, you must ensure the port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.
How to check the date and time – web-based manager
1. Go to System Information > System Time on the dashboard.
Alternately, you can check the date and time using the CLI commands execute date and execute time.
2. If required, select Change to adjust the date and time settings.
You can set the time zone, date and time, and select NTP usage. In the CLI, use the following commands to change the date and time:
config system global
set timezone (use ? to get a list of IDs and descriptions of their timezone)
end
config system ntp set type custom
config ntpserver edit 1
set server “ntp1.fortinet.net”
next edit 2
set server “ntp2.fortinet.net”
next
end
set ntpsync enable set syncinterval 60 end
Resource usage
Each program running on a computer has one or more processes associated with it. For example if you open a Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to share the system resources in FortiOS including memory and CPU.
Use get system performance status command to show the FortiOS performance status. Sample output:
FGT#get system performance status
CPU states: 0% user 0% system 0% nice 100% idle CPU0 states: 0% user 0% system 0% nice 100% idle CPU1 states: 0% user 0% system 0% nice 100% idle CPU2 states: 0% user 0% system 0% nice 100% idle CPU3 states: 0% user 0% system 0% nice 100% idle Memory states: 25% used
Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 0 kbps in 30 minutes
Average sessions: 5 sessions in 1 minute, 5 sessions in 10 minutes, 4 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 12 hours, 7 minutes
Monitor the CPU/memory usage of internal processes using the following command:
get system performance top <delay> <max_lines>
The data listed by the command includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU percentage being used, and the memory percentage being used.
Sample output:
FGT#get system performance top 10 100
Run Time: 0 days, 11 hours and 30 minutes
0U, 0S, 100I; 1977T, 1470F, 121KF
| pyfcgid | 120 | S | 0.0 | 1.3 | |
| pyfcgid | 121 | S | 0.0 | 1.3 | |
| pyfcgid | 122 | S | 0.0 | 1.3 | |
| pyfcgid | 53 | S | 0.0 | 1.3 | |
| ipsengine | 75 | S | < | 0.0 | 1.3 |
| ipsengine | 66 | S | < | 0.0 | 1.3 |
| ipsengine | 73 | S | < | 0.0 | 1.3 |
| ipsengine | 74 | S | < | 0.0 | 1.3 |
| ipsengine | 79 | S | < | 0.0 | 1.3 |
| ipsengine | 80 | S | < | 0.0 | 1.3 |
| cmdbsvr | 43 | S | 0.0 | 1.0 | |
| proxyworker | 110 | S | 0.0 | 1.0 | |
| proxyworker | 111 | S | 0.0 | 1.0 | |
| httpsd | 125 | S | 0.0 | 0.8 | |
| httpsd | 52 | S | 0.0 | 0.8 | |
| httpsd | 124 | S | 0.0 | 0.8 | |
| newcli | 141 | R | 0.0 | 0.7 | |
| newcli | 128 | S | 0.0 | 0.7 | |
| fgfmd | 102 | S | 0.0 | 0.7 | |
| iked | 86 | S | 0.0 | 0.7 |
Proxy operation
Monitor proxy operations using the following command:
diag test application <application> <option>
The <application> value can include the following:
http HTTP proxy.
smtp SMTP proxy.
ftpd FTP proxy.
pop3 POP3 proxy.
imap IMAP proxy.
nntp NNTP proxy.
proxyacceptor Proxy acceptor.
proxyworker Proxy worker.
scanunit Scanning unit.
sslacceptor SSL proxy.
sslworker SSL proxy.
ssh SSH proxy.
harelay HA relay daemon.
hasync HA sync daemon.
forticldd FortiCloud daemon.
miglogd Miglog logging daemon.
urlfilter URL filter daemon.
ovrd Override daemon.
ipsmonitor ips monitor
ipsengine ips sensor
ipldbd IP load balancing daemon.
ddnscd DDNS client daemon.
snmpd SNMP daemon.
acd Aggregate Controller.
dnsproxy DNS proxy.
sflowd sFlow daemon.
init init process.
l2tpcd L2TP client daemon.
dhcprelay DHCP relay daemon.
pptpcd PPTP client.
wccpd WCCP daemon.
wad WAD related processes.
radiusd RADIUS daemon.
sqldb SQL database daemon.
reportd Report daemon.
dlpfingerprint DLP fingerprint daemon.
dlpfpcache DLP fingerprint cache daemon.
wpad WPA daemon.
fsd FortiExplorer daemon.
ipsufd IPS urlfilter daemon.
stp Spanning Tree Protocol daemon.
lted USB LTE daemon.
swctrl_authd Switch controller authentication daemon.
forticron Forticron daemon.
uploadd Upload daemon.
quarantined Quarantine daemon.
dhcp6c DHCP6 client daemon.
info-sslvpnd SSL-VPN info daemon.
thmd Traffic history monitor daemon.
dsd DLP Statistics daemon.
lnkmtd Link monitor daemon.
dhcp6r DHCP6 relay daemon.
fnbamd Fortigate non-blocking auth daemon.
The <option> value depends from the application value used in the command. Here are some examples:
- If the application is http, the CLI command will be:
diag test application http <option>
The <option> value can be one from the following:
2 Drop all connections.
22 Drop max idle connections.
222 Drop all idle connections.
4 Display connection stat.
44 Display info per connection.
444 Display connections per state.
4444 Display per-VDOM statistics.
44444 Display information about idle connections.
55 Display tcp info per connection.
6 Display ICAP information.
70 Disable ICAP ‘Allow: 204’ (default).
71 Enable ICAP ‘Allow: 204’ .
72 Drop all ICAP server connections.
11 Display the SSL session ID cache statistics.
12 Clear the SSL session ID cache statistics.
13 Display the SSL session ID cache.
14 Clear the SSL session ID cache.
80 Show Fortinet bar SSL-VPN bookmark info.
81 Show Fortinet bar SSL-VPN bookmark cache.
82 Show Fortinet bar SSL-VPN bookmark LRU list.
- If the application is ipsmonitor, the CLI command will be
diag test application ipsmonitor <option>
The <option> value can be one from the following:
1 Display IPS engine information
2 Toggle IPS engine enable/disable status
3 Display restart log
4 Clear restart log
5 Toggle bypass status
6 Submit attack characteristics now
10 IPS queue length
11 Clear IPS queue length
12 IPS L7 socket statistics
13 IPS session list
14 IPS NTurbo statistics
15 IPSA statistics
97 Start all IPS engines
98 Stop all IPS engines
99 Restart all IPS engines and monitor
Hardware NIC
Monitor hardware network operations using the following command:
diag hardware deviceinfo nic <interface>
The information displayed by this command is important as errors at the interface are indicative of data link or physical layer issues which may impact the performance of the FortiGate unit.
The following is sample output when <interface> = internal:
System_Device_Name port5
Current_HWaddr 00:09:0f:68:35:60
Permanent_HWaddr 00:09:0f:68:35:60
Link up
Speed 100
Duplex full [……] Rx_Packets=5685708
Tx_Packets=4107073
Rx_Bytes=617908014
Tx_Bytes=1269751248
Rx_Errors=0
Tx_Errors=0
Rx_Dropped=0
Tx_Dropped=0 […..]
The diag hardware deviceinfo nic command displays a list of hardware related error names and values. The following table explains the items in the list and their meanings.
Possible hardware errors and meanings
Field Definition
Rx_Errors = rx error count Bad frame was marked as error by PHY.
Rx_CRC_Errors + Rx_Length_Errors – Rx_Align_Errors
This error is only valid in 10/100M mode.
Rx_Dropped or
Rx_No_Buffer_Count
Running out of buffer space.
Rx_Missed_Errors Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, whichis marked by PHY.
Tx_Errors = Tx_Aborted_ Errors
ECOL (Excessive Collisions Count). Only valid in half-duplex mode.
Tx_Window_Errors
LATECOL (Late Collisions Count). Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to100Mb/s data rate and 512-byte timeinto the transmission of the packet while working in the 1000Mb/s data rate. This register only increments if transmits are enabled and the device is in half-duplex mode.
Rx_Dropped See Rx_Errors.
Tx_Dropped Not defined.
Collisions Total number of collisions experienced by the transmitter. Valid in half- duplex mode.
Rx_Length_Errors Transmission length error.
Rx_Over_Errors Not defined.
Rx_CRC_Errors Frame CRC error.
Rx_Frame_Errors Same as Rx_Align_Errors. This error is only valid in 10/100M mode.
Field Definition
Rx_FIFO_Errors Same as Rx_Missed_Errors – a missed packet count.
Tx_Aborted_Errors See Tx_Errors.
Tx_Carrier_Errors
The PHY should assert the internal carrier sense signal during every trans- mission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if trans- mits are enabled. This register is not valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is only valid when the Ethernet controller is operating at full duplex.
Tx_FIFO_Errors Not defined.
Tx_Heartbeat_Errors Not defined.
Tx_Window_Errors See LATECOL.
Tx_Single_Collision_Frames
Counts the number of times that a successfully transmitted packed encountered a single collision. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode.
Tx_Multiple_Collision_Frames A Multiple Collision Count which counts the number of times that a trans- mit encountered more than one collision but less than 16. The value only increments if transmits are enabled and the Ethernet controller is in half- duplex mode.
Tx_Deferred
Counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer has not expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link is not up. This register only increments if transmits are enabled. This counter does not increment for streaming transmits that are deferred due to TX IPG.
Rx_Frame_Too_Longs The Rx frame is over size.
Rx_Frame_Too_Shorts The Rx frame is too short.
Rx_Align_Errors This error is only valid in 10/100M mode.
Symbol Error Count
Counts the number of symbol errors between reads – SYMERRS. The count increases for every bad symbol received, whether or not a packet is currently being received and whether or not the link is up. This register only increments in internal SerDes mode.
Traffic trace
Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the route you expected on your network.
View the characteristics of a traffic session though specific security policies using:
diag sys session
Trace per-packet operations for flow tracing using:
diag debug flow
Trace per-Ethernet frame using:
diag sniffer packet
Session table
A session is a communication channel between two devices or applications across the network. Sessions enable FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.
Use as a tool
Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening.
The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful summary information, particularly the current policy number that the session is using.
Web–based manager session information
In the web-based manager you can view session information in the FortiView page. Sessions are categorized by Sources, Applications, Destinations, and All Sessions.
How to find which security policy a specific connection is using
Every program and device on your network must have a communication channel, or session, open to pass information. The FortiGate unit manages these sessions with its many features from traffic shaping, to antivirus scanning, and even blocking known bad web sites. Each session has an entry in the session table.
You may want to find information for a specific session, say a secure web browser session, for troubleshooting. For example if that web browser session is not working properly, you can check the session table to ensure the session is still active, and that it is going to the proper address. It can also tell you the security policy number it matches, so you can check what is happening in that policy.
1. Know your connection information.
You need to be able to identify the session you want. For this you need the source IP address (usually your computer), the destination IP address if you have it, and the port number which is determined by the program being used. Some commons ports are:
- port 80 (HTTP for web browsing),
- port 22 (SSH used for secure login and file transfers)
- port 23 (telnet for a text connection)
- port 443 (HTTPS for secure web browsing
2. Find your session and policy ID.
Follow System > FortiView> All Sessions. Find your session by finding your source IP address, destination IP address if you have it, and port number. The policy ID is listed after the destination information. If the list of sessions is very long, you can filter the list to make it easier to find your session.
3. When there are many sessions, use a filter to help you find your session.
If there are multiple pages of sessions it is difficult to find a single session. To help you in your search you can use a filter to block out sessions that you don’t want. Click the search icon on the column heading to select the filter. Select Source IP and enter your source IP address. Now only sessions that originate from your IP address will be displayed in the session table. If the list is still too long, you can do the same for the Source port. That will make it easy to find your session and the security policy ID. When you are finished remember to clear the filters.
CLI session information
The session table output from the CLI (diag sys session list) is very verbose. Even on a system with a small amount of traffic, displaying the session table will generate a large amount of output. For this reason, filters are used to display only the session data of interest.
You can filter a column in the web-based manager by clicking the search icon on the column heading or from the CLI by creating a filter.
An entry is placed in the session table for each traffic session passing through a security policy. The following command will list the information for a session in the table:
diag sys session list
Sample Output:
FGT# diag sys session list
session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3 bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0
logtype=session ha_id=0 hakey=4450
tunnel=/
state=log shape may_dirty
statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2
orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100 hook=post dir=org act=snat 10.0.5.100:1251->192.168.11.254:22(192.168.11.105:1251) hook=pre dir=reply act=dnat 192.168.11.254:22->192.168.11.105:1251(10.0.5.100:1251) pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff
Since output can be verbose, the filter option allows specific information to be displayed, for example:
diag sys session filter <option>
The <option> values available include the following:
clear Clear session filter.
dintf Destination interface.
dport Destination port.
dst Destination IP address.
duration duration
expire expire
negate Inverse filter.
nport NAT’d source port
nsrc NAT’d source ip address
policy Policy ID.
proto Protocol number.
proto-state Protocol state.
sintf Source interface.
sport Source port.
src Source IP address.
vd Index of virtual domain. -1 matches all.
Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different states:
- UDP reply not seen with a value of 0
- UDP reply seen with a value of 1
The following illustrates FW session states from the session table:
State Meaning
log Session is being logged.
local Session is originated from or destined for local stack.
ext Session is created by a firewall session helper.
State Meaning
may_dirty
Session is created by a policy. For example, the session for ftp control chan- nel will have this state but ftp data channel will not. This is also seen when NAT is enabled.
ndr Session will be checked by IPS signature.
nds Session will be checked by IPS anomaly.
br Session is being bridged (TP) mode.
Firewall session setup rate
The number of sessions that can be established in a set period of time is useful information. A session is an end- to-end TCP/IP connection for communication with a limited lifespan. If you record the setup rate during normal operation, when you experience problems you have that setup rate with the current number to see if its very different. While this will not solve your problems, it can be a useful step to help you define your problem.
A reduced firewall session setup rate could be the result of a number of things from a lack of system resources on the FortiGate unit, to reaching the limit of your session count for your VDOM.
To view your session setup rate method 1- CLI
FGT# get sys performance status
CPU states: 0% user 0% system 0% nice 100% idle
Memory states: 10% used
Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes,
13 kbps in 30 minutes
Average sessions: 31 sessions in 1 minute, 30 sessions in 10 minutes, 31 sessions in 30 minutes
Average session setup rate: 0.5 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 44 days, 18 hours, 42 minutes
The information you are looking for is the Average sessions section, highlighted in the above output. In this example you can see there were 31 sessions in 1 minute, or an average of 0.5 sessions per second. The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate unit is working at maximum capacity. The smallest FortiGate unit can have 1 000 sessions established per second across the unit.
Remember that session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each one, the session setup rate per VDOM will be slower than if there were no VDOMs configured.
Finding object dependencies
An administrator may not be permitted to delete a configuration object if there are other configuration objects that depend on it. This command identifies other objects which depend on or make reference to the configuration object in question. If an error is displayed that an object is in use and cannot be deleted, this command can help identify the source of the problem.
Another use is if you have a virtual interface with objects that depend on it, you need to find and remove those dependencies before you delete that interface.
CLI method
When running multiple VDOMs, this command is run in the Global configuration only and it searches for the named object both in the Global and VDOM configuration most recently used:
diag sys checkused <path.object.mkey>
For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as follows:
diag sys checkused firewall.policy.policyid 1
To check what is referred to by interface port1, enter the following command:
diag sys checkused system.interface.name port1
To show all the dependencies for an interface, enter the command as follows:
diag sys checkused system.interface.name <interface name>
Sample Output:
entry used by table firewall.address:name ‘10.98.23.23_host’
entry used by table firewall.address:name ‘NAS’ entry used by table firewall.address:name ‘all’
entry used by table firewall.address:name ‘fortinet.com’
entry used by table firewall.vip:name ‘TORRENT_10.0.0.70:6883′ entry used by table firewall.policy:policyid ’21’
entry used by table firewall.policy:policyid ’14’ entry used by table firewall.policy:policyid ’19’
In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.
Web–based manager method
In the web-based manager, the object dependencies for an interface can be easily checked and removed.
To remove interface object dependencies – web-based manager
1. Go to System > Interfaces.
The number in the Ref. column is the number of objects that refer to this interface.
2. Select the number in the Ref. column for the desired interface.
A Window listing the dependencies will appear.
3. Use these detailed entries to locate and remove object references to this interface.
The trash can icon will change from gray when all object dependencies have been removed.
4. Remove the interface by selecting the check box for the interface, and select Delete.
Flow trace
To trace the flow of packets through the FortiGate unit, use the following command:
diag debug flow trace start
If your network is using IPv4, follow packet flow by setting a flow filter using this command:
diag debug flow filter <option>
Filtering options include the following:
addr IPv4 address
clear clear filter
daddr destination IPv4 address
dport destination port negate inverse IPv4 filter port port
proto protocol number saddr source IPv4 address sport source port
vd index of virtual domain, -1 matches all
If your network is using IPv6, follow packet flow by setting a flow filter using this command:
diag debug flow filter6 <option>
Filtering options include the following:
addr IPv6 address
clear clear filter
daddr destination IPv6 address
dport destination port negate inverse IPv6 filter port port
proto protocol number saddr source IPv6 address sport source port
vd index of virtual domain, -1 matches all
Enable the output to be displayed to the CLI console using the following command:
diag debug flow show console enable
diag debug flow output is recorded as event log messages and are sent to a FortiCloud or a FortiAnalyzer unit if connected. Do not let this command run longer than necessary since it generates significant amounts of data.
Start flow monitoring with a specific number of packets using this command:
diag debug flow trace start <N>
Stop flow tracing at any time using:
diag debug flow trace stop
The following is an example of the flow trace for the device at the following IP address: 203.160.224.97
diag debug enable
diag debug flow filter addr 203.160.224.97 diag debug flow show console enable
diag debug flow show function-name enable diag debug flow trace start 100
Flow trace output example – HTTP
Connect to the web site at the following address to observe the debug flow trace. The display may vary slightly:
Comment: SYN packet received:
id=20085 trace_id=209 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,
192.168.3.221:1487->203.160.224.97:80) from port5.”
SYN sent and a new session is allocated:
id=20085 trace_id=209 func=resolve_ip_tuple line=2799 msg=”allocate a new session-00000e90″
Lookup for next-hop gateway address:
id=20085 trace_id=209 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.11.254 via port6″
Source NAT, lookup next available port:
id=20085 trace_id=209 func=get_new_addr line=1219 msg=”find SNAT: IP-192.168.11.59, port-31925″ direction“
Matched security policy. Check to see which policy this session matches:
id=20085 trace_id=209 func=fw_forward_handler line=317 msg=”Allowed by Policy-3: SNAT”
Apply source NAT:
id=20085 trace_id=209 func= ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″
SYN ACK received:
id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 203.160.224.97:80-
>192.168.11.59:31925) from port6.”
Found existing session ID. Identified as the reply direction:
id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”
Apply destination NAT to inverse source NAT action:
id=20085 trace_id=210 func= ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925->192.168.3.221:1487″
Lookup for next-hop gateway address for reply traffic:
id=20085 trace_id=210 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.3.221 via port5″
ACK received:
id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,192.168.3.221:1487->203.160.224.97:80) from port5.”
Match existing session in the original direction:
id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”
Apply source NAT:
id=20085 trace_id=211 func= ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″
Receive data from client:
id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 192.168.3.221:1487->203.160.224.97:80) from port5.”
Match existing session in the original direction:
id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”
Apply source NAT:
id=20085 trace_id=212 func= ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″
Receive data from server:
id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,
203.160.224.97:80->192.168.11.59:31925) from port6.”
Match existing session in reply direction:
id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”
Apply destination NAT to inverse source NAT action:
id=20085 trace_id=213 func= ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925-
>192.168.3.221:1487″
Flow trace output example – IPsec (policy-based)
id=20085 trace_id=1 msg=”vd-root received a packet(proto=1, 10.72.55.240:1->10.71.55.10:8)
from internal.”
id=20085 trace_id=1 msg=”allocate a new session-00001cd3″
id=20085 trace_id=1 msg=”find a route: gw-66.236.56.230 via wan1″ id=20085 trace_id=1 msg=”Allowed by Policy-2: encrypt”
id=20085 trace_id=1 msg=”enter IPsec tunnel-RemotePhase1″
id=20085 trace_id=1 msg=”encrypted, and send to 15.215.225.22 with source 66.236.56.226″ id=20085 trace_id=1 msg=”send to 66.236.56.230 via intf-wan1“
id=20085 trace_id=2 msg=”vd-root received a packet (proto=1, 10.72.55.240:1-1071.55.10:8)
from internal.”
id=20085 trace_id=2 msg=”Find an existing session, id-00001cd3, original direction” id=20085 trace_id=2 msg=”enter IPsec =”encrypted, and send to 15.215.225.22 with source
66.236.56.226“ tunnel-RemotePhase1″
id=20085 trace_id=2 msgid=20085 trace_id=2 msg=”send to 66.236.56.230 via intf-wan1″
Packet sniffing and packet capture
FortiOS devices can sniff packets using commands in the CLI or capture packets using the web-based manager. The differences between the two methods are not large.
Packet sniffing in the CLI is well suited for spot checking traffic from the CLI, but if you have complex filters to enter it can be a lot of work to enter them each time. You can also save the sniffing output; however, you must log to a file and then analyze the file later by hand.
Packet capture in the web-based manager makes it easy to set up multiple filters at once and just run one or two as you need them. You also have controls to start and stop capturing as you wish. Packet capture output is downloaded to your local computer as a *.pcap file which requires a third party application to read the file, such as Wireshark. This method is useful to send Fortinet support information to help resolve an issue.
Features Packet sniffing Packet capture
Command location CLI web-based manager
Third party software required puTTY to log plaintext output Wireshark to read *.pcap files
Read output in plain text file yes no
Read output as *.pcap file using Wire-
shark no yes
Easily configure single quick and simple filter
yes no
Record packet interface yes no
Features Packet sniffing Packet capture
Configure complex sniffer filters on mul- tiple interface
no yes
sniff IPv6 hard easy
sniff non-IP packets no yes
Filter packets by protocol and/or port easy easy
Filter packets by source and/or des- tination address
easy easy
Packet sniffing
Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file — there can be huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate unit’s CLI. Then once the packet sniffing count is reached you can end the session and analyze the output in the file.
Details within packets passing through particular interfaces can be displayed using the packet sniffer with the following command:
diag sniffer packet <interface> <filter> <verbose> <count> <tsformat>
The <interface> value is required, with the rest being optional. If not included the default values will be “none”.
For example the simplest valid sniffer command would be:
diag sniffer packet any
The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all interfaces.
The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax. The <filter> value must be enclosed in quotes.
‘[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]]‘
If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a port, there are up to two parts — protocol and port number.
For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following:
‘udp port 1812 or tcp port 8080’
To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3:
‘ip src host 192.168.1.2 and dst host 192.168.2.3’
The <verbose> option allows different levels of information to be displayed. The verbose levels include:
1 Print header of packets
2 Print header and data from the IP header of the packets
3 Print header and data from the Ethernet header of the packets
4 Print header of packets with interface name
5 Print header and data from ip of packets with interface name
6 Print header and data from ethernet of packets with interface name
The <count> value indicates the number of packets to sniff before stopping. If this variable is not included, or is set to zero, the sniffer will run until you manually halt it with Ctrl-C.
The <tsformat> value define the format of timestamp. It can be:
a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms
l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms otherwise: relative to the start of sniffing, ss.ms
Packet capture
FortiOS 5.2 includes packet capture to the web-based manager. The FortiGate unit must have a disk and then capture-packet feature can be enabled from the CLI within the firewall policy as below:
config firewall policy edit <id>
set capture-packet enable end
To configure packet capture filters, go to System > Network > Packet Capture. When you add a packet capture filter, enter the following information and select OK.
Interface Select the interface to sniff from the dropdown menu.
You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields.
Max Packets to Capture
Enter the number of packets to capture before the filter stops.
This number cannot be zero. You can halt the capturing before this number is reached.
Enable Filters Select this option to specify your filter fields
Host(s)
Enter one or more hosts IP address
Separate multiple hosts with commas. Enter a range using a dash without spaces, for example 172.16.1.5-172.16.1.15 or enter a subnet.
Port(s) Enter one or more ports to capture on the selected interface.
Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90
VLAN(s)
Enter one or more vlans (if there is any).
Separate multiple vlans with commas.
Protocol Enter one or more protocol. Separate multiple protocol with commas. Enter a range using a dash without spaces, for example 1-6, 17, 21-25
Include IPv6 packets Select this option if you are troubleshooting IPv6 networking, or if your net- work uses IPv6. Otherwise, leave it disabled.
Capture Non-IP packets The protocols available in the list are all IP based except for ICMP (ping).
To capture non-IP based packets select this feature. Some examples of non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.
If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in the edit window, or downloading the captured packets. You can also see the filter status and the number of packets captured.
You can also select the filter and select Start to start capturing packets. While the filter is running, you will see the number of captured packets increasing until it reaches the max packet count or you select Stop. While the filter is running you cannot download the output file.
When the packet capture is complete, you can select Download to send the packet capture filter captured packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark or a similar third party application. Using this tool you will have extensive analytics available to you and the full contents of the packets that were captured.
NPU based interfaces
Many Fortinet products contain network processors such as NP1, NP2, and NP4 network processors. Therefore offloading requirements, vary by network processor model.
When using the NPU-based interfaces, only the initial session setup will be seen through the diag debug flow command. If the session is correctly programmed into the ASIC (fastpath), the debug flow command will no longer see the packets arriving at the CPU. If the NPU functionality is disabled, the CPU will see all the packets, however, this should only be used for troubleshooting purposes.
First, obtain the NP4 id and the port numbers with the following command:
diag npu np4 list
Sample output:
ID Model Slot Interface
0 On-board port1 fabric1 fabric3 fabric5
1 On-board fabric2 port2 base2 fabric4
Run the following commands:
diag npu np4 fastpaf th disable <dev_id>
(where dev_id is the NP4 number) Then, run this command:
diag npu np4 fastpath-sniffer enable port1
Sample output:
NP4 Fast Path Sniffer on port1 enabled
This will cause all traffic on port1 of NP4 to be sent to the CPU meaning a standard sniffer trace can be taken and other diag commands should work if it was a standard CPU driven port.
These commands are only for the newer NP4 interfaces.
Debug command
Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually generated even though output might not be displayed in the CLI console.
Debug information displayed in the console will scroll in the console display and may prevent CLI commands from being entered, for example, the command to disable the debug display. To turn off debugging output as the display is scrolling by, press the á key to recall the recent diag debug command, press backspace, and type “0”, followed by Enter.
Debug output display is enabled using the following command:
diag debug enable
When finished examining the debug output, disable it using:
diag debug disable
Once enabled, indicate the debug information that is required using this command:
diag debug <option> <level>
Debug command options include the following:
enable Enable debug output.
disable Disable debug output.
info Show active debug level settings.
reset Reset all debug level to default.
report Report for tech support.
crashlog Crash log info.
config-error-log Configure error log info.
sql-log-error SQL log database error info.
application application.
kernel kernel.
remote-extender remote-extender.
console console.
cli Debug CLI.
cmdb-trace Trace CLI.
rating Display rating info.
authd Authentication daemon.
fsso-polling FSSO active directory poll module.
flow Trace packet flow in kernel.
urlfilter urlfilter.
admin Admin user.
The debug level can be set at the end of the command. Typical values are 2 and 3, for example:
diag debug application DHCPS 2
diag debug application spamfilter 2
Fortinet support will advise which debugging level to use.
Timestamps can be enabled to the debug output using the following command:
diag debug console timestamp enable
Debug output example
This example shows the IKE negotiation for a secure logging connection from a FortiGate unit to a FortiAnalyzer system.
diag debug reset
diag vpn ike log-filter src-addr4 192.168.11.2 diag debug enable
Sample Output:
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, natt_mode=0 rekey=0 phase2=FGh_FtiLog1
FGh_FtiLog1: using existing connection, dpd_fail=0
FGh_FtiLog1: found phase2 FGh_FtiLog1
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating
FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2
FGh_FtiLog1: initiator quick-mode set pfs=1536…
FGh_FtiLog1: try to negotiate with 1800 life seconds.
FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0
Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348
Initiator: sent 192.168.10.201 quick mode message #1 (OK) FGh_FtiLog1: set retransmit: st=168, timeout=6.
In this example:
192.168.11.2->192.168.10.201:500 Source and Destination gateway IP address
dpd_fail=0 Found existing Phase 1
pfs=1536… Create new Phase 2 tunnel
The execute tac report command
exec tac report is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you are using certain features like HA, VPN tunnels, or a modem. The report takes a few minutes to complete due to the amount of output generated. If you have your CLI output logged to a file, you can run this command to familiarize yourself with the CLI commands involved.
When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current state using the output from this CLI command.
Other commands
ARP table
To view the ARP cache, use the following command:
get sys arp
To view the ARP cache in the system, use this command:
diag ip arp list
Sample output:
index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203 confirm=78203 update=72203 ref=1
index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179 ref=2 ? VIP
index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743 update=75743 ref=1
index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532 confirm=10532 update=12658 ref=4
To remove the ARP cache, use this command:
execute clear system arp table
To remove a single ARP entry, use:
diag ip arp delete <interface name> <IP address>
To add static ARP entries, use the following command:
config system arp-table
Time and date settings
Check time and date settings for log message timestamp synchronization (the Fortinet support group may request this) and for certificates that have a time requirement to check for validity. Use the following commands:
execute time
current time is: 12:40:48
last ntp sync:Thu Mar 16 12:00:21 2006 execute date
current date is: 2006-03-16
To force synchronization with an NTP server, toggle the following command:
Config system ntp
set ntpsync enable/disable end
If all devices have the same time, it helps to correlate log entries from different devices.
IP address
There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what you expect them to be. This is easily accomplished from the CLI using the following command.
diag ip address list
The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID number) and the devname is the name of the interface. While physical interface names are set, virtual interface names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_ fgfm, the IP addresses are the local host — these are internally used virtual interfaces.
# diag ip address list
IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal
IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm