Troubleshooting tools

FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup.

 

This section also contains information about troubleshooting FortiGuard issues. This section contains the following topics:

• FortiOS diagnostics
• FortiOS ports
• FortiAnalyzer/FortiManager ports
• FortiGuard troubleshooting

 

FortiOS diagnostics

A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring. Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands. Both commands display information about system resources, connections, and settings that enable you to locate and fix problems, or to monitor system performance.

 

This topic includes diagnostics commands to help with:

• Check date and time
• Resource usage
• Proxy operation
• Hardware NIC
• Traffic trace
• Session table
• Firewall session setup rate
• Finding object dependencies
• Flow trace
• Packet sniffing and packet capture
• NPU based interfaces
• Debug command
• The execute tac report command
• Other commands

 

Check date and time

The system date and time are important for FortiGuard services, when logging events, and when sending alerts. The wrong time will make the log entries confusing and difficult to use.

Use Network Time Protocol (NTP) to set the date and time if possible. This is an automatic method that does not require manual intervention. However, you must ensure the port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.

 

How to check the date and time – web-based manager

1. Go to System Information > System Time on the dashboard.

Alternately, you can check the date and time using the CLI commands execute date and execute time.

2. If required, select Change to adjust the date and time settings.

You can set the time zone, date and time, and select NTP usage. In the CLI, use the following commands to change the date and time:

config system global

set timezone (use ? to get a list of IDs and descriptions of their timezone)

end

config system ntp set type custom

config ntpserver edit 1

set server “ntp1.fortinet.net”

next edit 2

set server “ntp2.fortinet.net”

next

end

set ntpsync enable set syncinterval 60 end

 

Resource usage

Each program running on a computer has one or more processes associated with it. For example if you open a Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to share the system resources in FortiOS including memory and CPU.

Use get system performance status command to show the FortiOS performance status. Sample output:

FGT#get system performance status

CPU states: 0% user 0% system 0% nice 100% idle CPU0 states: 0% user 0% system 0% nice 100% idle CPU1 states: 0% user 0% system 0% nice 100% idle CPU2 states: 0% user 0% system 0% nice 100% idle CPU3 states: 0% user 0% system 0% nice 100% idle Memory states: 25% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 0 kbps in 30 minutes

Average sessions: 5 sessions in 1 minute, 5 sessions in 10 minutes, 4 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 12 hours, 7 minutes

 

Monitor the CPU/memory usage of internal processes using the following command:

 

get system performance top <delay> <max_lines>

 

The data listed by the command includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU percentage being used, and the memory percentage being used.

 

Sample output:

FGT#get system performance top 10 100

Run Time: 0 days, 11 hours and 30 minutes

0U, 0S, 100I; 1977T, 1470F, 121KF

 

pyfcgid	120	S		0.0	1.3
pyfcgid	121	S		0.0	1.3
pyfcgid	122	S		0.0	1.3
pyfcgid	53	S		0.0	1.3
ipsengine	75	S	<	0.0	1.3
ipsengine	66	S	<	0.0	1.3
ipsengine	73	S	<	0.0	1.3
ipsengine	74	S	<	0.0	1.3
ipsengine	79	S	<	0.0	1.3
ipsengine	80	S	<	0.0	1.3
cmdbsvr	43	S		0.0	1.0
proxyworker	110	S		0.0	1.0
proxyworker	111	S		0.0	1.0
httpsd	125	S		0.0	0.8
httpsd	52	S		0.0	0.8
httpsd	124	S		0.0	0.8
newcli	141	R		0.0	0.7
newcli	128	S		0.0	0.7
fgfmd	102	S		0.0	0.7
iked	86	S		0.0	0.7

 

Proxy operation

Monitor proxy operations using the following command:

diag test application <application> <option>

The <application> value can include the following:

 

http                    HTTP proxy.

 

smtp                    SMTP proxy.

 

ftpd                    FTP proxy.

 

pop3                    POP3 proxy.

 

imap                    IMAP proxy.

 

nntp                    NNTP proxy.

 

proxyacceptor          Proxy acceptor.

 

proxyworker            Proxy worker.

 

scanunit               Scanning unit.

 

sslacceptor            SSL proxy.

 

sslworker              SSL proxy.

 

ssh                     SSH proxy.

 

harelay                 HA relay daemon.

 

hasync                  HA sync daemon.

 

forticldd              FortiCloud daemon.

 

miglogd                 Miglog logging daemon.

 

urlfilter              URL filter daemon.

 

ovrd                    Override daemon.

 

ipsmonitor             ips monitor

 

ipsengine              ips sensor

 

ipldbd                  IP load balancing daemon.

 

ddnscd                  DDNS client daemon.

 

snmpd                   SNMP daemon.

 

acd                     Aggregate Controller.

 

dnsproxy               DNS proxy.

 

sflowd                  sFlow daemon.

 

init                    init process.

 

l2tpcd                  L2TP client daemon.

 

dhcprelay              DHCP relay daemon.

 

pptpcd                  PPTP client.

 

wccpd                   WCCP daemon.

 

wad                     WAD related processes.

 

radiusd                 RADIUS daemon.

 

sqldb                   SQL database daemon.

 

reportd                 Report daemon.

 

dlpfingerprint         DLP fingerprint daemon.

 

dlpfpcache             DLP fingerprint cache daemon.

 

wpad                    WPA daemon.

 

fsd                     FortiExplorer daemon.

 

ipsufd                  IPS urlfilter daemon.

 

stp                     Spanning Tree Protocol daemon.

 

lted                    USB LTE daemon.

 

swctrl_authd           Switch controller authentication daemon.

 

forticron              Forticron daemon.

 

uploadd                 Upload daemon.

 

quarantined            Quarantine daemon.

 

dhcp6c                  DHCP6 client daemon.

 

info-sslvpnd           SSL-VPN info daemon.

 

thmd                    Traffic history monitor daemon.

 

dsd                     DLP Statistics daemon.

 

lnkmtd                  Link monitor daemon.

 

dhcp6r                  DHCP6 relay daemon.

 

fnbamd                  Fortigate non-blocking auth daemon.

 

 

The <option> value depends from the application value used in the command. Here are some examples:

• If the application is http, the CLI command will be:

diag test application http <option>

 

The <option> value can be one from the following:

2                       Drop all connections.

22                      Drop max idle connections.

222                     Drop all idle connections.

4                       Display connection stat.

44                      Display info per connection.

444                     Display connections per state.

4444                    Display per-VDOM statistics.

44444                   Display information about idle connections.

55                      Display tcp info per connection.

6                       Display ICAP information.

70                      Disable ICAP ‘Allow: 204’ (default).

71                      Enable ICAP ‘Allow: 204’ .

72                      Drop all ICAP server connections.

11                      Display the SSL session ID cache statistics.

12                      Clear the SSL session ID cache statistics.

13                      Display the SSL session ID cache.

14                      Clear the SSL session ID cache.

80                      Show Fortinet bar SSL-VPN bookmark info.

81                      Show Fortinet bar SSL-VPN bookmark cache.

82                      Show Fortinet bar SSL-VPN bookmark LRU list.

• If the application is ipsmonitor, the CLI command will be

diag test application ipsmonitor <option>

The <option> value can be one from the following:

1                       Display IPS engine information

2                       Toggle IPS engine enable/disable status

3                       Display restart log

4                       Clear restart log

5                       Toggle bypass status

6                       Submit attack characteristics now

10                      IPS queue length

11                      Clear IPS queue length

12                      IPS L7 socket statistics

13                      IPS session list

14                      IPS NTurbo statistics

15                      IPSA statistics

97                      Start all IPS engines

98                      Stop all IPS engines

99                      Restart all IPS engines and monitor

 

Hardware NIC

Monitor hardware network operations using the following command:

diag hardware deviceinfo nic <interface>

The information displayed by this command is important as errors at the interface are indicative of data link or physical layer issues which may impact the performance of the FortiGate unit.

The following is sample output when <interface> = internal:

System_Device_Name port5

Current_HWaddr 00:09:0f:68:35:60

Permanent_HWaddr 00:09:0f:68:35:60

Link up

Speed 100

Duplex full [……] Rx_Packets=5685708

Tx_Packets=4107073

Rx_Bytes=617908014

Tx_Bytes=1269751248

Rx_Errors=0

Tx_Errors=0

Rx_Dropped=0

Tx_Dropped=0 […..]

 

The diag hardware deviceinfo nic command displays a list of hardware related error names and values. The following table explains the items in the list and their meanings.

 

Possible hardware errors and meanings

Field                                         Definition

Rx_Errors = rx error count            Bad frame was marked as error by PHY.

Rx_CRC_Errors + Rx_Length_Errors – Rx_Align_Errors

This error is only valid in 10/100M mode.

Rx_Dropped or

Rx_No_Buffer_Count

Running out of buffer space.

Rx_Missed_Errors                       Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, whichis marked by PHY.

Tx_Errors = Tx_Aborted_ Errors

ECOL (Excessive Collisions Count). Only valid in half-duplex mode.

Tx_Window_Errors

LATECOL (Late Collisions Count). Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to100Mb/s data rate and 512-byte timeinto the transmission of the packet while working in the 1000Mb/s data rate. This register only increments if transmits are enabled and the device is in half-duplex mode.

Rx_Dropped                                See Rx_Errors.

 

Tx_Dropped                                 Not defined.

 

Collisions                                     Total number of collisions experienced by the transmitter. Valid in half- duplex mode.

 

Rx_Length_Errors                       Transmission length error.

 

Rx_Over_Errors                           Not defined.

 

Rx_CRC_Errors                           Frame CRC error.

 

Rx_Frame_Errors                        Same as Rx_Align_Errors. This error is only valid in 10/100M mode.

 

Field                                         Definition

Rx_FIFO_Errors                          Same as Rx_Missed_Errors – a missed packet count.

 

Tx_Aborted_Errors                      See Tx_Errors.

 

Tx_Carrier_Errors

The PHY should assert the internal carrier sense signal during every trans- mission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if trans- mits are enabled. This register is not valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is only valid when the Ethernet controller is operating at full duplex.

 

Tx_FIFO_Errors                          Not defined.

 

Tx_Heartbeat_Errors                   Not defined.

 

Tx_Window_Errors                      See LATECOL.

 

Tx_Single_Collision_Frames

Counts the number of times that a successfully transmitted packed encountered a single collision. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode.

 

Tx_Multiple_Collision_Frames   A Multiple Collision Count which counts the number of times that a trans- mit encountered more than one collision but less than 16. The value only increments if transmits are enabled and the Ethernet controller is in half- duplex mode.

Tx_Deferred

Counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer has not expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link is not up. This register only increments if transmits are enabled. This counter does not increment for streaming transmits that are deferred due to TX IPG.

 

Rx_Frame_Too_Longs               The Rx frame is over size.

 

Rx_Frame_Too_Shorts               The Rx frame is too short.

 

Rx_Align_Errors                          This error is only valid in 10/100M mode.

 

 

Symbol Error Count

Counts the number of symbol errors between reads – SYMERRS. The count increases for every bad symbol received, whether or not a packet is currently being received and whether or not the link is up. This register only increments in internal SerDes mode.

 

Traffic trace

Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the route you expected on your network.

View the characteristics of a traffic session though specific security policies using:

diag sys session

 

Trace per-packet operations for flow tracing using:

diag debug flow

 

Trace per-Ethernet frame using:

diag sniffer packet

 

Session table

A session is a communication channel between two devices or applications across the network. Sessions enable FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.

 

Use as a tool

Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening.

The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful summary information, particularly the current policy number that the session is using.

 

Web–based manager session information

In the web-based manager you can view session information in the FortiView page. Sessions are categorized by Sources, Applications, Destinations, and All Sessions.

 

How to find which security policy a specific connection is using

Every program and device on your network must have a communication channel, or session, open to pass information. The FortiGate unit manages these sessions with its many features from traffic shaping, to antivirus scanning, and even blocking known bad web sites. Each session has an entry in the session table.

You may want to find information for a specific session, say a secure web browser session, for troubleshooting. For example if that web browser session is not working properly, you can check the session table to ensure the session is still active, and that it is going to the proper address. It can also tell you the security policy number it matches, so you can check what is happening in that policy.

 

1. Know your connection information.

You need to be able to identify the session you want. For this you need the source IP address (usually your computer), the destination IP address if you have it, and the port number which is determined by the program being used. Some commons ports are:

 

• port 80 (HTTP for web browsing),
• port 22 (SSH used for secure login and file transfers)
• port 23 (telnet for a text connection)
• port 443 (HTTPS for secure web browsing

2. Find your session and policy ID.

Follow System > FortiView> All Sessions. Find your session by finding your source IP address, destination IP address if you have it, and port number. The policy ID is listed after the destination information. If the list of sessions is very long, you can filter the list to make it easier to find your session.

3. When there are many sessions, use a filter to help you find your session.

If there are multiple pages of sessions it is difficult to find a single session. To help you in your search you can use a filter to block out sessions that you don’t want. Click the search icon on the column heading to select the filter. Select Source IP and enter your source IP address. Now only sessions that originate from your IP address will be displayed in the session table. If the list is still too long, you can do the same for the Source port. That will make it easy to find your session and the security policy ID. When you are finished remember to clear the filters.

 

CLI session information

The session table output from the CLI (diag sys session list) is very verbose. Even on a system with a small amount of traffic, displaying the session table will generate a large amount of output. For this reason, filters are used to display only the session data of interest.

You can filter a column in the web-based manager by clicking the search icon on the column heading or from the CLI by creating a filter.

An entry is placed in the session table for each traffic session passing through a security policy. The following command will list the information for a session in the table:

diag sys session list

 

Sample Output:

FGT# diag sys session list

session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3 bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0

logtype=session ha_id=0 hakey=4450

tunnel=/

state=log shape may_dirty

statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2

orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100 hook=post dir=org act=snat 10.0.5.100:1251->192.168.11.254:22(192.168.11.105:1251) hook=pre dir=reply act=dnat 192.168.11.254:22->192.168.11.105:1251(10.0.5.100:1251) pos/(before,after) 0/(0,0), 0/(0,0)

misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff

 

Since output can be verbose, the filter option allows specific information to be displayed, for example:

diag sys session filter <option>

The <option> values available include the following:

clear                   Clear session filter.

 

dintf                   Destination interface.

 

dport                   Destination port.

 

dst                     Destination IP address.

 

duration               duration

 

expire                  expire

 

negate                  Inverse filter.

 

nport                   NAT’d source port

 

nsrc                    NAT’d source ip address

 

policy                  Policy ID.

 

proto                   Protocol number.

 

proto-state            Protocol state.

 

sintf                   Source interface.

 

sport                   Source port.

 

src                     Source IP address.

 

vd                      Index of virtual domain. -1 matches all.

 

 

Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different states:

• UDP reply not seen with a value of 0
• UDP reply seen with a value of 1

 

The following illustrates FW session states from the session table:

 

State                        Meaning

 

log                              Session is being logged.

 

local                           Session is originated from or destined for local stack.

 

ext                              Session is created by a firewall session helper.

 

State                        Meaning

 

may_dirty

Session is created by a policy. For example, the session for ftp control chan- nel will have this state but ftp data channel will not. This is also seen when NAT is enabled.

 

ndr                             Session will be checked by IPS signature.

 

nds                             Session will be checked by IPS anomaly.

 

br                               Session is being bridged (TP) mode.

 

 

Firewall session setup rate

The number of sessions that can be established in a set period of time is useful information. A session is an end- to-end TCP/IP connection for communication with a limited lifespan. If you record the setup rate during normal operation, when you experience problems you have that setup rate with the current number to see if its very different. While this will not solve your problems, it can be a useful step to help you define your problem.

A reduced firewall session setup rate could be the result of a number of things from a lack of system resources on the FortiGate unit, to reaching the limit of your session count for your VDOM.

 

To view your session setup rate method 1- CLI

FGT# get sys performance status

CPU states: 0% user 0% system 0% nice 100% idle

Memory states: 10% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes,

13 kbps in 30 minutes

Average sessions: 31 sessions in 1 minute, 30 sessions in 10 minutes, 31 sessions in 30 minutes

Average session setup rate: 0.5 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 44 days, 18 hours, 42 minutes

 

The information you are looking for is the Average sessions section, highlighted in the above output. In this example you can see there were 31 sessions in 1 minute, or an average of 0.5 sessions per second. The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate unit is working at maximum capacity. The smallest FortiGate unit can have 1 000 sessions established per second across the unit.

Remember that session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each one, the session setup rate per VDOM will be slower than if there were no VDOMs configured.

 

Finding object dependencies

An administrator may not be permitted to delete a configuration object if there are other configuration objects that depend on it. This command identifies other objects which depend on or make reference to the configuration object in question. If an error is displayed that an object is in use and cannot be deleted, this command can help identify the source of the problem.

Another use is if you have a virtual interface with objects that depend on it, you need to find and remove those dependencies before you delete that interface.

 

CLI method

When running multiple VDOMs, this command is run in the Global configuration only and it searches for the named object both in the Global and VDOM configuration most recently used:

diag sys checkused <path.object.mkey>

 

For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as follows:

diag sys checkused firewall.policy.policyid 1

 

To check what is referred to by interface port1, enter the following command:

diag sys checkused system.interface.name port1

 

To show all the dependencies for an interface, enter the command as follows:

diag sys checkused system.interface.name <interface name>

 

Sample Output:

entry used by table firewall.address:name ‘10.98.23.23_host’

entry used by table firewall.address:name ‘NAS’ entry used by table firewall.address:name ‘all’

entry used by table firewall.address:name ‘fortinet.com’

entry used by table firewall.vip:name ‘TORRENT_10.0.0.70:6883′ entry used by table firewall.policy:policyid ’21’

entry used by table firewall.policy:policyid ’14’ entry used by table firewall.policy:policyid ’19’

In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.

 

Web–based manager method

In the web-based manager, the object dependencies for an interface can be easily checked and removed.

 

 

To remove interface object dependencies – web-based manager

1. Go to System > Interfaces.

The number in the Ref. column is the number of objects that refer to this interface.

2. Select the number in the Ref. column for the desired interface.

A Window listing the dependencies will appear.

3. Use these detailed entries to locate and remove object references to this interface.

The trash can icon will change from gray when all object dependencies have been removed.

4. Remove the interface by selecting the check box for the interface, and select Delete.

 

Flow trace

To trace the flow of packets through the FortiGate unit, use the following command:

diag debug flow trace start

 

If your network is using IPv4, follow packet flow by setting a flow filter using this command:

diag debug flow filter <option>

 

Filtering options include the following:

addr IPv4 address

clear clear filter

daddr destination IPv4 address

dport destination port negate inverse IPv4 filter port port

proto protocol number saddr source IPv4 address sport source port

vd index of virtual domain, -1 matches all

 

If your network is using IPv6, follow packet flow by setting a flow filter using this command:

diag debug flow filter6 <option>

 

Filtering options include the following:

addr IPv6 address

clear clear filter

daddr destination IPv6 address

dport destination port negate inverse IPv6 filter port port

proto protocol number saddr source IPv6 address sport source port

vd index of virtual domain, -1 matches all

 

Enable the output to be displayed to the CLI console using the following command:

diag debug flow show console enable

 

diag debug flow output is recorded as event log messages and are sent to a FortiCloud or a FortiAnalyzer unit if connected. Do not let this command run longer than necessary since it generates significant amounts of data.

 

Start flow monitoring with a specific number of packets using this command:

diag debug flow trace start <N>

 

Stop flow tracing at any time using:

diag debug flow trace stop

 

The following is an example of the flow trace for the device at the following IP address: 203.160.224.97

diag debug enable

diag debug flow filter addr 203.160.224.97 diag debug flow show console enable

diag debug flow show function-name enable diag debug flow trace start 100

 

Flow trace output example – HTTP

Connect to the web site at the following address to observe the debug flow trace. The display may vary slightly:

http://www.fortinet.com

 

Comment: SYN packet received:

 

id=20085 trace_id=209 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,

192.168.3.221:1487->203.160.224.97:80) from port5.”

 

SYN sent and a new session is allocated:

 

id=20085 trace_id=209 func=resolve_ip_tuple line=2799 msg=”allocate a new session-00000e90″

 

Lookup for next-hop gateway address:

 

id=20085 trace_id=209 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.11.254 via port6″

 

Source NAT, lookup next available port:

 

id=20085 trace_id=209 func=get_new_addr line=1219 msg=”find SNAT: IP-192.168.11.59, port-31925″ direction“

 

Matched security policy. Check to see which policy this session matches:

 

id=20085 trace_id=209 func=fw_forward_handler line=317 msg=”Allowed by Policy-3: SNAT”

 

Apply source NAT:

 

id=20085 trace_id=209 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

SYN ACK received:

 

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 203.160.224.97:80-

>192.168.11.59:31925) from port6.”

 

Found existing session ID. Identified as the reply direction:

 

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=210 func=  ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925->192.168.3.221:1487″

 

Lookup for next-hop gateway address for reply traffic:

id=20085 trace_id=210 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.3.221 via port5″

 

ACK received:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,192.168.3.221:1487->203.160.224.97:80) from port5.”

 

Match existing session in the original direction:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”

 

Apply source NAT:

id=20085 trace_id=211 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

Receive data from client:

id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 192.168.3.221:1487->203.160.224.97:80) from port5.”

 

Match existing session in the original direction:

id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”

 

Apply source NAT:

id=20085 trace_id=212 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

Receive data from server:

id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,

203.160.224.97:80->192.168.11.59:31925) from port6.”

 

Match existing session in reply direction:

id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=213 func=  ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925-

>192.168.3.221:1487″

 

 

Flow trace output example – IPsec (policy-based)

id=20085 trace_id=1 msg=”vd-root received a packet(proto=1, 10.72.55.240:1->10.71.55.10:8)

from internal.”

id=20085 trace_id=1 msg=”allocate a new session-00001cd3″

id=20085 trace_id=1 msg=”find a route: gw-66.236.56.230 via wan1″ id=20085 trace_id=1 msg=”Allowed by Policy-2: encrypt”

id=20085 trace_id=1 msg=”enter IPsec tunnel-RemotePhase1″

id=20085 trace_id=1 msg=”encrypted, and send to 15.215.225.22 with source 66.236.56.226″ id=20085 trace_id=1 msg=”send to 66.236.56.230 via intf-wan1“

id=20085 trace_id=2 msg=”vd-root received a packet (proto=1, 10.72.55.240:1-1071.55.10:8)

from internal.”

id=20085 trace_id=2 msg=”Find an existing session, id-00001cd3, original direction” id=20085 trace_id=2 msg=”enter IPsec =”encrypted, and send to 15.215.225.22 with source

66.236.56.226“ tunnel-RemotePhase1″

id=20085 trace_id=2 msgid=20085 trace_id=2 msg=”send to 66.236.56.230 via intf-wan1″

 

Packet sniffing and packet capture

FortiOS devices can sniff packets using commands in the CLI or capture packets using the web-based manager. The differences between the two methods are not large.

Packet sniffing in the CLI is well suited for spot checking traffic from the CLI, but if you have complex filters to enter it can be a lot of work to enter them each time. You can also save the sniffing output; however, you must log to a file and then analyze the file later by hand.

Packet capture in the web-based manager makes it easy to set up multiple filters at once and just run one or two as you need them. You also have controls to start and stop capturing as you wish. Packet capture output is downloaded to your local computer as a *.pcap file which requires a third party application to read the file, such as Wireshark. This method is useful to send Fortinet support information to help resolve an issue.

 

Features                                                    Packet sniffing                         Packet capture

Command location                                       CLI                                               web-based manager

 

Third party software required                        puTTY to log plaintext output      Wireshark to read *.pcap files

 

Read output in plain text file                         yes                                               no

 

Read output as *.pcap file using Wire-

shark                                                             no                                                yes

 

Easily configure single quick and simple filter

yes                                               no

 

Record packet interface                                yes                                               no

 

 

Features                                                    Packet sniffing                         Packet capture

Configure complex sniffer filters on mul- tiple interface

no                                                yes

 

sniff IPv6                                                       hard                                             easy

 

sniff non-IP packets                                      no                                                yes

 

Filter packets by protocol and/or port            easy                                             easy

 

Filter packets by source and/or des- tination address

easy                                             easy

 

 

Packet sniffing

Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file — there can be huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate unit’s CLI. Then once the packet sniffing count is reached you can end the session and analyze the output in the file.

Details within packets passing through particular interfaces can be displayed using the packet sniffer with the following command:

diag sniffer packet <interface> <filter> <verbose> <count> <tsformat>

The <interface> value is required, with the rest being optional. If not included the default values will be “none”.

For example the simplest valid sniffer command would be:

diag sniffer packet any

 

The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all interfaces.

 

The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax. The <filter> value must be enclosed in quotes.

‘[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]]‘

 

If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a port, there are up to two parts — protocol and port number.

For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following:

‘udp port 1812 or tcp port 8080’

 

To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3:

‘ip src host 192.168.1.2 and dst host 192.168.2.3’

 

The <verbose> option allows different levels of information to be displayed. The verbose levels include:

1 Print header of packets

2 Print header and data from the IP header of the packets

3 Print header and data from the Ethernet header of the packets

4 Print header of packets with interface name

5 Print header and data from ip of packets with interface name

6 Print header and data from ethernet of packets with interface name

 

The <count> value indicates the number of packets to sniff before stopping. If this variable is not included, or is set to zero, the sniffer will run until you manually halt it with Ctrl-C.

 

The <tsformat> value define the format of timestamp. It can be:

 

a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms

 

l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms otherwise: relative to the start of sniffing, ss.ms

 

Packet capture

FortiOS 5.2 includes packet capture to the web-based manager. The FortiGate unit must have a disk and then capture-packet feature can be enabled from the CLI within the firewall policy as below:

config firewall policy edit <id>

set capture-packet enable end

To configure packet capture filters, go to System > Network > Packet Capture. When you add a packet capture filter, enter the following information and select OK.

 

Interface                                     Select the interface to sniff from the dropdown menu.

You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields.

 

Max Packets to Capture

Enter the number of packets to capture before the filter stops.

 

This number cannot be zero. You can halt the capturing before this number is reached.

 

Enable Filters                            Select this option to specify your filter fields

 

Host(s)

Enter one or more hosts IP address

 

Separate multiple hosts with commas. Enter a range using a dash without spaces, for example 172.16.1.5-172.16.1.15 or enter a subnet.

 

 

Port(s)                                         Enter one or more ports to capture on the selected interface.

 

Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90

 

 

VLAN(s)

Enter one or more vlans (if there is any).

Separate multiple vlans with commas.

 

Protocol                                      Enter one or more protocol. Separate multiple protocol with commas. Enter a range using a dash without spaces, for example 1-6, 17, 21-25

 

Include IPv6 packets                Select this option if you are troubleshooting IPv6 networking, or if your net- work uses IPv6. Otherwise, leave it disabled.

 

Capture Non-IP packets           The protocols available in the list are all IP based except for ICMP (ping).

To capture non-IP based packets select this feature. Some examples of non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.

 

If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in the edit window, or downloading the captured packets. You can also see the filter status and the number of packets captured.

You can also select the filter and select Start to start capturing packets. While the filter is running, you will see the number of captured packets increasing until it reaches the max packet count or you select Stop. While the filter is running you cannot download the output file.

When the packet capture is complete, you can select Download to send the packet capture filter captured packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark or a similar third party application. Using this tool you will have extensive analytics available to you and the full contents of the packets that were captured.

 

NPU based interfaces

Many Fortinet products contain network processors such as NP1, NP2, and NP4 network processors. Therefore offloading requirements, vary by network processor model.

When using the NPU-based interfaces, only the initial session setup will be seen through the diag debug flow command. If the session is correctly programmed into the ASIC (fastpath), the debug flow command will no longer see the packets arriving at the CPU. If the NPU functionality is disabled, the CPU will see all the packets, however, this should only be used for troubleshooting purposes.

First, obtain the NP4 id and the port numbers with the following command:

diag npu np4 list

 

Sample output:

ID Model Slot Interface

0 On-board port1 fabric1 fabric3 fabric5

1 On-board fabric2 port2 base2 fabric4

 

Run the following commands:

diag npu np4 fastpaf th disable <dev_id>

 

(where dev_id is the NP4 number) Then, run this command:

diag npu np4 fastpath-sniffer enable port1

 

Sample output:

NP4 Fast Path Sniffer on port1 enabled

This will cause all traffic on port1 of NP4 to be sent to the CPU meaning a standard sniffer trace can be taken and other diag commands should work if it was a standard CPU driven port.

These commands are only for the newer NP4 interfaces.

 

Debug command

Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually generated even though output might not be displayed in the CLI console.

Debug information displayed in the console will scroll in the console display and may prevent CLI commands from being entered, for example, the command to disable the debug display. To turn off debugging output as the display is scrolling by, press the á key to recall the recent diag debug command, press backspace, and type “0”, followed by Enter.

Debug output display is enabled using the following command:

diag debug enable

 

When finished examining the debug output, disable it using:

diag debug disable

 

Once enabled, indicate the debug information that is required using this command:

diag debug <option> <level>

 

Debug command options include the following:

enable                  Enable debug output.

disable                 Disable debug output.

info                    Show active debug level settings.

reset                   Reset all debug level to default.

report                  Report for tech support.

crashlog               Crash log info.

config-error-log       Configure error log info.

sql-log-error          SQL log database error info.

application            application.

kernel                  kernel.

remote-extender        remote-extender.

console                 console.

cli                     Debug CLI.

cmdb-trace             Trace CLI.

rating                  Display rating info.

authd                   Authentication daemon.

fsso-polling           FSSO active directory poll module.

flow                    Trace packet flow in kernel.

urlfilter              urlfilter.

admin                   Admin user.

 

The debug level can be set at the end of the command. Typical values are 2 and 3, for example:

diag debug application DHCPS 2

diag debug application spamfilter 2

 

Fortinet support will advise which debugging level to use.

Timestamps can be enabled to the debug output using the following command:

diag debug console timestamp enable

 

Debug output example

This example shows the IKE negotiation for a secure logging connection from a FortiGate unit to a FortiAnalyzer system.

diag debug reset

diag vpn ike log-filter src-addr4 192.168.11.2 diag debug enable

 

Sample Output:

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, natt_mode=0 rekey=0 phase2=FGh_FtiLog1

FGh_FtiLog1: using existing connection, dpd_fail=0

FGh_FtiLog1: found phase2 FGh_FtiLog1

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating

FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2

FGh_FtiLog1: initiator quick-mode set pfs=1536…

FGh_FtiLog1: try to negotiate with 1800 life seconds.

FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0

Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348

Initiator: sent 192.168.10.201 quick mode message #1 (OK) FGh_FtiLog1: set retransmit: st=168, timeout=6.

 

In this example:

192.168.11.2->192.168.10.201:500      Source and Destination gateway IP address

dpd_fail=0                              Found existing Phase 1

pfs=1536…                            Create new Phase 2 tunnel

 

The execute tac report command

exec tac report is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you are using certain features like HA, VPN tunnels, or a modem. The report takes a few minutes to complete due to the amount of output generated. If you have your CLI output logged to a file, you can run this command to familiarize yourself with the CLI commands involved.

When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current state using the output from this CLI command.

Other commands

 

ARP table

To view the ARP cache, use the following command:

get sys arp

To view the ARP cache in the system, use this command:

diag ip arp list

 

Sample output:

index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203 confirm=78203 update=72203 ref=1

index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179 ref=2 ? VIP

index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743 update=75743 ref=1

index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532 confirm=10532 update=12658 ref=4

 

To remove the ARP cache, use this command:

execute clear system arp table

 

To remove a single ARP entry, use:

diag ip arp delete <interface name> <IP address>

 

To add static ARP entries, use the following command:

config system arp-table

 

Time and date settings

Check time and date settings for log message timestamp synchronization (the Fortinet support group may request this) and for certificates that have a time requirement to check for validity. Use the following commands:

execute time

current time is: 12:40:48

last ntp sync:Thu Mar 16 12:00:21 2006 execute date

current date is: 2006-03-16

 

To force synchronization with an NTP server, toggle the following command:

Config system ntp

set ntpsync enable/disable end

 

If all devices have the same time, it helps to correlate log entries from different devices.

 

IP address

There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what you expect them to be. This is easily accomplished from the CLI using the following command.

diag ip address list

 

The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID number) and the devname is the name of the interface. While physical interface names are set, virtual interface names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_ fgfm, the IP addresses are the local host — these are internally used virtual interfaces.

# diag ip address list

IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal

IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1

IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm