Chapter 26 – Troubleshooting

 

This handbook chapter describes concepts of troubleshooting and solving issues that may occur with FortiGate units.

This FortiOS Handbook chapter contains the following chapters:

Verifying FortiGate admin access security explains how to verify and configure administrative access. Troubleshooting resources walks you through Fortinet’s resources for troubleshooting.

Troubleshooting tools describes some of the basic commands and parts of FortiOS that can help you with troubleshooting.

Troubleshooting methodologies walks you through best practice concepts of FortiOS troubleshooting.

Technical Support Organization Overview describes how Fortinet Support operates, what they will need from you if you contact them, and what you can expect in general.

Common questions answers most of the common questions.

 

Troubleshooting methodologies

Before you begin troubleshooting anything but the most minor issues, you need to prepare. Doing so will shorten the time to solve your issue. This section helps to explain how you prepare before troubleshooting, as well as creating a troubleshooting plan and contacting support.

 

This section contains the following topics:

• Establish a baseline
• Define the problem
• Gathering Facts
• Create a troubleshooting plan
• Obtain any required additional equipment
• Ensure you have administrator level access to required equipment
• Contact Fortinet customer support for assistance

 

Establish a baseline

FortiGate units operate at all layers of the OSI model. For this reason troubleshooting problems can become complex. If you establish a normal operation parameters, or baseline, for your system before the problem occurs it will help reduce the complexity when you are troubleshooting.

Many of the guiding questions in the following sections are some form of comparing the current problem situation to normal operation on your FortiGate unit. For this reason it is a best practice that you know what your normal operating status is, and have a record of it you can refer to. This can easily be accomplished by monitoring the system performance with logs, SNMP tools, or regularly running information gathering commands and saving the output. This regular operation data will show trends, and enable you to see when changes happen and there may be a problem.

Back up your FortiOS configuration on a regular basis. This is a good practice for every- day as well as when troubleshooting. You can restore the backed up configuration when needed and save the time and effort of re-creating it from the factory default set- tings.

Some fundamental CLI commands you can use to obtain normal operating data for your system:

get system status              Displays versions of firmware and FortiGuard engines, and other system information.

 

get system performance status

Displays CPU and memory states, average network usage, aver- age sessions and session setup rate, virus caught, IPS attacks blocked, and uptime.

 

get hardware memory            Displays informations about memory

 

get system session status      Displays total number of sessions

 

get router info routing-table all

Displays all the routes in the routing table including their type, source, and other useful data.

 

get ips session                Displays memory used and max available to IPS as well and counts.

 

get webfilter ftgd-statistics Displays list of FortiGuard related counts of status, errors, and other data.

 

diagnose firewall statistic show

Displays the amount of network traffic broken down into cat- egories such as email, VoIP, TCP, UDP, IM, Gaming, P2P, and Streaming.

 

diag system session list       Displays current detailed sessions list

 

show system dns                Displays configured DNS servers

 

diag sys ntp status            Displays informations about ntp servers

 

These commands are just a sample. Feel free to include any extra information gathering commands that apply to your system. For example if you have active VPN connections, record information about them using the get vpn * series of commands.

For an extensive snapshot of your system, run the CLI command used by TAC to gather extensive information about a system — exec tac report. It runs many diagnostic commands that are for specific configurations. This means no matter what features you are using, this command will record their current state. Then if you need to perform troubleshooting at a later date, you can run the same command again and compare the differences to quickly locate suspicious output you can investigate.

 

Define the problem

The following questions can help determine the scope of the problem and isolate it:

 

• What is the problem?

Do not assume that the problem is being experienced is the actual problem. First determine that the problem does not lie elsewhere before starting to troubleshoot the FortiGate device.

• Has it ever worked before?

If the device never worked from the first day, you may not want to spend time troubleshooting something that could well be defective. See “Troubleshooting bootup”.

• Can the problem be reproduced at will or is it intermittent?

If the problem is intermittent, it may be dependent on system load. Also an intermittent problem can be very difficult to troubleshoot due to the difficulty reproducing the issue.

• What has changed?

Do not assume that nothing has changed in the network. Use the FortiGate event log to see if any configuration changes were made. The change could be in the operating environment, for example, a gradual increase in load as more sites are forwarded through the firewall.

If something has changed, see what the affect is if the change is rolled back.

• Determine the scope of the problem – after you have isolated the problem what applications, users, devices, and operating systems does it effect?

 

Before you can solve a problem, you need to understand it. Often this step can be the longest in this process. Ask questions such as:

• What is not working? Be specific.
• Is there more than one thing not working?
• Is it partly working? If so, what parts are working?
• Is it a connectivity issue for the whole device, or is there an application that isn’t reaching the Internet? Be as specific as possible with your answers, even if it takes awhile to find the answers.

These questions will help you define the problem. Once the problem is defined, you can search for a solution and then create a plan on how to solve it.

 

Gathering Facts

Fact gathering is an important part of defining the problem. Record the following information as it applies to the problem:

• Where did the problem occur?
• When did the problem occur and to whom?
• What components are involved?
• What is the affected application?
• Can the problem be traced using a packet sniffer?
• Can the problem be traced in the session table or using system debugging?
• Can log files be obtained that indicate a failure has occurred?

Answers to these questions will help you narrow down the problem, and what you have to check during your troubleshooting. The more things you can eliminate, the fewer things you need to check during troubleshooting. For this reason, be as specific and accurate as you can while gathering facts.

 

Create a troubleshooting plan

Once you have defined the problem, and searched for a solution you can create a plan to solve that problem. Even if your search didn’t find a solution to your problem you may have found some additional things to check to further define your problem.

The plan should list all the possible causes of the problem that you can think of, and how to test for each possible cause.

Your troubleshooting plan will act as a checklist so that you know what you have tried and what is left to check. This is important to have if more than one person will be doing the troubleshooting. Without a written plan, people will become easily confused and steps will be skipped. Also if you have to hand over the problem to someone else, providing them with a detailed list of what data has been gathered and what solutions have been already tried demonstrates a good level of professionalism.

Be ready to add to your plan as needed. After you are part way through, you may discover that you forgot some tests or a test you performed discovered new information. This is normal.

Also if you contact support, they will require information about your problem as well as what you have already tried to fix the problem. This should all be part of your plan.

 

Providing Supporting Elements

If the Fortinet Technology Assistance Center (TAC) needs to be contacted to help you with your issue, be prepared to provide the following information:

• The firmware build version (use the get system status command)
• A network topology diagram
• A recent configuration file
• Optionally, a recent debug log
• Tell the support team what troubleshooting steps have already been performed and the results.

Do not provide the output from exec tac report unless Support requests it. The output from that command is very large and is not required in many cases.

For additional information about contacting Fortinet Customer Support, see Technical Support Organization Overview.

All of this is your troubleshooting plan.

 

Obtain any required additional equipment

You may require additional networking equipment, computers, or other equipment to test your solution. Normally network administrators have additional networking equipment available either to loan you, or a lab where you can bring the FortiGate unit to test.

If you do not have access to equipment, check for shareware applications that can perform the same task. Often there are software solutions when hardware is too expensive.

 

Ensure you have administrator level access to required equipment

Before troubleshooting your FortiGate unit, you will need administrator access to the equipment. If you are a client on a FortiGate unit with virtual domains enabled, often you can troubleshoot within your own VDOM. However, you should inform your FortiGate unit’s super admin that you will be doing troubleshooting.

Also, you may need access to other networking equipment such as switches, routers, and servers to help you test. If you do not normally have access to this equipment, contact your network administrator for assistance.

 

Contact Fortinet customer support for assistance

You have defined your problem, researched a solution, put together a plan to find the solution, and executed that plan. At this point if the problem has not been solved, its time to contact Fortinet Customer Support for assistance.

For more information, see Technical Support Organization Overview.