External resources for webfilter

Introduction

External Resources is a new feature introduced in FortiOS 6.0, which provides a capability to import an external blacklist which sits on an HTTP server. This feature helps FortiGate retrieve a dynamic URL/Domain Name/IP Address/Malware hash list from an external HTTP server periodically. FortiGate uses these external resources as web filter’s remote categories, DNS filter’s remote categories, policy address objects or AntiVirus profile’s malware definitions. If the external resource is updated, FortiGate objects will update dynamically.

External Resource are categorized into 4 types:

• URL list (Type=category) l Domain Name List (Type=domain) l IP Address list (Type=address) l Malware hash list (Type=malware)

For Web Filter profile, it can use category type external resources. Category type external resources file is a URL entries list in a plain text file.

When a category type external resource is configured in Web Filter profile, it will be treated as a Remote Category. If the URL in a HTTP/HTTPS request matches the entry inside this external resource file, it will be treated as the Remote Category and follow the action configured for this category in Web Filter profile.

External resource type category also can be used in ssl-ssh-profile configuration for category-based SSL-Exempt. When a Remote Category is configured in ssl-ssh-profile SSL-Exempt, if a HTTPS request’s URL matches in the Remote Category’s entry list, HTTPS request with destination for this URL can be exempted from SSL Deep Inspection. External Resources File Format

External Resources File should follow the following requirements:

• The external resource file is a plain text format file and each URL list/IP Address/Domain Name occupies a single line. l The file is limited to 10M, line is limited 128K (128 x 1024 entries), and the line length limit is 4K characters. l The entries limited also follow table size limitation defined by CMDB per model. l The external resource update period can be set to 1 minute, hourly, daily, weekly, or monthly (43200 min, 30 days).
• The external resource type as category (URL list) and domain (Domain Name list) share the category number range 192-221 (total 30 categories). l There’s no duplicated entry validation for external resources file (entry inside each file or inside different files).

For URL list (Type=category):

Scheme is optional, and will be truncated if found (http://, https:// is not needed).

Wildcard (*) is supported (from 6.2). It supports the ‘*’ at beginning and ending of URL, and not in the middle of URL as follows:

+ support *.domain2.com, domain.com.* + not support: domain3.*.com IDN (International Domain Name) and UTF encoding URL is supported (from 6.2).

IPv4,IPv6 format URL is supported. IPv6 in URL list must in [ ] form.

Configure External Resources from CLI

We can use CLI to configure the external resources files that is located on external HTTP Server. Under Global, configure the external resource file location and specify the resource type.

Web Filter will use category type external resources as Remote Categories. In the following example, it is configured a file Ext-Resource-Type-as-Category-1.txt as type as category, it will be treated in Web Filter as Remote Category, the category name configured as Ext-Resource-Type-as-Category-1 and category-id as 192:

config system external-resource edit “Ext-Resource-Type-as-Category-1”

set type category <—-

set category 192 <—-

set resource “http://172.16.200.66/external-resources/Ext-Resource-Type-as-Category-

1.txt” set refresh-rate 1

next

end

Now in each VDOM, category type external resource can be used in Web Filter as Remote Cateogry. In the example above, URL list in “Ext-Resource-Type-as-Category-1.txt” file will be treated as remote category (category-id 192). Configure the action for this remote category in Web Filter profile and apply it in the policy:

config webfilter profile edit “webfilter” config ftgd-wf unset options config filters edit 1 set category 2 set action warning

next ……

edit 24 set category 192 <—set action block

next edit 25 set category 221 set action warning

next edit 26 set category 193

next

end

end

set log-all-url enable

next

end

config firewall policy edit 1 set name “WebFilter” set srcintf “port10” set dstintf “port9” set srcaddr “all” set dstaddr “all” set action accept set schedule “always” set service “ALL” set utm-status enable set logtraffic all set webfilter-profile “webfilter” set profile-protocol-options “protocol” set ssl-ssh-profile “protocols”

set nat enable

next end

Configure External Resources from GUI

Configure, edit or view the Entries for external resources from GUI.

1. GUI > Global > Fabric Connectors page:
2. GUI > Global > Fabric Connectors page > Create New. Click Create New button, and select Threat Feeds Type FortiGuard
3. GUI > Global > Fabric Connectors page. Enter the Resource Name, URL Location of the resource file, resource authentication credential, Refresh Rate or comment, and click OK to finish the Threat Feeds configuration.
4. GUI > Global > Fabric Connectors page. After a few minutes, double-click the Threat Feeds Object you just configured. It is shown in the Edit Click View Entries to view the entry list in the external resources file:
5. GUI > VDOM > Web Filter Profile page. The configured external resources is shown and configured in each Web

Filter Profile:

Log Example

If a HTTP/HTTPS request URL matched in Remote Category’s entry list, it will override its original FGD URL rating and it is treated as Remote Category.

GUI > VDOM > Log & Report > Web Filter:

CLI Example:

1: date=2019-01-18 time=15:49:15 logid=”0316013056″ type=”utm” subtype=”webfilter” eventtype=”ftgd_blk” level=”warning” vd=”vdom1″ eventtime=1547855353 policyid=1 sessionid=88922 srcip=10.1.100.18 srcport=39886 srcintf=”port10″ srcintfrole=”undefined” dstip=216.58.193.67 dstport=443 dstintf=”port9″ dstintfrole=”undefined” proto=6 service=”HTTPS” hostname=”www.fortinet.com” profile=”webfilter” action=”blocked” reqtype=”direct” url=”/” sentbyte=752 rcvdbyte=10098 direction=”outgoing” msg=”URL belongs to a denied category in policy” method=”domain” cat=192 catdesc=”Ext-Resource-Type-as-Category-1″

Remote Category in ssl-ssh-profile category-based SSL-Exempt

Remote Category can be applied in ssl-ssh-profile category-based SSL-Exempt.

GUI > VDOM > Security Profiles > SSL/SSH Inspection:

HTTPS Request URL matched in this Remote Category will be exempted from SSL Deep Inspection.

Log example:

3: date=2019-01-18 time=16:06:21 logid=”0345012688″ type=”utm” subtype=”webfilter” eventtype=”ssl-exempt” level=”information” vd=”vdom1″ eventtime=1547856379 policyid=1 sessionid=90080 srcip=10.1.100.18 srcport=39942 srcintf=”port10″ srcintfrole=”undefined” dstip=216.58.193.67 dstport=443 dstintf=”port9″ dstintfrole=”undefined” proto=6 service=”HTTPS” hostname=”www.fortinet.com” profile=”webfilter” action=”passthrough” reqtype=”direct” url=”/” sentbyte=517 rcvdbyte=0 direction=”outgoing” msg=”The SSL session was exempted.” method=”domain” cat=192 catdesc=”Ext-Resource-Type-as-Category-1″ urlsource=”exempt_type_user_cat”

Local Category and Remote Category Priority

Web Filter can have both local category and remote category at the same time. There’s no duplication check between local category URL override and remote category resource file. For example, a URL like www.example.com may be shown both in remote category entry list and in FortiGate’s local category URL override configuration. We recommend avoiding this scenario since FortiGate does not check for duplicates. However, if a URL is duplicated in both local category and remote category, it is rated as local category.