FGT2 16 Intrusion Prevention System

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

In this lesson, we will show you how to use FortiGate IPS. IPS is part of what makes FortiGate a UTM that can keep pace with the latest attacks. Beyond simply TCP stateful inspection and masking internal network IPs, modern FortiGate UTM firewalls can detect and block exploit attempts in higher layer protocols.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

After completing this lesson, you should have these practical skills. Essentially, you will learn how to use your FortiGate to study what is normal for your network, then detect and block rate anomalies and mechanism attacks. Lab exercises can help you to test and reinforce your skills.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Before we begin, it’s important to understand: Not all attacks can be 100% positively identified. Sometimes, there is uncertainty. What is the difference between an attack and an anomaly? To compare, FortiGate IPS uses attack signatures where it can detect an attack with relative certainty and performance. But the IPS engine also can use heuristic methods to find statistical anomalies – unusual order in the packet flow, or suspicious volumes of certain packet types. An example: the client uses the HTTP “MKCOL” method, but your web site has only static web pages, so it’s suspicious to use a method for dynamic sites. Many anomalies indicate a DoS attempt. So the IPS engine is also used by DoS policies, except where it’s performed in specialized hardware – FortiASIC chips – instead of in the kernel, on the CPU. If an anomaly is actually normal for your specific network, to reduce false positives, disable that signature in your IPS profile or DoS policy.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) Let’s define what IPS currently means on FortiGate. You may be surprised. On older systems, IPS might have meant purely a Snort-style signature matching. It was similar to anti-virus signatures, but for protocols instead of files. But on FortiGate UTM, IPS has evolved to also detect anomalous traffic patterns, such as a flood of traffic exceeding the usual bandwidth volume, and to apply heuristics that prevent an unexpected behavior of the protocol. (click) Why? Aren’t IPS signatures enough? Some attacks can’t be successfully or efficiently defined in a signature. If the attack is qualitatively or quantitatively too similar to legitimate traffic, IPS false positives will block your network service – not the result you want.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) How does the IPS engine determine if a packet contains an attack or anomaly? Protocol decoders parse each packet according to the protocol specifications. Some protocol decoders do require a port number specification (configured in the CLI), but usually, the protocol is automatically detected. If the traffic doesn’t conform to specification – if, for example, it sends malformed or invalid commands to your servers – then the protocol decoder detects the error. For example, a stream of packets might match the HTTP decoder’s pattern named “Cisco.CatOS.CiscoView.HTTP.Server.Buffer.Overflow”. (click) A default, initial set is included in each FortiGate firmware. FortiGuard IPS service updates them, sometimes daily, with new signatures. That way, IPS remain effective against new exploits. Unless a protocol specification or RFC changes (which is not very often), protocol decoders are rarely updated. The IPS engine itself changes more frequently, but still not often. What part of IPS is updated most? The IPS signatures. New signatures are identified and built during the day by FortiGuard research teams, just like with antivirus. So if your FortiGuard Services contract expires, you can still use IPS. However, just like with anti-virus scans, IPS scans will over time become increasingly ineffective – old signatures won’t defend against new attacks.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Regular updates are vital. If your FortiGate doesn’t have the latest signatures, your network is vulnerable. Always make sure that your FortiGate has a reliable Internet connection, and that it is scheduled to often request updates from FortiGuard. What is included in a FortiGuard IPS update? Protocol decoders, the engine, and signatures. The signature database is subdivided into Regular and Extended.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Regular signatures are common attacks whose signatures, during testing prior to release on the FortiGuard Distribution Network, caused rare or no false positives. So it’s a smaller database, and its default action is to block the detected attack. Extended signatures contain everything else. In FortiOS 5.2, the IPS extended database is enabled by default for all FortiGate models that have multiple CP8. Otherwise, they are disabled, because either: • Performance impact is significant, or • Nature of the attack doesn’t support blocking By default, the Regular database is selected, not the Extended. In fact, due to its size, the extended database is not available for FortiGate models with a smaller disk and/or RAM. But for high security networks, you may be required to enable extended signatures. In that case, you should mark the “Enable Extended IPS Signature Package” option on System > Config > FortiGuard.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

When your FortiGate downloads new IPS signatures, or a new engine, syntax may change. So if you write your own custom signatures, especially after upgrading your FortiGate’s firmware, you may need to check if it’s still compatible. IPS involves anomaly inspection, deep packet inspection, full content inspection, activity inspection, and heuristic detection. Some software does not maintain a constant pattern. Skype and other peer-to-peer software, for example, periodically change in order to avoid detection. So in order to correctly identify it, IPS requires heuristics and adaptive detection. As a result, FortiGuard IPS also provides updates for application control, for example.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

When your FortiGate downloads a FortiGuard IPS package, new signatures will appear in the signature list. For each sensor that uses a signature, when configuring, you can change its “Action” setting. The default often is correct, except if: • Your software vendor releases a security patch. Continuing to scan for exploits will waste FortiGate resources. • Your network has a custom application with traffic that inadvertently triggers an IPS signature. You can disable it until you notify Fortinet so that FortiGuard can modify the signature to avoid false positives. The list of IPS signatures also indicates the severity level. What do the indicators mean?

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

The FortiGuard severity level is based on the CVSS 2 rating system. There are many contributing factors. For details, go to the first.org web site. Do all severity levels match CVSS exactly? No. Fortinet always marks remote code execution as high or critical severity, regardless of the CVSS rating. Details are explained on the FortiGuard web site.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Do you have the CVE ID or Microsoft ID for a specific vulnerability, but don’t know if there is a corresponding IPS signature yet? On the FortiGuard web site, you can search for the latest IPS signatures. But you can also read details about recently discovered zero-day attacks, white papers, blogs and security advisories.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

If you’re not sure if you should enable an IPS signature on your FortiGate, you can search the FortiGuard web site’s encyclopedia. The encyclopedia has useful information such as affected systems and recommended corrective actions. So if you don’t use that protocol or don’t have a vulnerable system, you can safely disable the corresponding signature. But if you are vulnerable, the encyclopedia can provide information about how to protect yourself. The FortiGuard encyclopedia only contains publicly disclosed vulnerabilities, though. Obviously it can’t contain vulnerabilities that, for whatever reason, can’t yet be responsibly disclosed.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Exploits for unknown vulnerabilities – called zero-day attacks – are sold for large amounts of money on the black market. Since these exploits aren’t known to their vendors, nor to security experts, there’s no available patch or signature for detection. That’s what makes them so dangerous. Some companies and organizations like Facebook and Google have offered bounties for the responsible disclosure of these exploits, but there’s a very profitable market for black hat hackers to sell these discoveries to everyone from covert government surveillance to organized crime syndicates. Zero-day attacks are the keys to your network’s kingdom.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

If you notice an attack, your initial self-defense instinct may be to immediately take the server offline, then format it to remove all traces of malware. But by doing this, you’ll alert the attacker, and destroy forensic evidence. For motivated attackers, this will only educate them – their next attack will be harder to detect, and more sophisticated. Make sure your PSIRT team understands the most appropriate way to respond to each different type of intrusion. If you’re vigilant, and if you have the resources, you can also write your own custom IPS signatures. We’ll talk about how to do that next.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Before you write custom IPS signatures, let’s first explain how the IPS engine works. FortiGate doesn’t compare traffic to each signature individually. This would require the CPU to load from disk and then evaluate each complete signature. In total, when fully enabled, this would be more than 8,000 disk accesses and comparisons. So instead, IPS compiles them into a decision tree, similar to the example shown here.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

FortiGate loads this entire decision tree into RAM. This can increase memory usage significantly, especially on desktop FortiGate models that don’t have much RAM. So if your RAM usage is already high, you should reduce it first before enabling IPS. Otherwise, your FortiGate may immediately enter conserve mode, and refuse to accept any more configuration changes! But the advantage is that the tree takes much less CPU and total RAM for a full IPS scan. To make the tree, FortiGate breaks down signatures into identical pieces – port, protocol, etc. – and shares the evaluation. So if traffic does not match that part, then the IPS engine can bypass comparisons with all similar signatures. But if it does match, then IPS continues with the next shared segment of the signature. When it finds a match, FortiGate applies its corresponding action. Remember discussing the difference between attacks and anomalies? Detecting uncertain attacks can require even more ongoing analysis, and more RAM to store traffic statistics. So if your CPU usage or RAM usage is high, and if you don’t require anomaly analysis for all protocols, clients, or servers, disable it. Better yet, offload it to an NP FortiASIC if your FortiGate model has them. Hardware accelerated anomaly detection can be configured in the CLI.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

To write custom signatures, first use packet capture to record packet samples. Understand and avoid mismatches with normal packets on your network, including at other OSI layers such as Layer 2 and Layer 3, which will be evaluated first. Remember: if you misconfigure a custom signature, or if you configure a custom signature that is no longer supported after you update the FortiGate firmware or IPS engine, problems like this often aren’t included in Fortinet Technical Support. So if possible, you should also test your custom signatures in a lab.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) We’ll show one example here. (click) All start with “F-SBID(”. (click) After that, protocol-specific key words define what part of the packet to search for a match, and what values comprise a match. Usually, a keyword is followed by a corresponding value that is its setting, except for a few standalone keywords such as “--no_case”. Each key-value pair ends with a semi-colon and a space. You can include multiple key-value pairs. The signature ends with the closing parenthesis. A reference to syntax for custom IPS signatures is in the FortiGate Handbook. Supported key words vary by the protocol decoders. For example, the SMTP protocol supports the “VRFY” command, and so there is a protocol decoder flag for it.. So if you create custom signatures, you should be sure to read the Release Notes and new Handbook before upgrading, and (if possible) test the firmware before installing it in a live traffic environment. Let’s see some examples.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) Here is a sample custom signature called “Ping.Death”. It searches for ICMP traffic that exceeds about 32 KB. (click) After you create and save the signature, FortiGate will automatically add an attack ID. So don’t include it when you enter the signature. (click) Next is a signature for HTTP. It searches for the pattern “POST” in a very specific location inside the packet. In normal HTTP POST requests, the method should be in this specific location. This prevents IPS from scanning the entire HTTP payload, which could contain a web page that accidentally matches, for example, due to the words “POSTAL CODE.” Your signature should be specific, but not too specific – extra comparisons reduce performance.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Once you have created your custom signature, pair it with an action within an IPS sensor. Then reference that IPS sensor in a firewall policy. The steps are the same, by the way, regardless of whether you want to use custom signatures or ones predefined by FortiGuard.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Here’s an example of an IPS filter being created. To include all signatures in the filter, we’ve marked ALL options. To include only a few signatures in the filter, we would only mark one option. For example, if we only marked the “Client” option, only 4 signatures would be included in the filter. Each individual signature can have multiple tags, such as HTTP, Microsoft, IIS, and TCP. The more specific you can make your filter, the less resources will be used to scan your traffic, because its parts will seldom match and so the IPS engine will quickly continue with the next comparison or scan.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

When the IPS engine compares traffic with the signatures in each filter, order matters. The rules are similar to firewall policy matching: topmost filters are evaluated first, and the first match applies. Subsequent filters are skipped. So position most likely matching filters at the top of the list, unless they might cause false positives. (Position those last, so that FortiGate will test them only if no previous, more sure signature matches.) Avoid making too many filters, since this will increase evaluations and CPU usage. Also avoid making very large signature trees in each filter, which will increase RAM usage – all unique pieces of the attack pattern must be loaded into RAM. Strike a balance. If an attack can be prevented in hardware (by NP FortiASIC chips, for example), or by another method (by disallowing an unnecessary protocol at the firewall level, for example), do this first. Then, for the remaining, craft careful IPS sensors to protect relevant vulnerabilities. For rate-based signatures (previously called anomalies), you can choose how to match: by source IP, destination IP, DHCP Client MAC, or DNS Domain Name. Choose whichever will generate the least entries yet behave correctly. For Internetfacing policies, this is unfortunately one that requires IPS to analyze many clients’ connections: Source IP. So enable only rate-based signatures for vulnerable protocols you actually use. Then block malicious clients for extended periods. This saves system resources and can discourage a repeat attack: FortiGate will not track statistics for that client while it is temporarily blacklisted.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

To apply an IPS sensor, enable IPS and then select the sensor in a firewall policy.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) So far we’ve shown signatures that match illegal commands and invalid protocol implementations. Those are easy to confirm as an attack. What about attacks that function by exploiting asymmetric processing, or bandwidth between clients and servers? There are many ways to make a Denial of Service attack. Some denial of service (DoS) attacks, for example, exhaust limited serverside bandwidth or sockets. Unless you know what bandwidth is abnormal for your network, you may not be able to confirm an attack. (click) The goal is to overwhelm the target – to consume resources until it can’t respond to legitimate traffic. This can be done in various ways. High bandwidth usage is only one type of DoS. Many sophisticated DoS such as Slowloris don’t require high bandwidth. For high-bandwidth DoS, remember that although your FortiGate blocks traffic floods, the flood is still consuming bandwidth up to the point of its external interface. So your servers are protected from impact, but if the upstream network is not, so your servers may still be effectively unavailable. Especially for distributed denial of service attacks, you must work with your ISP to fully prevent high-bandwidth DoS.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

To block DoS attacks, apply a DoS policy on a FortiGate that is between attackers and all resources that you want to protect.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

DoS protection exists for 4 protocols: TCP, UDP, ICMP and SCTP. Each one has 4 different types of anomaly detection. • A flood sensor detects a high volume of that particular protocol, or signal in the protocol. • Sweep/Scan detects attempts to map which of a host’s ports respond and therefore may be vulnerable. • Source signatures look for large volumes of traffic originating from a single IP. • Destination signatures looks for large volumes of traffic destined for a single IP.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

If you do not have an accurate baseline for your network, then when you implement DoS for the first time, be careful not to completely block network services. To prevent this, initially configure the DoS policy to log – but not block. Using the logs, you can analyze and determine normal and peak levels for each protocol. Then adjust the thresholds to comfortably, but not loosely, allow the usual peaks. Thresholds that are too high can allow your resources to be exhausted before the DoS policies trigger. Thresholds that are too low will cause FortiGate to drop normal traffic.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) Now we will take a look at some common types of DoS attacks. The first is called a ‘SYN flood’. In TCP, the client sends a ‘SYN’ signal to initiate a connection. The server must respond, then remember the start of the connection in RAM while it waits for the client to acknowledge (or ACK). Until ACK, the connection is only half-formed, so the attack won’t show up in a connection table. Normal clients will quickly ACK and begin to transmit data. But malicious clients continue – quickly or slowly, to avoid detection – to send more SYN packets, half-opening more connections, until the server’s table is full. Then, the server cannot accept more. It begins to ignore all new clients. Depending on the system, this attack can also damage hardware. (click) To defend against this, FortiGate acts as a pseudo-proxy. It waits until the client has finished connection build-up to form the back-end connection. If this doesn’t complete quickly, FortiGate begins to drop the attacker’s connection requests from the table.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) Another type of anomaly is an ICMP sweep. ICMP is used during troubleshooting: devices will respond with success or error messages. But attackers can use this to probe the network for valid routes and responsive hosts. (click) This provide information about your network before the attacker crafts more serious exploits.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) An individual DoS attack is a flood of traffic coming from a single address. It can originate from the Internet or even from your internal network. Typically a single device makes many connections or sessions, and possibly uses much bandwidth to a single location. (click) All four protocols in the DoS profile (ICMP, TCP, UDP, SCTP) have an anomaly sensor for the source. These are built to examine the traffic each IP is generating and compare that to the threshold value.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) A variation of this is the DDOS, or Distributed Denial of Service attack. It has many of the same characteristics. The main difference is that multiple devices are all attacking at the same time. This could be 5, or maybe 50, or 500 or more devices attacking together. (click) Remember earlier when we showed that despite FortiGate protecting the host, the resource could still become unavailable if the bandwidth to the ISP was consumed? Think about how these detections work. They do not trigger until the threshold is reached. Let’s say, for example, that the DoS sensor doesn’t trigger until 5000 sessions occur within 1 second. These 5000 sessions are allowed: first come, first served. So if multiple external devices are all generating connections to the same destination, attackers which are creating connections the fastest, will be the ones most likely to get the connections. Many of these DoS attacks can physically damage systems, so the goal is to prevent that from happening and prevent this kind of damage. But how can you find the right threshold? You must know what normal traffic thresholds are on your network – in other words, the baseline.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Everything we have shown so far is inline scanning: traffic passes through FortiGate from one interface to another. But you can also deploy FortiGate outside of the direct path of packets, in a one-arm topology with a monitor-only mechanism. This is also called “sniffer mode” because it detects but does not block. To do this, connect FortiGate to a switch’s SPAN or mirroring port. The switch will send a duplicate of egressing packets to FortiGate, which FortiGate then scans. Notice that because it’s scanning a copy – not the original packet – it can’t modify or block the original packet.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

When should you use one-arm IPS? Historically, when IPS scanning was first invented, it was slow. Old IPS could introduce high latency. So one-arm deployment was common, but IPS on an inline firewall wasn’t. Now, hardware performance is much better. And one-arm has a significant limitation: one-arm FortiGate cannot block traffic. Because it’s on a mirrored port on the switch, not directly in between the attacker and your protected network, FortiGate isn’t placed to intervene. So today, most people use one-arm only during testing or evaluation. Think of “one-arm” IPS as “log-don’t-block.”

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Before sniffer mode, the only way you could demonstrate a FortiGate without changing IP addresses was to put it transparently inline with the traffic. This could potentially disrupt the network if you didn’t understand the Layer 2 topology. But now, there is no risk.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Sniffer mode is enabled on a FortiGate’s physical interface, not a logical interface such as a VLAN. After you select “One-Arm Sniffer” on an interface, you can choose any security profile that uses the IPS engine. For example, you can use an application control profile if it is flow-based, since flow-based scans use the same engine as IPS. (Onearm DLP is also configurable, but via the CLI only.) FortiGate won’t allow you to choose proxy-based profiles that aren’t supported in one-arm inspection. Why aren’t all profiles/actions supported? It’s not technically possible. This is due to the nature of the topology and asynchronous scanning. To modify traffic or proxy connections, FortiGate must be in line – not out of band on a SPAN port – and stop the packet until it finishes scanning. That is, inspection must be in sync with the connection. However, one-arm scans after the interface has already forwarded the packet. Scanning and forwarding are out of sync. Since the packet has already egressed, FortiGate can’t proxy or block. That’s why it’s not possible to support all features in this mode.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Now let’s see some logs that are generated by IPS. Anomalies and signature matches have different logs associated with them. Since an anomaly’s name already gives information about the traffic and the attack, such as protocol and source address, many details in the logs aren’t needed. But you often will require information about which applications or operating systems are vulnerable. You also need to know the action – whether FortiGate blocked or simply monitored (“detected”) the attack. If you configured FortiGate to only monitor, you may need to forensically investigate the targeted host. This is where host-based tripwires can be useful.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

IPS sensors are not the only way that IPS can generate logs, however. When DoS policies generate logs, they are aggregated. When several incidents occur together, this reduces the number of log messages. In large attacks, the number of incidents can easily reach 100,000 in a few seconds. Generating a log entry for every packet that matches would completely utilize the CPU. So instead, FortiGate collapses incidents by periodically recording only one message for all of them, and noting the number of incidents. Here, the detection threshold was 50, and the total count is 75. So FortiGate doesn’t make 24 separate log entries (1 for each incident above 50). It’s just one log message.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

What commands can you use if IPS is dropping packets unexpectedly? In the CLI, use ‘diag ips anomaly list’ to show all hosts that are currently being limited by DoS policies, and by what signature. If there’s no matching traffic, then it will not display any output.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Another available diagnostic command is ‘diag autoupdate version’. This lists various IPS databases and engines that are installed on the FortiGate. It also displays the results of the last update attempt. So it can be useful if you suspect interruptions to FortiGuard connectivity.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Another command that can be used is troubleshoot the IPS is ‘diag test app ipsm’. For example, you could type ‘diag test app ipsm 99’.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

(slide contains animation) What does the IPSEngine actually do? Notice that if you run the ‘diag test app ipsm 5’ command, and if you have any kind of flow-based inspection profile, the CPU usage of the IPSEngine process drops dramatically, but doesn’t reach 0. This is because IPSEngine is responsible for all of the things we’ve shown in this class: intrusion protection, DoS policies and protocol decoders. It’s also responsible for application control, flow-based policies for antivirus, web filtering, email filtering, and DLP. So relatedly, it’s also responsible for session helpers. (click) Session helpers aren’t an inspection option; they are automatic. To stop them, you must stop IPSEngine.

DO NOT REPRINT © FORTINET

 Intrusion Prevention System

Here is a review of what we discussed. We showed: • The difference between a signature that matches a known attack, versus one that matches a traffic pattern anomaly • How protocol decoders find anomalies, and how this is different than proxy-based scans • Severity levels • How to configure IPS sensors, including ones with custom signatures • Denial of Service attacks, which are a type of anomaly • One-arm deployment, both its limitations and purpose • IPS logs • Diagnostic commands for IPS, including expected output, since some processes of the IPS engine are used by other scans