Troubleshooting Firewalls Eric Stuhl Senior Network Consultant Chesapeake NetCraftsmen
[email protected]
Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Packet Flow
Copyright 2005
Understanding the Packet Flow • • • •
To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/PIX®/FWSM, always – Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol – Determine the interfaces through which the flow passes
Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both
Copyright 2005
Example Flow • Flow –
SRC IP: 10.1.1.9
SRC Port: 11030
–
DST IP: 198.133.219.25 DST Port: 80
Protocol: TCP
• Interfaces –
Source: Inside
Destination: Outside
Servers
Client: 10.1.1.9
Packet Flow Eng
Accounting Outside Server: 198.133.219.25
With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside
Copyright 2005
Understanding the Packet Flow • Once the device and flow have been identified, walk the path of the packet through the device • The packet path through the firewall is illustrated in the next several slides • For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process
Copyright 2005
Packet Processing Flow Diagram • The diagram below will be referenced on the following slides; it is shown here enlarged for reference
Copyright 2005
Packet Processing: Ingress Interface
• Packet arrives on ingress interface • • •
Input counters incremented Software input queue is an indicator of load “No buffers” indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)
Copyright 2005
Packet Processing: Locate Connection
• Check first for existing connection • If connection exists, flow is matched; bypass ACL check • If no existing connection – TCP non-SYN packet, drop and log – TCP SYN or UDP packet, pass to ACL checks Established Connection: ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet: ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside
Copyright 2005
Packet Processing: ACL Check
• First packet in flow is processed through interface ACLs • ACLs are first match • First packet in flow matches ACE, incrementing hit count by one • Denied packets are dropped and logged Packet Permitted by ACL: ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL: ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside" Copyright 2005
Packet Processing: Match Translation
• • • • •
First packet in flow must match a translation rule* A quick route lookup is done only to determine egress interface Translation rule can be to NAT, or not to NAT NAT order of operations dictates what happens with overlapping translation rules Once translation rule is matched, connection is created
Translation Exists: ASA-5540# show xlate debug NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global) ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80 ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80 Copyright 2005
Translation and NAT Order of Operations
First Match
1.
nat 0 access-list (nat-exempt)
2.
Match existing xlates
3.
Match static commands (Cisco ASA/PIX first match; FWSM best match)
4.
–
Static NAT with and without access-list
–
Static PAT with and without access-list
Match nat commands –
nat access-list (first match)
–
nat (best match) •
If the ID is 0, create an identity xlate
•
Use global pool for dynamic NAT
•
Use global pool for dynamic PAT Copyright 2005
Packet Processing: Inspections/Sec Checks
• • • • •
Inspections are applied to ensure protocol compliance (Optional) Customized AIC inspections NAT embedded IPs in payload Additional security checks are applied to the packet (Optional) Packets passed to Content Security and Control (CSC) Module
Syslog from Packets Denied by Security Check: ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP Copyright 2005
Packet Processing: NAT IP Header
• • • •
Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module
Copyright 2005
Packet Processing: Egress Interface
• Packet is “virtually” forwarded to egress interface (i.e., not forwarded to the driver yet) • Egress interface is determined first by translation rules • If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface • Example:
172.16.0.0/16
Outside DMZ
Inside
Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics
172.16.12.0/24 172.16.12.4
static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0 Copyright 2005
Packet Processing: L3 Route Lookup
• Once on egress interface, an interface route lookup is performed • Only routes pointing out the egress interface are eligible • Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface Syslog from Packet on Egress Interface with No Route Pointing Out Interface: ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
Copyright 2005
Packet Processing: L2 Address Lookup
• Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed • Layer 2 rewrite of MAC header • If Layer 2 resolution fails—no syslog • show arp will not display an entry for the L3 next hop • debug arp will indicate if we are not receiving an ARP reply
Copyright 2005
Packet Processing: Transmit Packet
• Packet is transmitted on wire • Interface counters will increment on interface • Output hardware and software queues indicate buffering at driver level, interface is busy ASA-5540# show interface gb-ethernet0 interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0) Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Cisco ASA/PIX—Understanding the Architecture • Cisco ASA/PIX platforms process all packets in software (via the central CPU) – All packets are processed first in…usually also first out
• No software limits on the number of ACEs (rules) that can be configured. – Each ACE takes a minimum of 212 bytes of RAM.
• Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)
Copyright 2005
Classifier in Multimode • FWSM has a single MAC address for all interfaces • Cisco ASA/PIX has single MAC for ‘shared’ interfaces (physical interfaces have unique MACs) – Cisco ASA/PIX 7.2 introduces an option to change this
• When the firewall receives a packet, it must ‘classify’ it to determine where to send the packet • Packets are classified based on the following – Unique ingress interface/VLAN – Packet’s destination IP matches a global IP
Copyright 2005
Classifier in Multimode Example • Inbound traffic is ‘classified’ to context CTX3, based on the global IP in the static FWSM
VLAN 4
CTX1
.1
10.1.1.2 Inside VLAN 5
CTX2
.2
CTX3
.3
10.1.2.2
VLAN 3—10.14.3.x
Inside
DST IP
SRC IP
10.14.3.89
192.168.5.4
Inbound Packet Outside MSFC
Inside VLAN 6 10.1.3.2
Shared Interface
static (inside,outside) 10.14.3.89 10.1.3.2 Copyright 2005
Classifier in Multimode • If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80
*Added to FWSM 3.1 Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Failover Basics • Active/standby vs. primary/secondary • Serial vs. LAN failover • Stateful failover (optional) • A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall • Both firewalls swap MAC and IP addresses when a failover occurs • Level 1 syslogs will give reason of failover
Internet
Stateful LAN/Serial Secondary (Standby)
Primary (Active)
Corp
Copyright 2005
Verifying Failover Configuration PIX(config)# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Primary Failover LAN Interface: Failover Ethernet5 (up) Unit Poll frequency 1 seconds, holdtime 3 seconds Interface Poll frequency 5 seconds, holdtime 25 seconds Monitored Interfaces 2 of 250 maximum Version: Ours 7.2(3), Mate 7.2(2) Last Failover at: 18:30:43 UTC Apr 12 2007 This host: Primary - Active Active time: 5371 (sec) Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal Stateful Failover Logical Update Statistics Link : Failover Ethernet5 (up) Stateful Obj xmit xerr General 86 0 sys cmd 74 0
rcv 73 73
Interface Monitoring
rerr 0 0 Copyright 2005
What Triggers a Failover? • Power loss/reload (this includes crashes) on the active firewall • SSM interface/module failure • The standby becoming “healthier” than the active firewall
Copyright 2005
What Triggers a Failover? • Two consecutive “hello” messages missed on any monitored interface forces the interface into testing mode • Both units first verify the link status on the interface • Next, both units execute the following tests – Network activity test – ARP test – Broadcast ping test
• The first test passed causes the interface on that unit to be marked “healthy”; only if all tests “fail” will the interface be marked “failed” Copyright 2005
What to Do After a Failover • Always check the syslogs to determine root cause • Example: switch port failed on inside interface of active firewall Syslogs from Primary (Active) Firewall ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status ‘Down’ on interface 1 ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier
Syslogs from Secondary (Standby) Firewall ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active
Copyright 2005
What to Do After a Failover • Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state • This information is not saved across reboots
ASA# show failover state This host
-
Other host -
State Primary Failed Secondary Active
Last Failure Reason
Date/Time
Ifc Failure Inside: Failed
12:56:00 UTC May 6 2007
None
====Configuration State=== Sync Done ====Communication State=== Mac set
Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Troubleshooting Tools • • • • •
Syslogs Debug commands Show commands Packet capture Packet tracer
Copyright 2005
Uses of Syslogs • Primary mechanism to record traffic to and through the firewall • The best troubleshooting tool available Archival Purposes
Debugging Purposes
Console Syslog Server
Internet
SSH Client Trap
Syslog.
Buffered
SNMP Server
Copyright 2005
ASA Syslog Level vs. Number of Messages
Number of Messages (SUM)
Log Level
Description
0
Ver. 6.3
Ver. 7.0
Ver. 7.2
Ver. 8.0
Ver. 8.1
Emergencies
0
0
0
0
0
1
Alerts
41 (41)
62 (62)
77 (77)
78 (78)
87 (87)
2
Critical
21 (62)
29 (91)
35 (112)
49 (127)
50 (137)
3
Errors
74 (136)
274 (365)
334 (446)
361 (488)
363 (500)
4
Warnings
56 (192)
179 (544)
267 (713)
280 (768)
281 (781)
5
Notifications
21 (213)
161 (705)
206 (919)
216 (984)
218 (999)
6
Informational
95 (308)
234 (939)
302 (1221)
335 (1319)
337 (1336)
7
Debugging
15 (323)
217 (1156)
258 (1479)
266 (1585)
267 (1603)
Copyright 2005
FWSM Syslog Level vs. Number of Messages
Number of Messages (SUM)
Log Level
Description
0
Ver. 2.3
Ver. 3.1
Ver. 3.2
Ver. 4.0
Emergencies
0
0
0
0
1
Alerts
58 (58)
67 (67)
67 (67)
67 (67)
2
Critical
21 (79)
29 (96)
29 (96)
29 (96)
3
Errors
94 (173)
305 (401)
306 (402)
318 (414)
4
Warnings
131 (304)
194 (595)
196 (598)
199 (613)
5
Notifications
26 (330)
167 (762)
169 (767)
178 (791)
6
Informational
116 (446)
245 (1007)
248 (1015)
255 (1046)
7
Debugging
23 (469)
225 (1232)
225 (1240)
226 (1272)
Copyright 2005
What Are Modifiable Syslog Levels? [no] logging message level
• Modifiable syslog levels – Allows one to move any syslog message to any level
• Problem – You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)
Levels 0—Emergency 1—Alert 2—Critical 3—Errors 4—Warnings 5—Notifications
%PIX-7-111009: User ‘johndoe’ executed cmd: show run
6—Informational 7—Debugging
The problem is we don’t want to log all 1602 other syslogs that are generated at debug level Copyright 2005
How to Create Modifiable Syslog Levels Solution [no] logging message level
• Lower syslog message 111009 to level 3 (error) – ASA(config)# logging message 111009 level 3
– Or – ASA(config)# logging message 111009 level error
• Now our syslog looks as follows – %ASA-3-111009: User ‘johndoe’ executed cmd: show run
• To restore the default syslog level – ASA(config)# no logging message 111009 level error
– Or – ASA(config)# logging message 111009 level 7 http://www.cisco.com/en/US/docs/security/asa/asa80/system/m essage/logmsgs.html Copyright 2005
Debug Commands 1. Debugs should not be the first choice to troubleshoot a problem 2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution 3. Debugs are not conditional* 4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0
Copyright 2005
Debug ICMP Trace Internet http://www.cisco.com
• Valuable tool used to troubleshoot connectivity issues • Provides interface and translation information to quickly determine flow • Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled Example debug icmp trace output ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2 Copyright 2005
Logging Debugs to Syslog • Problem – Log only debug output to syslog
• Solution – Create a logging list with only syslog ID 711001 – Enable debug output to syslogs – Log on the logging list
ASA(config)# logging list C-MUG message 711001 ASA(config)# logging debug-trace ASA(config)# logging trap C-MUG
Copyright 2005
Show Output Filters show | begin|include|exclude|grep [-v]
• Use output filters to filter the output of show command to only the information you want to see • To use them, at the end of show , use the pipe character “|” followed by – begin
match of output – include – exclude – grep – grep –v
Start displaying the output beginning at the first the RegEx, and continue to display the remaining Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude
Copyright 2005
Example: Show Output Filters show | begin|include|exclude|grep [-v]
Examples • Display the interface stats starting with the ‘inside’ interface – show interface | begin inside
• Display the access-list entries that contain address 10.1.1.5 –
show access-list | grep 10.1.1.5
• Display the config, except for the access-lists –
show run | exclude access-list
• Display only access-list entries that have non-zero hitcounts –
show access-list | grep –v hitcnt=0
• Display a count of the number of connections each host has –
show local-host | include host|count/limit
Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted Copyright 2005
Show CPU Usage • Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets • FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled • The show cpu usage command displays the CPU over time as a running average pixfirewall# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4% *First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1) Copyright 2005
Show Traffic • The show traffic command displays the traffic received and transmitted out each interface of the firewall ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec Copyright 2005
Show Xlate and Show Xlate Debug • The show xlate command displays information about the translations through the firewall • You can limit the output to just the local or global IP ASA# show xlate 2 in use, 2381 most used Global 172.18.124.68 Local 10.1.1.9 PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101) “debug” Adds interface names, idle and xlate ASA# show xlate debug timeouts 2 in use, 2381 most used Flags: D - DNS, d - dump, I - identity, i - inside, n - no random, o - outside, r - portmap, s - static NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00 TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30 Copyright 2005
Show Conn and Show Conn Detail ‘real’ Interface names added in 7.2(4), 8.0(4)
Idle Time, Bytes Transferred
Connection Flags
ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –
ASA# show conn detail 2 in use, 64511 most used Flags: A B E G i k O R R s X
-
“detail” Adds uptime and timeout in 7.2(4), 8.0(4)
awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, outside back connection, F - outside FIN, f - inside FIN, group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, incomplete, J - GTP, j - GTP data, K - GTP t3-response Skinny media, M - SMTP data, m - SIP media, n - GUP outbound data, P - inside back connection, q - SQL*Net data, outside acknowledged FIN, UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431 Copyright 2005
Example—Connection Build Up 1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags “saA” 2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show “A” 3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered “up” (U flag) 4. The outside device sends the first data packet; the connection is updated and an “I” is added to the flags to indicate the firewall received Inbound data on that connection 5. Finally, the inside device has sent a data packet and the connection is updated to include the “O” flag
3 5 1
SYN+ACK ACK SYN Data
Connection Flags 42
saA U UI UIO A Inside Client
Outside Server Copyright 2005
Example—Connection Teardown 1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an “f” to indicate that the FIN was received on the Inside interface 2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show “UfFR” 3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
1 3
FIN+ACK FIN ACK
Connection Flags 2
UfUfFR UfFRr Inside Client
Outside Server Copyright 2005
Connection Flags—Quick Reference Outbound Connection
Inbound Connection
Copyright 2005
TCP Connection Termination Reasons • If a TCP connection is built through the firewall, it will always have a teardown reason • The TCP teardown syslog is logged at level 6 • If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check the teardown reason ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]
Copyright 2005
TCP Connection Termination Reasons— Quick Reference Reason
Description
Conn-Timeout
Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout
Deny Terminate
Flow Was Terminated by Application Inspection
Failover Primary Closed
The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit
FIN Timeout
Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout
Flow Closed by Inspection
Flow Was Terminated by Inspection Feature
Flow Terminated by IPS
Flow Was Terminated by IPS
Flow Reset by IPS
Flow Was Reset by IPS
Flow Terminated by TCP Intercept
Flow Was Terminated by TCP Intercept
Invalid SYN
SYN Packet Not Valid
Idle Timeout
Connection Timed Out Because It Was Idle Longer Than the Timeout Value
IPS Fail-Close
Flow Was Terminated Due to IPS Card Down
SYN Control
Back Channel Initiation from Wrong Side Copyright 2005
TCP Connection Termination Reasons— Quick Reference (Cont.) Reason
Description
SYN Timeout
Force Termination After Two Minutes Awaiting Three-Way Handshake Completion
TCP Bad Retransmission
Connection Terminated Because of Bad TCP Retransmission
TCP Fins
Normal Close Down Sequence
TCP Invalid SYN
Invalid TCP SYN Packet
TCP Reset-I
TCP Reset Was Sent From the Inside Host
TCP Reset-O
TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Size Variation
Connection Terminated Due to a Variation in the TCP Window Size
Tunnel Has Been Torn Down
Flow Terminated Because Tunnel Is Down
Uauth Deny
Connection Denied by URL Filtering Server
Unknown
Catch-All Error
Xlate Clear
User Executed the ‘Clear Xlate’ Command Copyright 2005
show local-host • A local-host entry is created for any IP tracked through the firewall • It groups the xlates, connections, and AAA information • Very useful for seeing the connections terminating on servers ASA# show local-host 10.1.1.9 detail Interface inside: 1131 active, 2042 maximum active, 0 denied local host: , TCP connection count/limit = 1/unlimited TCP embryonic count = 0 TCP intercept watermark = 50 UDP connection count/limit = 0/unlimited AAA: user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10) absolute timeout: 0:05:00 inactivity timeout: 0:00:00 Xlate(s): Global 172.18.124.69 Local 10.1.1.9 Conn(s): TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO Copyright 2005
show service-policy • The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: ftp, packet 43, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 562, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 349, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349 Copyright 2005
show service-policy flow • Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) • Eventually all policies will be in MPF
ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside_policy Class-map: inbound_class Match: access-list telnet_inbound Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet Action: Output flow: set connection timeout tcp 0:05:00
Copyright 2005
show asp drop • Packets dropped in the Accelerated Security Path (ASP) will increment a counter • Frame drop counters are per packet, flow drops are per flow • Some counters have corresponding syslogs ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) Invalid tcp length (invalid-tcp-hdr-length) Invalid udp length (invalid-udp-length) No valid adjacency (no-adjacency) No route to host (no-route) Reverse-path verify failed (rpf-violated) Flow is denied by access rule (acl-drop) First TCP packet not SYN (tcp-not-syn) Bad TCP flags (bad-tcp-flags) TCP option list invalid (tcp-bad-option-list) TCP MSS was too large (tcp-mss-exceeded) Bad TCP Checksum (bad-tcp-cksum)
10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893
*Drop counters are documented in the CMD Ref, under ‘show asp drop’
Copyright 2005
Packet Capture capture [access-list ] [buffer ] [ethernet-type ] [interface ] [packet-length ] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user ] [match {host | | any} [eq | lt |gt ] {host | | any} [eq | lt | gt ]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count ]]
• Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the “debug packet” command • 7.2(3) and 8.0(3) added a ‘real-time’ option • ASDM 6.0 adds a ‘capture wizard’ • Capture sniffs packets on an interface that match an ACL, or match line • Key steps – Create an ACL that will match interesting traffic – Define the capture and bind it to an access-list and interface – View the capture on the firewall, or copy it off in .pcap format
Copyright 2005
Packet Capture (Cont.) • Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface • Capture buffer saved in RAM (default size 512KB) • Default is to stop capturing when buffer is full • Default packet length is 1518 bytes • Copy captures off via TFTP or HTTPS
Capture In Inside
Capture Out Outside
Copyright 2005
Where Packets Are Captured in Packet Flow
Ingress Packets Captured
Egress Packets Captured
• Packets are captured at the first and last points they can be in the flow • Ingress packets are captured before any packet processing has been done on them • Egress packets are captured after all processing (excluding L2 source MAC rewrite) Copyright 2005
Capture Command: Example • Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (197.133.219.25); the user is getting PATed to 192.168.2.2 Capture In Inside 10.1.3.2
10.1.3.2
Capture Out Outside
www.cisco.com
Internet
192.168.2.2
198.133.219.25
Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program Copyright 2005
Capture Command: Example •
Step 1: Create ACL for both inside and outside interface –
! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2
•
Step 2: Create captures on both inside and outside interface –
• •
capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518
Step 3: Have inside user access www.cisco.com Step 4: Copy the captures off to a TFTP server –
! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap
– Or copy using https: –
https:///capture/out/pcap Copyright 2005
Packet Capture: Example •
Step 5: Analyze captures with sniffer program
Outside CAP
Inside CAP
Outbound SYN, No SYN+ACK
Copyright 2005
Capturing Packets Dropped by the ASP • Capture all packets dropped by the ASP – ASA# capture drops type asp-drop all
• Capture on a specific drop reason – ASA# capture drops type asp-drop invalid-tcp-hdr-length
ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags
Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags
Copyright 2005
Packet Tracer • Packet tracer is the future of troubleshooting configuration issues (and many other issues) • Introduced in version 7.2 and ASDM 5.2 • A packet can be traced by: – Defining the packet characteristics via the CLI – Capturing the packets using the ‘trace’ option
Copyright 2005
Packet Tracer: Overview • A packet tagged with the ‘trace’ option is injected into the interface, and processed in the data-plane • Each action taken on the packet is recorded in the packet itself • When the packet reaches the egress interface, or is dropped, it is punted to the control-plane • The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
Copyright 2005
Packet Tracer: Creating Packet via CLI • From the CLI, define the input interface along with source and destination IPs and ports packet-tracer input
• Example—Trace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25) ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80
Copyright 2005
Packet Tracer: Example Output ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80 Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information: Copyright 2005
Packet Tracer: Example Output (Cont.) ... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ... Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path 198.133.219.25.80: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: ...
• Then select that packet to be traced
ASA# show capture inside trace packet-number 4
S S . P .
ack ack ack Ack
.
Copyright 2005
Packet Tracer: ASDM • ASDM includes a nice GUI front-end to the packet tracer tool • It is located off the Tools menu • Input the packets characteristics in the top half • Actions taken on the packet are shown in the bottom half, along with associated config and links back to modify that config entry in ASDM
Copyright 2005
Packet Tracer: ASDM (Screen Shot)
Define Packet
Action Matching Config
Link Back to Edit Rule
Final Result
Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Case Study • Intermittent Access to Web Server
Copyright 2005
Case Study: Intermittent Access to Web Server Problem • Most external clients are not able to load company’s web page
NATed to 10.1.1.50
HTTP Requests to 192.168.1.50
Internet Web Server
ASA-5510
10.1.1.50
Clients
Copyright 2005
Case Study: Intermittent Access to Web Server
Traffic Spike
Copyright 2005
Case Study: Intermittent Access to Web Server • show perfmon indicates high number of embryonic connections
ASA-5510# show perfmon PERFMON STATS: Xlates Connections TCP Conns UDP Conns URL Access URL Server Req TCP Fixup TCP Intercept Established Conns TCP Intercept Attempts TCP Embryonic Conns Timeout HTTP Fixup FTP Fixup AAA Authen AAA Author AAA Account VALID CONNS RATE in TCP INTERCEPT:
Current 0/s 2059/s 2059/s 0/s 0/s 0/s 0/s 0/s 0/s 1092/s 0/s 0/s 0/s 0/s 0/s Current N/A
Average 0/s 299/s 299/s 0/s 0/s 0/s 0/s 0/s 0/s 4/s 0/s 0/s 0/s 0/s 0/s Average 95.00%
ASA-5510#
Copyright 2005
Case Study: Intermittent Access to Web Server • Issue show conn to see ‘who’ is creating the connections Random Sources
Embryonic Conns
ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .
Copyright 2005
Case Study: Intermittent Access to Web Server
Traffic Permitted Connection Count Jumps
SYN Flood Detected
Copyright 2005
Case Study: Intermittent Access to Web Server • Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside
Copyright 2005
Case Study: Intermittent Access to Web Server
TCP Intercept applied
Few clients represent 50+ % of traffic
Copyright 2005
Case Study: Intermittent Access to Web Server • Apply per-client-max option to limit the number of connections any single client can establish
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside
Copyright 2005
Case Study: Intermittent Access to Web Server per-client-max
TCP Intercept
Copyright 2005
Case Study: Intermittent Access to Web Server
Attacks Being Mitigated by ASA
Attacks Still Occurring Copyright 2005
Case Study • Poor Voice Quality
Copyright 2005
Case Study: Poor Voice Quality Problem • Poor Outbound Voice Quality at SOHO sites
Outbound RTP Stream
100 Mbps
100 Mbps
Cable Modem
2 Mbps
WAN
ASA-5505
Copyright 2005
Case Study: Poor Voice Quality Solution: Traffic Shaping • What is Traffic Shaping, and why is it needed here? • Why won’t Policing work? • Why won’t Priority Queuing alone work? Shape to 2 Mbps
Cable Modem 100 Mbps
ASA-5505
WAN 2 Mbps
100 Mbps
Copyright 2005
Case Study: Poor Voice Quality – Configuration Example (Traffic Shaping) Solution Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface. class-map voice-traffic match dscp af13 ef ! policy-map qos_class_policy class voice-traffic priority ! policy-map qos_outside_policy class class-default shape average 2000000 service-policy qos_class_policy ! service-policy qos_outside_policy interface outside
To view statistics on the operation of the shaper, use the command show service-policy shape Copyright 2005
Case Study: Poor Voice Quality Things to keep in mind: Shaping can only be applied to the class ‘class-default’ Shaping only works in the outbound direction on an interface The shaping value is in bits per second, and must be a multiple of 8000 The shaping policy is applied to all sub-interfaces on a physical interface Not supported on the ASA-5580 platform Not supported in Transparent or Multi-context mode Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Tools • ASDM • Output interpreter • Online learning modules
Copyright 2005
ASDM • Run as a standalone application using the ADSM Launcher • This allows for one-stop access to multiple firewalls • ASDM 6.0 adds “Upgrade Wizard” to upgrade ASA and ASDM software direct from cisco.com • ASDM 6.1 works with both ASA 8.1 and 8.0 releases • ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
Copyright 2005
ASDM Home Page
Device Information
CPU, Memory, Conns/Sec, Interface Traffic
Real-Time Syslogs Copyright 2005
Using ASDM for Monitoring
Up to Four Different Graphs Can Be Displayed
Great for Monitoring Trends
Copyright 2005
ASDM: Editing Rules from the Log Viewer
Select Log Entry from Viewer
Right-Click on Message to View or Edit Associated Rule
Copyright 2005
ASDM: Syslogs Explained
Copyright 2005
ASDM 6.0 Drag-and-drop and inplace editing for simplified policy editing User interface customization with dockable windows and toolbars New Firewall Dashboard that provides at-a-glance status of firewall services Live ACL hitcount in firewall rule table for easy policy auditing
Copyright 2005
Output Interpreter Linked Off the Technical Support and Documentation— Tools and Resources Section on CCO
Great Tool for Catching Configuration Errors
Paste in the “show run” Output and Hit “Submit”
Copyright 2005
Output Interpreter: Example Output
Warning: Unused Statics
Warning: Unapplied Crypto Map
Warning: Invalid Crypto Map
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl Copyright 2005
Online Learning Modules on CCO • Great way to learn about new features in the ASA • Located on CCO • From http://www.cisco.com select: Products and Solutions – Security • ASA 5500 Series Adaptive Security Appliances
– Training resources » Online learning modules
• Direct link – http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.html
Copyright 2005
Agenda • • • • • • •
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Copyright 2005
Cisco ASA/PIX/FWSM Best Practices • Enable ip verify reverse-path on all interfaces • Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-clientmax
• Configure logging to syslog server • Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you don’t want to see • Disable telnet access! Use SSH for management access • Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback Copyright 2005
Cisco ASA/PIX/FWSM Best Practices • Restrict DMZ access inbound to your internal networks • Baseline CPU load, connection counts, xlate counts, and traffic (per interface) • Run the latest maintenance release in your train • Upgrade major feature trains only when you need new features, or after train has ‘matured’
Copyright 2005
Cisco ASA/PIX Release Process Interim Images
7.0(2.1)
7.0(2)
7.0(2.2)
7.0(4)
7.0(2.3)
7.0(5)
7.0(6)
GD
GD
7.0(7)
7.0(8)
7.1(2)
Maintenance Trains
Future 7.0(9) EoL
7.2(2)
Bug Fixes Waterfall Down
7.2(3)
7.2(4) 8.0(3)
7.2(5) 8.0(4) 8.1(2)
7.0(1)
7.1(1)
7.2(1)
8.0(2)
8.1(1)
8.2(1)
Major Feature Releases Time Copyright 2005
Cisco PIX - End of Sale / End of Life Milestones
Cisco has Announced the End-of-Sale and End-of-Life Dates for Cisco PIX Security Appliances End of Sale: July 28, 2008 Last day of sale for software, accessories, and licenses: January 28, 2009 End of Software Maintenance Releases: July 28, 2009 End of Support / End of Life: July 27, 2013
Copyright 2005
Gotcha’s in Upgrading to 7.0 • Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode • Upgrading from monitor mode requires you to copy the 7.0 image over twice – Once from monitor mode (to boot 7.0 and format flash) – Once after 7.0 is up and flash has been formatted (to save image in flash)
• The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI • If there were any errors during the config conversion process, view them by issuing – show startup-config errors Copyright 2005
Cisco PIX Password Recovery • Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP • Password recovery will also remove any AAA commands • A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible – Example: np63.bin for Cisco PIX 6.3 and prior releases
http://www.cisco.com/warp/public/110/34.shtml Copyright 2005
Example: Cisco PIX Password Recovery Example TFTP Inside monitor> interface 1 monitor> address 14.36.1.88 monitor> file np63.bin 172.18.108.26 14.36.1.88 monitor> gateway 14.36.1.1 monitor> server 172.18.108.26 monitor> tftp tftp
[email protected] via 14.36.1.1............... Received 92160 bytes Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration: enable password 8Ry2YjIyt7RRXU24 encrypted passwd 2KFQnbNIdI.2KYOU encrypted aaa authentication serial console LOCAL aaa authentication telnet console LOCAL aaa authentication ssh console LOCAL aaa authentication enable console LOCAL Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. Rebooting.. Copyright 2005
Case Study • Out-of-order packet buffering
Copyright 2005
Case Study: Out-of-Order Packets • Inspections require ordered packets • Packets sent to the SSM (AIP and CSC) require ordered packets • Cisco ASA/PIX will buffer up to three packets by default • Buffering can be increased on ASA by using the ‘queue-limit’ option under the tcp-map
Copyright 2005
Case Study: Out-of-Order Packets Problem • Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows • If the out-of-order packet buffer isn’t large enough, traffic is dropped and packets must be retransmitted Inside
192.168.1.30
Outside
Client
10.16.9.2 Server
Dropped on Network Buffer Dropped by Firewall
Packet 10 Packet 11 Packet 12 Packet 13 Packet 14 Packet 15
Copyright 2005
Case Study: Out-of-Order Packet Buffering Example • How to detect? ASA# show asp drop Frame drop: ... TCP packet SEQ past window TCP packet buffer full ...
46331 90943
How to fix? access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global Copyright 2005
Case Study: Out-of-Order Packet Buffering Example • How to verify? ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0
Copyright 2005
Case Study • TCP MSS (Maximum Segment Size)
Copyright 2005
Case Study: TCP MSS • MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet • The MSS is set in the SYN packets • The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS
Copyright 2005
Case Study: TCP MSS Problem • Some servers have broken TCP stacks and ignore the MSS advertised by the Client • The firewall will drop packets that exceed the advertised MSS Outside
Inside
192.168.1.30 Client
10.16.9.2 Server
SYN
MSS=1380
SYN+ACK MSS=1400
DATA=1390 Copyright 2005
Case Study: TCP MSS Example • How to detect? ASA# show asp drop Frame drop: TCP MSS was too large
943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
How to fix? access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global Copyright 2005
Case Study: TCP MSS Example • How to verify? ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown
How else could you verify?
Copyright 2005
Case Study • Out of Memory
Copyright 2005
Case Study: Out of Memory Problem • Users are unable to access the Internet • No new connections are working • All old (long lived) connections continue to work
Step 1: Check the Syslogs %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available Hardware:
PIX-515E, 64 MB RAM
pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ---------------------------Total memory: 67108864 bytes Copyright 2005
Case Study: Out of Memory Step 3: What Eats Up Memory (RAM) on the Cisco PIX? • • • • •
Cisco PIX image (run from RAM) Configuration IPSec database Xlates (translations) Connections
Step 4: Let’s Check the Translations
What Can Eat Up 64 MB on a Cisco PIX515E?
A Small Global Pool Is Used, Overloading to a PAT Address
pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
Varied Source IPs Copyright 2005
Case Study: Out of Memory Step 5: Check the Connections pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . .
Q: Why is the connection count so high? Copyright 2005
Case Study: Out of Memory
Inside
Traffic Flow
pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec
Outside
Take a Look at the Traffic Load
• Vast majority of traffic is coming in the inside interface and going out the outside interface Copyright 2005
Case Study: Out of Memory Step 6: Review What We Know and Take Action pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used
Conn Count Is Very High, but xlate Count Is Low Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s)
Copyright 2005
Case Study: Out of Memory Step 7: Find the Host(s) Generating All the Connections pixfirewall# show local-host | local host: , TCP connection count/limit UDP connection count/limit local host: , TCP connection count/limit UDP connection count/limit local host: , TCP connection count/limit UDP connection count/limit . . . local host: , TCP connection count/limit UDP connection count/limit
include host|count/limit = 0/unlimited = 0/unlimited = 2/unlimited = 0/unlimited
Only Show Lines That Have the Word “host” or “count/limit” in Them
= 0/unlimited = 0/unlimited
= 146608/unlimited = 0/unlimited
Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections Copyright 2005
Case Study: Out of Memory Step 8: Now That We Found the Host, Let’s Look at the Connections It Is Generating pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: , TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections UDP connection count/limit = 0/unlimited Are Embryonic Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags saA TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags saA TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags saA Connections to Random TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags saA Destinations on TCP/135– TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags saA MS0:01:43 Blaster Bytes 0 flags saA TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags saA TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags saA TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags saA . . . Copyright 2005
Case Study: Out of Memory • Cisco PIX provides two methods to limit the number of connections per host – TCP intercept – Max connections
Question: Which One Can Be Used Here?
• TCP intercept won’t help because the source address is valid • Limiting the maximum number of connections each internal host can have is the only option
Copyright 2005
Case Study: Out of Memory Step 9: Limit Infected Host(s) Impact on Network • Configure the MAX TCP connections for NATed hosts to be 50 pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
• Note: The local-host must be cleared before the new connection limits are applied pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: , The Infected Host Is TCP connection count/limit = 50/50 TCP embryonic count = 50 Limited to 50 TCP TCP intercept watermark = unlimited Connections UDP connection count/limit = 0/unlimited . . .
Copyright 2005
Case Study: Out of Memory Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ---------------------------Total memory: 67108864 bytes
• Things look much better now • Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus? nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Copyright 2005
