VPN Troubleshooting for Checkpoint

VPN TROUBLESHOOTING: REFFER: http://www.cpug.org/forums/vpns-virtual-private-networks/4764-vpn-trouble-shooting.html Basics: IKE negotiation consists of two phases - Phase I (Main mode which is six packets) and Phase II (Quick Mode which is three packets).

The $FWDIR/log/ike.elg file contains this information ( once

debugging is enabled).

To enable debugging, you need to login to your firewall and enter the command "vpn debug on Check Point have a tool called

;

vpn debug ikeon" or "vpn debug trunc".

IKEView.exe which parses the information of ike.elg into a GUI making this easier to view.

Note that another useful tool is "vpn debug on mon" which writes all of the IKE captured data into a file ikemonitor.snoop which you can open with wireshark or ethereal.

PHASE1: negotiates encryption methods (DES/3DES/AES etc), the key length, the hash Algorithm (MD5/SHA1) and creates a key to protect the messages of the exchange . It does this in 5 stages: 1. 2. 3. 4. 5.

Peers Authenticate using Certificates or a pre-shared secret. Each peer generates a private Diffie-Hellman key from random bits and from that derives a DH public key. These are then exchanged. Each peer generates a shared secret from its private key and its peers public key, this is the DH key. The peers exchange DH Key material (random bits and mathematical data) and methods for PhaseII are agreed for encryption and integrity. Each side generates a symmetric key (based upon the DH key and key material exchanged).

In IkeView under the IP address of the peer , open > "P1 Main Mode ==>" for outgoing or "P1 Main Mode " for outgoing or "P2 Quick Mode 224.0.0.18: VRRPv2-adver 20: vrid 100 pri 100 [tos 0xc0] 00:46:12.399982 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 100 pri 100 [tos 0xc0] 00:46:13.479985 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 100 pri 100 [tos 0xc0] 00:46:14.560007 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 100 pri 0 [tos 0xc0]

If both firewalls are broadcasting vrrp, and the packets are not seen by the other firewall, there could be a communication problem between the firewalls. Also ensure that the vrid matches on both firewalls. Proper VRRP failovers usually only cause 1 or 2 packets lost . VRRP multicast address is 224.0.0.18 To capture vrrp traffic in fw monitor: fw monitor -e “accept ip_p = 112;” Clish show vrrp This will show you which devices are in master and backup Example: PrimaryFW-A> sh vrrp VRRP State Flags: On 6 interface enabled 6 virtual routers configured 0 in Init state 0 in Backup state 6 in Master state PrimaryFW-A> PrimaryFW-A> exit Bye. PrimaryFW-A[admin]# SecondaryFW-B[admin]# iclid SecondaryFW-B> sh vrrp

VRRP State Flags: On 6 interface enabled 6 virtual routers configured 0 in Init state 4 in Backup state 2 in Master state SecondaryFW-B> SecondaryFW-B> exit

show vrrp interfaces Detailed configuration of VRRP, including priority, hello interval, and VRID clish -c "show interfacemonitor" Displays interface transitions cphaprob -i list Displays Checkpoint critical processes and their timeouts. To log critical process failures: ipsctl -w net:log:partner:status:debug 1 That will log to the console and to /var/log/messages. If you want to turn off: ipsctl -w net:log:sink:console 0 To change the timeout value of a monitored process: cphaprob -d [device] -t [timeout] -s [state] -p register

Resolving local logging issues on Checkpoint If logs are not appearing in Smartview Tracker, they are probably logging locally. To determine if logs are being stored locally on the gateway, go to $FWDIR/log. Locate the fw.log file and see if it's size is incrementing. There may also be additional fw*.log files that have rolled over. To resolve the issue, first try restarting the MLM (in a Provider environment or the Log Services in a Smartcenter Server environment). Next, restart the firewall services on the gateway (fw kill fwd followed by fwd). If that does not work, try restarting the firewall. Once resolved, you can pull the stored logs from the gateway by running "fw fetchlog " from the log server. In R70, there is also an option to fetch logs in Smartview Tracker (Tools>Remote Files Mgmt)

Configuring SNMP on SPLAT step 1: service snmpd restart step 2: edit /etc/snmp/snmpd.users.conf and replace public with your actual snmp community string step 3: service snmpd restart step 4: netstat -an | grep 161 for checkpoint snmpd port 260: step 1: modify the $FWDIR/conf/snmp.C file and place the actual snmp community inside the read and write (). If you leave the write empty, it will use "private" as the community string. This is a security risk. step 2: run sysconfig and start the checkpoint snmpd extension step 3: perform cpstop;cpstart step 4: netstat -an | grep 260

Creating a Read Only SPLAT user Creating a user 1. SSH to the firewall where account will be setup on. 2. From the command line type ―adduser ‖, here we will add the user with username testuser. The command should read ―adduser testuser‖ 3. Input the desired password when prompted to do so Changing the users shell 1. Open the passwd file for editing by typing ―vi /etc/passwd‖ 2. Find the line corresponding to the user you just created. If you have created a user with username ―testuser‖, the line you are looking for is ―testuser:x:0:0::/home/test:/bin/cpshell‖ 3. Change the users shell, to do this we will change ―/bin/cpshell‖ to ―/path/to/shell‖.

Before the change the line should read: ―testuser:x:0:0::/home/test:/bin/cpshell‖ After the change the line will read: ―testuser:x:0:0::/home/test:/etc/scripts/myshell.sh‖

Checkpoint port list TCP 256: CA and DH key exchange, net topology fetch on older SC versions, and port used to push policy to remote firewalls. TCP 257: Logging TCP 258: Mgt console listens for remote GUI connections TCP 259: Client Auth via telnet UDP 259: manages encrypted sessions UDP 260: SNMP for the Checkpoint daemon TCP 262: Single Sign-on daemon. TCP 264: SC topology fetch. UDP 500: ISAKMP TCP 900: HTTP client auth. TCP 4532: Session Auth agent TCP 18181: Content Vectoring Protocol TCP 18182: URL Filtering Protocol. TCP 18183: Suspecious Activity Monitoring for IPS. TCP 18186: SIC between OPSEC products and the gateway. TCP 18190: Gateway listens for management clients. TCP 18191: CPD process for communications such as policy installation and certificate revocation. TCP 18192: CPD monitoring

Cisco ASA Policy Based Nat Example: Source address 10.1.1.1 should be translated to 192.168.1.1 when going to 172.16.1.1 and translated to 192.168.1.2 when going to 172.16.1.2 access-list policy_nat1 permit ip host 10.1.1.1 host 192.168.1.1 access-list policy_nat2 permit ip host 10.1.1.1 host 192.168.1.2 static (inside,outside) 172.16.1.1 access-list policy_nat1 static (inside,outside) 172.16.1.2 access-list policy_nat2

Cisco ASA Reverse Path Forwarding Reverse Path Forwarding RPF errors are typically NAT related (traffic is natted one way in one direction and another way in the other direction). Example: --->no nat sh vrrp VRRP State Flags: On 6 interface enabled 6 virtual routers configured 0 in Init state 0 in Backup state 6 in Master state FW-1> FW-1> exit FW-2[admin]# iclid FW-2> sh vrrp VRRP State Flags: On 6 interface enabled 6 virtual routers configured 0 in Init state 4 in Backup state 2 in Master state A TCPDUMP can confirm that VRRP packets are reaching each interface. On the Primary: FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp tcpdump: listening on eth-s4p2c0 00:46:11.374424 O 192.168.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0] 00:46:12.344334 O 192.168.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0] Secondary: FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp tcpdump: listening on eth-s4p2c0 00:19:38.533454 O 192.168.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0] 00:19:39.544322 O 192.168.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0] Now you can see that the interface on both the primary and the secondary firewalls are broadcasting vrrp multicasts. This is because the vrrp multicasts are not reaching the firewalls interfaces. This means there is a communication breakdown which can be possibly caused by network issues. In another example you will see that the VRIDS dont match FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp 00:46:11.206994 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0] 00:46:11.379961 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0] FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp 00:19:38.507294 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0] 00:19:38.630075 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]

Usefull Nokia IPSO Commands newimage -R -k -l ipso.tgz - install a new IPSO image newpkg –i installs software from given location (firewall software, VPN accel driver, etc) voyager –e 0 80 resets voyager after a failed ssl config attempt dbpasswd admin -Changes the password from the command line ipsofwd on admin -turns on ip forwarding when firewall is stopped ipsofwd list -displays ipso properties (flowpath, etc) ipsofwd slowpath -turns off flows (flowpath turns back on) iclid -vrrp utility that shows status - show vrrp -iclid command that shows # of interfaces and their respective states - get vrrp -shows iclid stats: active interfaces/checksum/version/id

-show vrrp interface -displays interface stats for VRRP boot –s {from > prompt at boot time) boots into single-user mode Nokia IPSO has 2 shells, IPSO and Clish. After logging in, you are in the IPSO shell. To enter the Clish shell, type "clish" To remove old config: Either rm /active/config or config/active depending on version.

Usefull Checkpoint Commands o view the active connections table: fw tab -t host_table –s To pull the latest policy from the management station: fw fetch Display the name of the policy installed and the date it was received: fw stat View the Checkpoint version installed: fw ver Display cpu, memory, and disk usage: fw ctl pstat Delete all hosts from the connections table: fw tab -t host_ip_addrs –x Display logs on the firewall for a specific IP: fw log –n –ft | grep Troubleshoot source/destination access issues: fw monitor -m iIOo -e 'accept src=10.33.76.82 and dst=10.33.76.82;' Manage VPN connections (view and delete): vpn tu Turn on debugging for VPN's: vpndebug on and vpn debug ikeon This will create 2 files in $FWDIR/logs. vpnd.elg (this can be viewed on the firewall using cat. It will show highlevel VPN connection information), and ike.elg (this is the bread and butter of Checkpoint VPN troubleshooting. Click here to read my ikeview guide). Display SIC key: cp_conf sic get High Availabiliy: cphaprob stat -display HA status cphaprob -i -display HA interface stats cphastop/cphastart -stop/start HA View license key installed: cplic print Delete all active hosts: fw tab -t host_ip_addrs –x Common Clish commands on Nokia IPSO appliances ---setting default gateway set static-route default nexthop gateway address 192.168.29.2 priority 1 on ---adding static routes set static-route 172.23.124.150/32 nexthop gateway address 192.168.29.50 on ---Add proxy arp add arpproxy address 192.168.29.56 macaddress 0:a0:8e:7d:13:d0 add arpproxy address 192.168.29.57 macaddress 0:a0:8e:7d:13:d0 ---Add an interface set interface eth1 speed 100M duplex full active on add interface eth1c0 address 192.168.29.54/24 set interface eth1c0 enable ---VRRP set vrrp accept-connections on set vrrp coldstart-delay 60 set set set set set set set set

vrrp vrrp vrrp vrrp vrrp vrrp vrrp vrrp

interface interface interface interface interface interface interface interface

eth1c0 eth1c0 eth1c0 eth1c0 eth1c0 eth1c0 eth1c0 eth1c0

---Set ntp servers

monitored-circuit monitored-circuit monitored-circuit monitored-circuit monitored-circuit monitored-circuit monitored-circuit monitored-circuit

vrid vrid vrid vrid vrid vrid vrid vrid

54 54 54 54 54 54 54 54

monitored-interface eth2c0 on monitored-interface eth2c0 priority-delta 10 monitored-interface eth3c0 on monitored-interface eth3c0 priority-delta 10 priority 100 hello-interval 1 vmac-mode default-vmac backup-address 192.168.29.1 on

add ntp server 10.1.1.2 version 3 prefer yes add ntp server 10.1.1.1 version 3 prefer yes ---Setting Time zone set date timezone-city "Greenwich (GMT)" ---Add hostname set hostname testbox ---Add Host address assignments add host name testbox ipv4 192.168.29.54

Nokia IPSO Password reset Boot the Nokia device into single user mode To boot an IP440 into single user mode first restart the box.. When you see the "boot:" prompt enter "-s" and press "enter" within 10 seconds. When it boots into single user mode it will ask for the shell, just press "enter" to accept the default "sh." To boot an IP500 or higher into single user mode, first restart the box. When you will see the prompt "Entering autoboot mode. Type any character to enter command mode." You have 5 seconds to press any key. To boot at IP300 device into single user mode, first restart the box. When you see the prompt "Verifying DMI Pool Data" press the number 1. Then you will see a "Type any character to enter command mode." You now have 5 seconds to press any key. After pressing any key type "boot -s" to enter single user mode. Change Password in IPSO 3.5 and Higher Run "/etc/overpw" from the single user shell and follow the prompts to change the password. Type "reboot" to boot into multi-user mode, go into voyager and change to a permanent password.

Cisco VPN Troubleshooting Guide Cisco PIX 7.0 VPN Troubleshooting Quick overview of IPSEC It is important to understand how IPSEC works in order to understand how to troubleshoot a VPN connection. This is a quick overview of IPSEC and is by no means a complete detailed guide. IPSEC is a suite of protocols, defined in RFC 2401, that is used to protect information as it travels from one private network to another private network over a public network. IPSEC consists of Security Protocols (AH and ESP), Key Management (ISAKMP, IKE, and SKEME), and Algorithms (3DES, AES256, etc). ISAKMP defines the procedures and packet formats used to establish, negotiate, and modify Security Associations. ISAKMP communicates over UDP 500. Security Protocols consist of AH (Authentication Header) and ESP (Encapsulating Security Payload). AH communicates over IP 51 and provides data authentication, integrity, and replay protection (for man in the middle attacks), but does not provide confidentiality. It is important to understand that AH encapsulates the IP packet but does not encrypt it. ESP communicates over IP 50 and provides the same service as AH in addition to providing data confidentiality by encrypting the original payload and encapsulating the packet. SA‘s (Security Associations): In order to have an IPSEC conversation, you first need a security association. Each device must agree on the policies or rules of the conversation by negotiating these policies with their potential peers. The SA represents a unidirectional instance of a security policy for a given connection. Main mode IPSEC packet exchange: --Initiator--- ---Responder------------pk#1—Policy Proposal------> Packet handling order: Step 1 Access lists applied to an interface and crypto map are used by Cisco IOS software to select interesting traffic to be encrypted. Step 2 Cisco IOS software checks to see if IPSec SAs have been established. Step 3 If the SA has already been established by manual configuration using the crypto ipsec transform-set and crypto map commands or has been previously set up by IKE, the packet is encrypted based on the policy specified in the crypto map and is transmitted out of the interface. Step 4 If the SA has not been established, Cisco IOS software checks to see if an IKE SA has been configured and set up. Step 5 If the IKE SA has been set up, the IKE SA governs negotiation of the IPSec SA as specified in the IKE policy configured by the crypto isakmp policy command, the packet is encrypted by IPSec, and it is transmitted. Step 6 If the IKE SA has not been set up, Cisco IOS software checks to see if certification authority (CA) has been configured to establish an IKE policy. Step 7 If CA authentication is configured with the various crypto ca commands, the router uses public and private keys previously configured, obtains the CA's public certificate, gets a certificate for its own public key, and then uses the key to negotiate an IKE SA, which in turn is used to establish an IPSec

SA to encrypt and transmit the packet. Configuring Phase 1: The first 2 octets of IPs have been replaced with "y.y." Phase I is not configured on a per connection basis. When a Phase I connection is being established, configured ISAKMP policies will be tried one at a time until a match is found. Example of an ISAKMP policy: #isakmp policy 20 authentication pre-share #isakmp policy 20 encryption 3des #isakmp policy 20 hash md5 #isakmp policy 20 group 2 #isakmp policy 20 lifetime 43200 Troubleshooting Phase I: Check the syslogs Show run isakmp This will show the isakmp policies for all VPN connections. To view a specific ISAKMP policy type show run isakmp | grep show vpn-sessiondb detail l2l Show crypto isakmp sa detail – This command will display the state of Phase I of the IPSEC tunnel. A state of MM_Active indicates that Phase I was successfully completed. If Phase I does not complete, refer to the table below to find out exactly what state the Phase I connection is currently in. This will give you an indication of where the problem has occurred. More specific information can be found by running a debug(discussed later). State Description OAK_MM_No_STATE This is the initial state of Phase I. If you see Phase I In this state for longer than a few seconds, this is an indication that a failure of tunnel establishment for Phase I has occurred. OAK_MM_SA_SETUP The peers have agreed on parameters for the ISAKMP SA. Phase I will be in this state after packet 1 and packet 2 exchange of the Main Mode negotiation (see above). MM_WAIT_MSG The firewall is waiting on the remote end device to respond with DH and public key. OAK_MM_KEY_EXCH The peers have exchanged DH public keys and have generated a shared secret. OAK_MM_KEY_AUTH The ISAKMP SA has been authenticated. The debug crypto isakmp 5 command will display real time information on every step of the Phase I connection. Debug level 5 should be sufficient for most troubleshooting however level 7 provides more detailed information if necessary. Please note that you cannot limit the debug output to a specific tunnel. IKMP_NO_ERROR_NO_TRANS indicates a matching transform set was not found No Proposal Chosen=isakmp policy mismatch syslog sample of a completed connection: Mar 10 2008 18:47:05: %PIX-3-713119: Group = y.y.41.250, IP = y.y..41.250, PHASE 1 COMPLETED Sample Debug output: The following shows the initiation of the first packet for an IPSEC tunnel. 58534 02/27/2004 07:42:38.600 SEV=4 IKE/41 RPT=8619 y.y.11.49 IKE Initiator: New Phase 1, Intf 2, IKE Peer The following indicates that the IKE Phase I policy was accepted by the remote gateway. 58534 02/27/2004 07:42:38.600 IP = y.y.11.49, Oakley proposal is acceptable This indicates Phase I has completed. 58534 02/27/2004 07:42:38.600 Group= y.y.11.49, IP=y.y.11.49, Oakley begin quick mode The following indicates that the remote gateway has indicated that none of the policies are acceptable. 5|Oct 02 2006 09:41:41|713904: IP = y.y.138.12, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping To clear the Security Associations related to Phase 1, use the clear crypto isakmp command. This will clear ALL of the SA‘s currently built on this firewall. To confirm that the IPSEC packets are reaching the firewall, a capture can be created for all UDP 500 traffic. First create an access-list for the traffic you would like to capture. Access-list capture1 permit udp any any eq 500 Next create a capture. Capture cap1 access-list capture1 interface outside Next display the results of the capture. Show capture cap1 detail ciscoasa#show capture cap1 detail 1: 13:04:06.284897 192.168.1.50 > 192.168.1.1: UDP:500

View capture on web https://capture/pcap/cap1 View pre-shared keys: more system:running-config Configuring Phase 2: A transform set combines encryption method and authentication method. During the IPSec security association negotiation with ISAKMP, the peers agree to use a particular transform set to protect a particular data flow. The transform set must be the same for both peers. You can create multiple transform sets, and then specify one or more of these transform sets in a crypto map entry. You can view previously created transform sets by typing the show crypto ipsec transform-set command. If the desired transform set has not been previously defined, the crypto ipsec transform-set command is used to create it. Example: #(config)crypto ipsec transform-set 3desmd5 esp-3des esp-md5-hmac An access-list is used to define the ―interesting traffic‖ or the traffic that should be encrypted and allowed through the VPN Tunnel. The access-list should always be defined from local to remote. The subnet sizes need to match on the remote gateway. Example: #(config) access-list tunnel1 extended permit ip y.y..191.0 255.255.255.0 host y.y..155.12 If port filtering is being used, and traffic is being initiated from the remote side, the destination port of the remote host must be the source port of the local matching acl. A tunnel group is used to identify specific connection parameters and the definition of a group policy. The default tunnel groups are DefaultRAGroup (used for Remote Access tunnels) and DefaultL2Lgroup(used for IPSEc Lan-to-Lan tunnels). Example: #(config)tunnel-group y.y.155.1 type IPsec_l2l #(config)tunnel-group y.y.155.1 ipsec-attributes #(config-attributes) pre-shared-key abc123 The crypto map ties together several components that define the VPN tunnel. This is where the peer defined in the tunnel-group command is tied to the access-list and transform-set. The crypto map must be assigned a unique map id #. To view the previously used crypto map id numbers run the show ru crypto command. Example: #(config)crypto map mymap 10 match address tunnel1 #(config)crypto map mymap 10 set peer y.y,155.1 #(config)crypto map mymap 10 set transform-set 3desmd5 Nat considerations: If a local address is going to be natted outbound, the crypto acl should use the outside ip address. Troubleshooting Phase II: Check syslogs Show crypto ipsec sa- This command shows the output of the IPSEC SA‘s. The SA will include the ip address of the local and remote endpoints, encryption domains (interesting traffic), transform set (what encryption and hash is being used), key lifetime, and # of packet encrypt/decrypts. debug crypto engine—Displays the traffic that is encrypted. Example of an IPSEC SA: This shows the crypto map used for this connection. Crypto map tag: vpn_map, seq num: 130, local addr: x.x.160.45 The following line shows the crypto acl that includes the traffic to be protected. access-list VPN-CIDS704976 permit ip x.x.190.0 255.255.254.0 host 10.2 5.4.80 local ident (addr/mask/prot/port): (x.x.190.0/255.255.254.0/0/0) remote ident (addr/mask/prot/port): (10.25.4.80/255.255.255.255/0/0) current_peer: y.y.227.136 Encrypts indicate that this side is encrypting and sending traffic. Decrypts indicates that the other side is sending traffic. #pkts encaps: 5, #pkts encrypt: 5, #pkts digest: 5 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 5, #pkts comp failed: 0, #pkts decomp failed: 0 #send errors: 0, #recv errors: 0 This lists the local and remote endpoints. local crypto endpt.: x.x.160.45, remote crypto endpt.: y.y.227.136 path mtu 1500, ipsec overhead 74, media mtu 1500 current outbound spi: 2AFEA5C7 There is a separate sa for inbound and outbound. inbound esp sas: spi: 0x9D111D2A (2635144490) transform: esp-aes-256 esp-sha-hmac none in use settings ={L2L, Tunnel, PFS Group 5, } slot: 0, conn_id: 317225, crypto-map: vpn_map sa timing: remaining key lifetime (kB/sec): (4275000/28789) IV size: 16 bytes replay detection support: Y outbound esp sas:

spi: 0x2AFEA5C7 (721331655) transform: esp-aes-256 esp-sha-hmac none in use settings ={L2L, Tunnel, PFS Group 5, } slot: 0, conn_id: 317225, crypto-map: vpn_map sa timing: remaining key lifetime (kB/sec): (4274999/28789) IV size: 16 bytes replay detection support: Y Clear crypto ipsec sa peer will clear the Phase 2 SA‘s for a given peer. debug crypto ipsec—Displays the IPSec negotiations of phase 2. No Valid SA/ Identity mismatch – Transform set or crypto acl Sample Debug output: The following shows that the tunnel group configuration was found. Oct 26 15:42:43 [IKEv1]: IP =y.y.205.92, Connection landed on tunnel_group y.y,.205.92 Sample syslog errors: This shows interesting traffic ACL getting exchanged. 1754 11/29/2001 16:20:18.500 SEV=7 IKEDBG/0 RPT=546 y.y.205.92 Transmitting Proxy Id: Remote host: 192.168.1.1 Protocol 0 Port 0 Local host: 10.64.10.9 Protocol 0 Port 0 Completion of Phase II. 1949 11/29/2001 16:20:18.540 SEV=4 IKE/49 RPT=3 y.y.205.92 Security negotiation complete Responder, Inbound SPI = 0x11a56495, Outbound SPI = 0xb17718a5 Mar 10 2008 18:47:05: %PIX-5-713120: Group = y.y.41.250, IP = y.y.41.250, PHASE 2 COMPLETED (msgid=0f78e513) Pre-shared key mismatch. 1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received an un-encrypted PAYLOAD_MALFORMED notify message, dropping. Pre-shared key mismatch reported by the report peer(receiving peer): :713903: Group = 172.1.12.1, IP = 172.1.12.1 ERROR. peer has indicated that something is wrong with our message. This could i ndicate a pre-shared key mismatch. Transform-set mismatch. 1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received non-routine Notify message: No Proposal Chosen Transform-set mismatch on remote peer(receiving peer): 713904‖ IP = 10.51.16.1, Received encrypted packet with no matching SA, dropping 713048: IP = 10.51.16.1 Error processing payload. Payload ID 1 The following indicates that the remote gateway is not finding matching interesting traffic. 1754 11/29/2001 16:20:18.500 Group = y.y.172.63, IP = y.y.172.63, Received non-routing Notify message: Invalid ID info (18) The following indicates that the local gateway is not finding matching interesting traffic. 1754 11/29/2001 16:20:18.500 Group =y.y.172.63, IP = y.y.172.63, Static Crypto Map check, map = mymap, seq = 10, ACL does not match proxy IDs src:192.168.1.0 dst:192.168.2.0 PFS mismatch:

713068: Group – 172.1.12.1, Received non-rouing Notify message; No Proposal chosen (14) PFS turned on on the remote peer. Local peer reports the following: 713902; Group = 10.51.16.1. QM FSM error (p2 struct &0x296fde8, mess id 0x518e80d)! QM FSM is a generic message indicating that the phase II connection was rejected by the remote peer. This indicated that the remote peer is natting: %ASA-4-402116: IPSEC: Received an ESP packet (SPI= 0x72DEC2AA, sequence number= 0x41) from y.y.28.178 (user= y.y.28.178) to y.y.83.194. The decapsulated inner packet doesn't match the negotiated policy in the SA. The packet specifies its destination as y.y.83.194, its source as y.y.28.178, and its protocol as 1. The SA specifies its local proxy as y.y.10.16/255.255.255.240/0/0 and its remote_proxy as y.y.63.0/255.255.255.0/0/0.

When reverse route is turned on: Jan 26 2009 18:15:07: %ASA-6-713211: Group =y.y43.160, IP = y.y.43.160, Adding static route for L2L peer coming in on a dynamic map. address: 192.168.8.5, mask: 255.255.255.255 Jan 26 2009 18:57:54: %ASA-6-713213: Group = y.y.43.160, IP =y.y43.160, Deleting static route for L2L peer that came in on a dynamic map. address: 192.168.8.5, mask: 255.255.255.255 Saturday, February 28, 2009

Checkpoint Tables and the FW Tab Command The fw tab command displays the contents of the INSPECT tables. These tables hold all state information on the firewall, including connections, VPNS, nats, etc.

The following options are commonly used: -s provides a summary the tables -t tname only displays the requested table -x tname delete all entries in the specified table -d debug mode -all all tables Useful tables: host_table This host_table holds the IP addresses of internal machines protected by the VPN-1/FireWall-1 NG Enforcement Module. The table only exists where the VPN-1/FireWall-1 license is limited. The maximum number of entries in this table is the licensed number of internal machines. arp_table The arp_table holds the IP addresses for which the machine is willing to proxy ARP. Proxy ARP is sometimes required when using NAT. If IP addresses that the machine is to resolve are specified in local.arp , this table can be used to check that the IP addresses were correctly specified. The automatic ARP feature also uses this table to cause the machine to resolve translated IP addresses. Entry format: 1. IP address (name resolving may occur). 2. MAC address of gateway machine interface that will answer ARP requests for the IP address. 3. Interface name (optional). 4. Name of the gateway interface that will proxy ARP for IP addresses. The fwx_alloc table uses the following formats. First entry: < 0, hiding IP address, IP protocol, first high port used; next high port to be allocated> The first field is a space holder and is always 0. The first high port to be used is always 10000. fwx_alloc The fwx_alloc table holds information about the allocation of ports for the translated packets. fwx_cntl_dyn_tab The fwx_cntl_dyn_tab table holds information about the allocated IP addresses from the IP Pool of the Enforcement Module, and the connections using the IP addresses. EXAMPLE attributes: keep, expires never, limit 25000, hashsize 512, free function 40550248 0 00000001, 00000e10, 000000c3>

fwx_auth The fwx_auth table holds the original information of a folded connection, so that the Security Server will know the original destination IP and port of the connection. EXAMPLE attributes: expires 300, limit 25000, refresh, keep

c7cb47e3, 00000017; 286/300>

sam_blocked_ips

All IP and network addresses that were stipulated in SAM requests, are shown in sam_blocked_ips , with a requests counter for each prototype of filter that is enforced over each certain IP and network address. Note the overshadowed requests are also accounted for, if present. EXAMPLE -------- sam_blocked_ips --------dynamic, id 8141, attributes: keep, limit 25000, hashsize 512 00000000, 00000000, 00000000>

host_ip_addrs The host_ip_addrs table contains the list of IP addresses in the VPN-1/FireWall-1 NG Enforcement Module.

fwx_ip_lookup_tab The fwx_ip_lookup_tab table holds information used for IP Pool allocation queries. EXAMPLE attributes: keep, limit 25000, hashsize 512 fwz_crypt_pending The fwz_crypt_pending table is used to record a possibly encrypted connection that should obtain their encryption/decryption key. This table is accessed from the VPN kernel, and the VPN daemon. The kernel can obtain a new key from this table stored by the daemon, which negotiated the key exchange with a trusted peer. This table also passes error messages. The fwz_crypt_pending table is dynamic. forbidden_tab Each embedded VPN-1/FireWall-1 system has a feature that indicates how many hosts can be located ‗behind‘ it. The number of hosts can be set to unlimited. This limitation is enforced in the Inspect code using the macro COUNT_HOST .

COUNT_HOST records each packet that comes from the internal interface in a table until the limit is exceeded

IKE_SA_table IKE SAs are stored in IKE_SA_table . The table entries have four possible formats:

 The top two formats are only used on SecuRemote.  All non-expired SAs are stored using format 2.  Format 1 is used to store and retrieve the latest IKE SA.  Table entries are used to conduct IKE Quick Mode negotiation of IPSEC_SA .  Entries are extracted from this table when the vpn daemon is trapped for IPSEC_SA renewal. IKE daemon tries to use previously negotiated IKE SA.  The IKE_SA_table is dynamic. ATTRIBUTES expires

3600

limit

25000

sync keep hashsize 512 kbuf

1

KEYS PeerAddress ipaddr

IKE peer address.

Me

ipaddr

on SecuRemote it is the IP address used by SecuRemote when it negotiated this IKE SA; field is used to prevent using an old IKE SA negotiated before SecuRemote obtained a new IP address; if initiator or responder

CookieI

u_long [2]

initiator cookie (8 bytes). host byte order

CookieR

u_long [2]

responder cookie (8 bytes). host byte order

VALUES IKE_SA

kbuf A kbuf storing the fwisakmp_sa structure.

Flags

uint currently only 2 flags are defined: (PEER_MOBILE, INITIATOR); used so Enforcement Module does not need to retrieve the whole kbuf from the kernel if we only want to know if this SA was established with a mobile user

RenegotiationTime uint renegotiation time of the SA EXAMPLE -------- IKE_SA_table -------dynamic, id 77, attributes: keep, sync, expires 3600, limit 25000, hashsize 512, kbuf 1, free function

Cisco Anyconnect sample config config t webvpn svc image disk0:/anyconnect-win-2.0.0343-k9.pkg 1 ! this is a customerized vpn profile, if client does not needed, you can remove the following line using cisco default ! svc profiles VitalProf disk0:/vpn-vig-tdc.xml tunnel-group-list enable enable outside svc enable exit ip local pool SSLClientPool 192.168.100.1-192.168.100.50 mask 255.255.255.0 access-list NONAT extended permit ip 192.168.5.0 255.255.255.0 192.168.100.0 255.255.255.0 access-list vpnssl-split extended permit ip 192.168.5.0 255.255.255.0 192.168.100.0 255.255.255.0 nat (inside) 0 access-list NONAT username userA password test123 username userA attributes service-type remote-access exit username userB password test12345 username userB attributes service-type remote-access exit group-policy SSLCLientPolicy internal group-policy SSLCLientPolicy attributes dns-server value 192.168.1.51 192.168.1.61 wins-server value 192.168.1.51 192.168.1.61 address-pools value SSLClientPool split-tunnel-policy tunnelspecified split-tunnel-network-list value vpnssl-split webvpn vpn-tunnel-protocol svc svc keep-installer installed !svc profiles value VitalProf exit sysopt connection permit-vpn tunnel-group SSLClientProfile type remote-access tunnel-group SSLClientProfile general-attributes default-group-policy SSLCLientPolicy tunnel-group SSLClientProfile webvpn-attributes group-alias SSLVPNClient enable exit wr mem wr stand debug command sh vpn-sessiondb svc, please be noticed, the default license for asa for web vpn or ssl vpn is only 2, you need to notify the client for this license limitation

Viewing all hidden commands on a Cisco ASA The "show parser dump all" command will display all valid commands, including undocumented commands. The full output of the command came out to over 300 pages in MS WORD in 11 font.

A few examples of hidden VPN commands are as follows: clear/show ipsec sa counters clear/show crypto ipsec sa map clear/show crypto protocol statistics ssl clear/show crypto protocol statistics ssh clear/show crypto protocol statistics srtp clear/show crypto protocol statistics other clear/show crypto protocol statistics all clear/show vpn-sessiondb statistics clear/show vpn-sessiondb statistics debug vpnlb debug vpn-sessiondb debug crypto isakmp timers debug crypto ca messages debug crypto condition peer subnet debug crypto condition peer subnet debug crypto condition peer debug debug debug debug

crypto crypto crypto crypto

condition condition condition condition

user unmatched isakmp error ipsec error isakmp

http://www.netleets.com/search/label/tcpdump

Encryption Failure: Packet Is Dropped as There Is No Valid SA You might see this error message when both ends of the VPN do not have the same definition for the encryption domain. First ensure that both ends of the VPN are defined with the same encryption domain. If they are the same, you should create objects that are exactly the same size as what is created on the remote end. One annoying behavior FireWall-1 NG exhibits that FireWall-1 4.1 and earlier did not is the automatic simplification of subnets in IPSec SAs. For example, if your encryption domain contains explicit objects for 192.168.0.0/24 and 192.168.1.0/24, Check Point would attempt to negotiate an IPSec SA with 192.168.0.0/23 instead of generating SAs based on the network objects you created. To eliminate this behavior, use dbedit to make the following changes on your management console (see FAQ 4.2 for details on editing objects_5_0.C): dbedit> modify properties firewall_properties ike_use_largest_possible_subnets false dbedit> update properties firewall_properties You must then reload the security policy for this change to take effect.

VPN Fails When Transferring Large Packets Some applications set the Don't Fragment bit on certain packets. When the IPSec headers are added to the already large packet, the packet requires fragmentation in order to pass through the firewall. When Check Point creates the IPSec packet, the Don't Fragment bit from the original packet is maintained. FireWall-1 creates a fragmented packet that has the Don't Fragment bit set, so it cannot be fragmented and thus gets dropped at the next router. You can force FireWall-1 to clear the Don't Fragment bit by changing the ipsec_dont_fragment property in objects_5_0.C to false. You can do this with the following commands in dbedit on the management console (craig is the firewall in this example): dbedit> modify network_objects craig VPN:ipsec_dont_fragment false dbedit> update network_objects craig Alternatively, you can use the GUIdbedit tool to change the parameter. Either way, you must then reinstall the security policy for this change to take effect.

Debugging Interoperability Issues with IKE Everyone has a different interpretation about how to follow standards. As a result, when third-party products talk to one another, communication doesn't always work. One way to debug is to turn on IKE debugging. In FireWall-1 4.1, it was necessary to stop and restart FireWall-1 in order to enable debugging. In NG, you can enable this on the firewall module with a simple command: vpn debug ikeon. You can also disable it with vpn debug ikeoff. When you enable debugging, $FWDIR/log/ike.elg gets created. This file contains the results of all IKE negotiations that occur. This file is a little difficult to read on its own. Fortunately, Check Point has a tool called IKEView that allows you to view this file in a more readable form. Unfortunately, it is available only to Check Point Certified Service Partners.

Check Point Logging Troubleshooting Guide Are the logs being sent to the manager? Ok, so first of all are the logs being sent to the Smart Centre Manager or the necessary Log Manager ? We can check this by confirming whether the gateway is sending the log packets via the FW Log port tcp/257 upon the gateway and the manager. To do this use either or both of the following commands, netstat -an | grep 257 - This will show the state of the TCP sockets. tcpdump -ni [interface name] port 257 - This will show a packet capture of the FW Log packets on the subsequent interface. If the gateway is not sending the logs then this can be down to one of the following issues, 1.

SIC is not established.

2.

The Logging configuration for the Gateway is not configured correctly.

3.

The SmartCentre/Log Manager is not listening on port tcp/257.

4.

There is an issue with FWD on the gateway. In some instances you may need to restart FWD via a cpstart. Though the root cause could be down to a number of factors.

Why are the logs not being displayed within SmartView tracker ? Ok so the manager is receiving the logs but you may still not see them within the SmartView tracker this will be down to either the FWD (Firewall Daemon) or the log files being corrupted. Log Files Corrupted If the log files are corrupted you should expect to see no logs within the SmartView Tracker. If this is the case you will need to action the following steps : 1. Close the Log Viewer/SmartView Tracker and Policy Editor/SmartDashboard.

2. Execute the fwstop or cpstop command (depending on the version) from the command line. 3. Remove all files starting with fw.log and fw.logptr from the $FWDIR\log directory. 4. Execute the fwstart or cpstart (depending on the version) command. Full details can be found at Check Points KB within Solution ID sk6432. Only some of the logs are not being displayed If only some of the logs are not being displayed then this could point to an issue with the trust between the manager and the gateway. To confirm the issue you will need to debug FWD using the following steps. root@cp-mgnt# fw debug fwd on TDERROR_ALL_ALL=5 root@cp-mgnt# tail -f $FWDIR/log/fwd.elg root@cp-mgnt# tail -f $FWDIR/log/fwd.elg | grep -i "Certificate is revoked" root@cp-mgnt# fw debug fwd off Within these steps we first enable the debug. Then we run a live tail on the log file. And then we run a grep on the live tail for a specific error. The live tail allows us to view the end of the log file in real time. We finally turn off the debug. Below shows an example of an error with the SIC trust between the Gateway and Manager obtained from the $FWDIR/log/fwd.elg, [FWD 2177 1]@cp-mgnt[22 Jan 14:47:32] fwCert_ValCerts: Certificate is revoked. CN=cp-fw1,O=cp-mgnt..bizt7z [FWD 2177 1]@cp-mgnt[22 Jan 14:47:41] fwCert_ValCerts: Certificate is revoked. CN=cp-fw2,O=cp-mgnt..bizt7z In this instance resetting SIC would resolve this issue

Troubleshooting Checkpoint ClusterXL I recently came across an issue where SmartView Monitor showed an error for ClusterXL on a freshly rebuilt Checkpoint IP565 firewall. Both Synchronization and Filter were stuck in an initilizing state, we tried the following troubleshooting steps initially to no avail: 5.

cphastop followed by cphastart

6.

cpstop followed by cpstart

7.

reboot of the affected firewall

On digging deeper we noticed that one of the firewall devices was configured to use multicast and one for broadcast cluster communications, this was identified using the following command 'cphaprob -a if' which presents the following output: eth-s1p3c0 non sync(non secured) eth-s4p3c0 non sync(non secured) eth-s4p4c0 non sync(non secured) eth-s1p1c0 non sync(non secured) eth-s1p4c0 sync(secured), multicast eth-s1p2c0 non sync(non secured) eth-s4p1c0 non sync(non secured) eth-s4p2c0 non sync(non secured) Virtual cluster interfaces: 7 eth-s1p3c0 xx.xx.xx.xx eth-s4p3c0 xx.xx.xx.xx

eth-s4p4c0 xx.xx.xx.xx eth-s1p1c0 xx.xx.xx.xx eth-s1p2c0 xx.xx.xx.xx eth-s4p1c0 xx.xx.xx.xx eth-s4p2c0 xx.xx.xx.xx

Both firewalls must be configured to use the same method of communication, which can be changed using the following command 'cphaconf set_ccp multicast' or 'cphaconf set_ccp broadcast'. Providing your switching infrastructure supports multicast you should use this mode due to the performance overhead of broadcast communication. This command failed to change the method of communication and left us with no other option than to perform the following steps: 1. Set Checkpoint Packages as in-active, then delete them ensuring that the Connectra package is removed first. 2. Re-install the Checkpoint R65 IPSO Wrapper 3. Re-install HFA 70 4. Re-establish SIC via CPConfig and SmartDashboard 5. Unassign and re-assign license via SmartUpdate 6. Push policy from the SmartDashboard After performing thse steps the cluster CCP was back to multicast (bizare really...). We had to perform a reboot of the second device once this was completed, at which point both nodes of the cluster reported no ClusterXL errors, 'cphaprob list' showed the following output: # cphaprob list Registered Devices: Device Name: Synchronization Registration number: 0 Timeout: none Current state: OK Time since last report: 213003 sec Device Name: Filter Registration number: 1 Timeout: none Current state: OK Time since last report: 213003 sec Device Name: cphad Registration number: 2 Timeout: 5 sec Current state: OK Time since last report: 0.7 sec Device Name: fwd Registration number: 3 Timeout: 5 sec Current state: OK Time since last report: 0.5 sec 'fw ctl pstat' should also list the Synch as 'Able to Send/Receive sync packets' : # fw ctl pstat

Machine Capacity Summary: Memory used: 14% (90MB out of 637MB) - below low watermark Concurrent Connections: 26% (17876 out of 67900) - below low watermark Aggressive Aging is in monitor only Hash kernel memory (hmem) statistics: Total memory allocated: 200278016 bytes in 48894 4KB blocks using 2 pools Initial memory allocated: 20971520 bytes (Hash memory extended by 179306496 bytes) Memory allocation limit: 536870912 bytes using 10 pools Total memory bytes used: 23487660 unused: 176790356 (88.27%) peak: 34170776 Total memory blocks used: 7126 unused: 41768 (85%) peak: 9164 Allocations: 1183931215 alloc, 0 failed alloc, 1183678473 free System kernel memory (smem) statistics: Total memory bytes used: 250335916 peak: 300842432 Blocking memory bytes used: 1865892 peak: 2596156 Non-Blocking memory bytes used: 248470024 peak: 298246276 Allocations: 160033475 alloc, 0 failed alloc, 160032829 free, 0 failed free Kernel memory (kmem) statistics: Total memory bytes used: 73389696 peak: 101169940 Allocations: 1184023246 alloc, 0 failed alloc, 1183769860 free, 0 failed free External Allocations: 0 for packets, 0 for SXL Kernel stacks: 0 bytes total, 0 bytes stack size, 0 stacks, 0 peak used, 0 max stack bytes used, 0 min stack bytes used, 0 failed stack calls INSPECT: 1029526467 packets, -2128289516 operations, 373013811 lookups, 2035 record, 183665476 extract Cookies: -1649393933 total, 0 alloc, 0 free, 4607 dup, -1525329462 get, 138972711 put, -1565092568 len, 217535 cached len, 0 chain alloc, 0 chain free Connections: 54513276 total, 52537755 TCP, 1898998 UDP, 76506 ICMP, 17 other, 49485065 anticipated, 1 recovered, 17882 concurrent, 24286 peak concurrent Fragments: 213594 fragments, 105472 packets, 389 expired, 0 short, 0 large, 0 duplicates, 0 failures NAT: 23444077/0 forw, 29804768/0 bckw, 53234829 tcpudp, 14016 icmp, 702040-723136 alloc Sync: Version: new Status: Able to Send/Receive sync packets Sync packets sent: total : 78286072, retransmitted : 16171, retrans reqs : 20, acks : 3 Sync packets received: total : 17030603, were queued : 16591, dropped by net : 15 retrans reqs : 8840, received 3 acks retrans reqs for illegal seq : 0 dropped updates as a result of sync overload: 0

Check Point Firewall-1 Useful Firewall-1 command line utilities: Unload current security policy fw unloadlocal VPN Tunnel command line access (e.g. delete SAs) vpn tu Display overlapping VPN Encryption Domains vpn overlap_encdom [communities|traditional] List current Firewall interfaces fw ctl iflist Show HA / ClusterXL state cpstat ha cphaprob state cphastop / cphastart Display State of Checkpoint HA Interfaces cphaprob -a if Stop/Start Checkpoint HA/ClusterXL cphastop / cphastart Display State of Checkpoint HA Interfaces cphaprob -a if Manually failover cphaprob -d STOP -s problem -t 0 register cphaprob list cphaprob -d STOP unregister Display State of ClusterXL IGMP cphaprob stat (Notify if IGMP membership is supported) cphaprob igmp (Display the current IGMP membership settings)

SmartCenter Backup and Restore SmartCenter upgrade_export $FWDIR/bin/upgrade_tools/upgrade_import Check whether licensed for management high availability (Management HA) cplic check mgmtha

SecurePlatform SecurePlatform configuration commands: Configure Interfaces, Routes etc sysconfig Add static routes config route add dest 192.168.1.0/24 via 192.168.0.1 dev eth0 metric 0 s-persistant on apply on Configure Network Interfaces config conn help config conn set name eth1 type eth onboot on iff-up on local 192.168.1.2/24 broadcast 192.168.1.255 s-persistant on s-code up mtu 1500 Configure Bonded Network Interfaces (NIC Team, 2 physical, 1 logical interface) config conn add name bond0 type bond onboot on iff-up on mtu 1500 bond-mode active-backup bond-miimon 100 bonddowndelay 200 bond-updelay 200 bond-primary eth1 local 192.168.1.2/24 config conn add name eth1 type eth onboot on iff-up on mtu 1500 master-bond bond0 config conn add name eth4 type eth onboot on iff-up on mtu 1500 master-bond bond0 Useful SecurePlatform command line utilities: Enter OS commands expert Assign interfaces to correct physical NICs (Edit /etc/sysconfig/ethtab) [Expert@FIREWALL]# cat ethtab eth0 00:21:5A:27:DC:E6 eth1 00:21:5A:27:DC:E4 eth2 00:1F:29:5C:82:F5

Set Kernel parameters (Edit $FWDIR/boot/modules/fwkern.conf) fwha_mac_magic=0×11 fwha_mac_forward_magic=0×10 fwha_monitor_if_link_state=1 fwha_enable_igmp_snooping=1 fwha_igmp_version=2 Flag disconnected NICs echo eth6 >> $FWDIR/conf/discntd.if Show status of Bonded Network Interfaces cphaconf show_bond -a Display Versions SPLAT: ver Firewall: fw ver Performance Pack: sim ver –k Linux: uname -a Change shell to permit WinSCP connection usermod -s /bin/bash fwadmin Change shell timout (cpshell) idle mm where mm = timeout in minutes (permanent change, updates /etc/cpshell/cpshell.state and is passed on to expert shell) Change shell timout (bash) TMOUT = ss where ss = timeout in minutes export TMOUT Display the number of CPUs presented to SecurePlatform OS grep ‘physical id’ /proc/cpuinfo|wc -l Display the CoreXL CPU Affinity fw ctl affinity -l Advanced Routing (gated) Commands

ps -eaf | grep gated cpwd_admin list