Troubleshooting VRRP on Nokia Checkpoint Firewalls

The purpose of this article is to help in troubleshooting VRRP related issues on NOkia Checkpoint Firewalls. One of the most common problems faced in Nokia VRRP Implementations is that interfaces on active and standby firewalls go into the master master state. THe main reason for this is because the individual vrids of the master and backup firewall are not able to see the vrrp multicast requests of each other.

The first step is to check the vrrp state of the interfaces. THis is how you can check that:

PrimaryFW-A[admin]# iclid
PrimaryFW-A> sh vrrp

VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
0 in Backup state
6 in Master state
PrimaryFW-A>
PrimaryFW-A> exit

Bye.
PrimaryFW-A[admin]#

SecondaryFW-B[admin]# iclid
SecondaryFW-B> sh vrrp

VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
4 in Backup state
2 in Master state
SecondaryFW-B>
SecondaryFW-B> exit

Bye.
SecondaryFW-B[admin]#

In the example shown you see that 2 interfaces each from both firewalls are in the Master state.

The next step should involve running tcpdumps to see if the vrrp multicasts are reaching the particular interface.

As the first troubleshooting measure, put a tcpdump on the problematic interface of the master and backup firewalls. If you want to know what the problematic interface is, "echo sh vrrp int | iclid" should give you the answer. It is that interface on the backup firewall which would be in a Master state.

PrimaryFW-A[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:46:11.379961 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:12.399982 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:13.479985 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:14.560007 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]

When you put a tcpdump on the Primary Firewall, you see that the vrrp multicast request is leaving the interface.

Next put the tcpdump on the secondary firewall.

SecondaryFW-B[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:19:38.507294 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:39.527316 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:40.607328 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:41.687351 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:42.707364 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]

Now you can see that the interface on both the primary and the secondary firewalls are broadcasting vrrp multicasts. This is because the vrrp multicasts are not reaching the firewalls interfaces. This means there is a communication breakdown which can be possibly caused by network issues.

Once the network issue is resolved, communication would be possible and the interface with the lower priority will go as the secondary or backup state.

Now let us discuss another scenario where there is a problem with the firewall interfaces in Master Master state.

Again put a tcpdump on both the interfaces in question:

PrimaryFW-A[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:46:11.206994 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:11.379961 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:12.286990 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:12.399982 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:13.307014 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:13.479985 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:14.387098 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:14.560007 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:15.467064 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:15.580010 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]

SecondaryFW-B[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:19:38.507294 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:38.630075 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]
00:19:39.527316 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:39.710131 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]
00:19:40.607328 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:40.790142 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]
00:19:41.687351 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:41.810150 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]

In the above example look at the vrid numbers of the incoming and outgoing packets. From the vrids you see that that the vrids donot match. This is an indication that the cabling is not correct. The cables going to vrid 102 and 103 are not connected correctly and they need to be swapped to fix this issue.

Swap the cables and the issue will be resolved. The firewall with the higher priority will go into the Master state.

A properly functioning firewall will be like this:

PrimaryFW-A[admin]# iclid
PrimaryFW-A> sh vrrp

VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
0 in Backup state
6 in Master state
PrimaryFW-A> exit

Bye.
PrimaryFW-A[admin]#

SecondaryFW-B[admin]# iclid
SecondaryFW-B> sh vrrp

VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
6 in Backup state
0 in Master state
SecondaryFW-B> exit

Bye.
SecondaryFW-B[admin]#

If you were to tcpdump the healthy interface, this is how it would look:

PrimaryFW-A[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
18:25:44.015711 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:25:45.095726 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:25:46.175751 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:25:47.195770 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:25:48.275819 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:25:49.355812 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
^C
97 packets received by filter
0 packets dropped by kernel
PrimaryFW-A[admin]#

SecondaryFW-B[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
18:26:07.415446 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:08.495451 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:09.515480 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:10.595486 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:11.675485 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:12.695522 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
18:26:13.775590 I 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
^C
14 packets received by filter
0 packets dropped by kernel
SecondaryFW-B[admin]#

Usefull Nokia IPSO Commands

newimage -R -k -l ipso.tgz - install a new IPSO image

newpkg –i installs software from given location (firewall software, VPN accel driver, etc)


voyager –e 0 80 resets voyager after a failed ssl config attempt

dbpasswd admin -Changes the password from the command line

ipsofwd on admin -turns on ip forwarding when firewall is stopped

ipsofwd list -displays ipso properties (flowpath, etc)

ipsofwd slowpath -turns off flows (flowpath turns back on)

iclid -vrrp utility that shows status

- show vrrp -iclid command that shows # of interfaces and their respective states

- get vrrp -shows iclid stats: active interfaces/checksum/version/id

-show vrrp interface -displays interface stats for VRRP
boot –s {from > prompt at boot time) boots into single-user mode

Nokia IPSO has 2 shells, IPSO and Clish.

After logging in, you are in the IPSO shell. To enter the Clish shell, type "clish"

To remove old config:
Either rm /active/config or config/active depending on version.

Nokia Top Clish Commands Reference

This is a quick reference guide to the most popular and widely used Nokia Clish Commands. You can manage the Nokia firewall as much from the Command Line Interface as from Voyager.

---setting default gateway
set static-route default nexthop gateway address 192.168.29.2 priority 1 on

---adding static routes
set static-route 172.23.124.150/32 nexthop gateway address 192.168.29.50 on

---Add proxy arp
add arpproxy address 192.168.29.56 macaddress 0:a0:8e:7d:13:d0
add arpproxy address 192.168.29.57 macaddress 0:a0:8e:7d:13:d0

---Add an interface
set interface eth1 speed 100M duplex full active on
add interface eth1c0 address 192.168.29.54/24 enable

---VRRP

set vrrp accept-connections on
set vrrp coldstart-delay 60

set vrrp interface eth1c0 monitored-circuit vrid 54 monitored-interface eth2c0 on
set vrrp interface eth1c0 monitored-circuit vrid 54 monitored-interface eth2c0 priority-delta 10
set vrrp interface eth1c0 monitored-circuit vrid 54 monitored-interface eth3c0 on
set vrrp interface eth1c0 monitored-circuit vrid 54 monitored-interface eth3c0 priority-delta 10
set vrrp interface eth1c0 monitored-circuit vrid 54 priority 100
set vrrp interface eth1c0 monitored-circuit vrid 54 hello-interval 1
set vrrp interface eth1c0 monitored-circuit vrid 54 vmac-mode default-vmac
set vrrp interface eth1c0 monitored-circuit vrid 54 backup-address 192.168.29.1 on

---Set ntp servers

add ntp server 10.1.1.2 version 3 prefer yes
add ntp server 10.1.1.1 version 3 prefer yes

---Setting Time zone

set date timezone-city "Greenwich (GMT)"

---Add hostname

set hostname testbox

---Add Host address assignments

add host name testbox ipv4 192.168.29.54

IPSO - Commands

Below are the command IPSO commands that can be used,
IPSO commands
newimage	Installs IPSO OS from the local machine
newpkg -m localhost	Checkpoint package Install
clish	IPSO OS CLI
ipsctl -a	displays all of the IPSO Settings and Values
ipsctl -a ifphys:eth-s5p1:errors|more	display errors on eth-s5p1
ipsctl -w net:ip:tcp:default_mss 1460	Change MSS to 1460
netstat 1	shows network stats every second
ipsofwd list	displays ipso properties (flowpath, etc)
ipsofwd slowpath	turns off flows (flowpath turns back on)
fsck -fyb 32	check the file system on a flash based nokia (KB 1355433)
Bootmgr
printenv	print environment variables
install	install an image across the network
boot	boot an image
clish commands
show useful-stats	Shows Disk, VRRP, RAM summary
show package all
show package active
show package inactive
show images
show image current
delete image [name]
set hostname testbox	Set Hostname
set date timezone-city "Greenwich (GMT)"	Set Timezone
set static-route default nexthop gateway address 192.168.29.2 priority 1 on	Set default gateway
set static-route 10.2.2.15/32 nexthop gateway address 192.168.0.1 on	Add static routes
set interface eth2 speed 100M duplex full active on --- add interface eth2c0 address 192.168.1.1/24 enable	Add an interface
set interface eth-s3/s1p1 active off
set hostname testbox	set hostname
set package name name [on | off]	set package name
add arpproxy address 192.168.1.1 macaddress 0:a0:1b:3e:33:f1	Add Proxy arp
add ntp server 10.1.1.2 version 3 prefer yes	Add an NTP server
add package media local name [opt/packages/IPSO-3.9.tgz]
add host name testbox ipv4 192.168.29.54	set hostname assignment

Checkpoint - Commands

Checkpoint commands generally come under,
cp - general
fw - firewall
fwm - management
CP, FW & FWM Commands
cphaprob stat	List cluster status
cphaprob -a if	List status of interfaces
cphaprob syncstat	shows the sync status
cphaprob list	Shows a status in list form
cphastart/stop	Stops clustering on the specfic node
cp_conf sic	SIC stuff
cpconfig	config util
cplic print	prints the license
cprestart	Restarts all Checkpoint Services
cpstart	Starts all Checkpoint Services
cpstop	Stops all Checkpoint Services
cpstop -fwflag -proc	Stops all checkpoint Services but keeps policy active in kernel
cpwd_admin list	List checkpoint processes
cplic print	Print all the licensing information.
cpstat -f all polsrv	Show VPN Policy Server Stats
cpstat	Shows the status of the firewall
fw tab -t sam_blocked_ips	Block IPS via SmartTracker
fw tab -t connections -s	Show connection stats
fw tab -t connections -f	Show connections with IP instead of HEX
fw tab -t fwx_alloc -f	Show fwx_alloc with IP instead of HEX
fw tab -t peers_count -s	Shows VPN stats
fw tab -t userc_users -s	Shows VPN stats
fw checklic	Check license details
fw ctl get int [global kernel parameter]	Shows the current value of a global kernel parameter
fw ctl set int [global kernel parameter] [value]	Sets the current value of a global keneral parameter. Only Temp ; Cleared after reboot.
fw ctl arp	Shows arp table
fw ctl install	Install hosts internal interfaces
fw ctl ip_forwarding	Control IP forwarding
fw ctl pstat	System Resource stats
fw ctl uninstall	Uninstall hosts internal interfaces
fw exportlog .o	Export current log file to ascii file
fw fetch	Fetch security policy and install
fw fetch localhost	Installs (on gateway) the last installed policy.
fw hastat	Shows Cluster statistics
fw lichosts	Display protected hosts
fw log -f	Tail the current log file
fw log -s -e	Retrieve logs between times
fw logswitch	Rotate current log file
fw lslogs	Display remote machine log-file list
fw monitor	Packet sniffer
fw printlic -p	Print current Firewall modules
fw printlic	Print current license details
fw putkey	Install authenication key onto host
fw stat -l	Long stat list, shows which policies are installed
fw stat -s	Short stat list, shows which policies are installed
fw unloadlocal	Unload policy
fw ver -k	Returns version, patch info and Kernal info
fwstart	Starts the firewall
fwstop	Stop the firewall
fwm lock_admin -v	View locked admin accounts
fwm dbexport -f user.txt	used to export users , can also use dbimport
fwm_start	starts the management processes
fwm -p	Print a list of Admin users
fwm -a	Adds an Admin
fwm -r	Delete an administrator
Provider 1
mdsenv [cma name]	Sets the mds environment
mcd	Changes your directory to that of the environment.
mds_setup	To setup MDS Servers
mdsconfig	Alternative to cpconfig for MDS servers
mdsstat	To see the processes status
mdsstart_customer [cma name]	To start cma
mdsstop_customer [cma name]	To stop cma
cma_migrate	To migrate an Smart center server to CMA
cmamigrate_assist	If you dont want to go through the pain of tar/zip/ftp and if you wish to enable FTP on Smart center server
VPN
vpn tu	VPN utility, allows you to rekey vpn
vpn ipafile_check ipassignment.conf detail‏	Verifies the ipassignment.conf file
dtps lic	show desktop policy license status
cpstat -f all polsrv	show status of the dtps
vpn shell /tunnels/delete/IKE/peer/[peer ip]	delete IKE SA
vpn shell /tunnels/delete/IPsec/peer/[peer ip]	delete Phase 2 SA
vpn shell /show/tunnels/ike/peer/[peer ip]	show IKE SA
vpn shell /show/tunnels/ipsec/peer/[peer ip]	show Phase 2 SA
vpn shell show interface detailed [VTI name]	show VTI detail
Debugging
fw ctl zdebug drop	shows dropped packets in realtime / gives reason for drop
SPLAT Only
router	Enters router mode for use on Secure Platform Pro for advanced routing options
patch add cd	Allows you to mount an iso and upgrade your checkpoint software (SPLAT Only)
backup	Allows you to preform a system operating system backup
restore	Allows you to restore your backup
snapshot	Performs a system backup which includes all Checkpoint binaries. Note : This issues a cpstop.
VSX
vsx get [vsys name/id]	get the current context
vsx set [vsys name/id]	set your context
fw -vs [vsys id] getifs	show the interfaces for a virtual device
fw vsx stat -l	shows a list of the virtual devices and installed policies
fw vsx stat -v	shows a list of the virtual devices and installed policies (verbose)
reset_gw	resets the gateway, clearing all previous virtual devices and settings.