Solution
The following article is a list of steps one should go through when troubleshooting logging related issues in a distributed setup.
1. Ensure that you have not run out of disk space on the hard disk that the logs are being sent to. If this is the case, delete or move the logs to an external storage device.2. Is there communication between the MS and the Module? Test using ping to the MS from the module and then from the Module to the MS (your rules must allow for this). If this fails, and your rules allow for this, then it is most likely a routing issue.
3. Check to see if the fw.log file is growing on the module. It should be if the logs are not going to the MS. From the console run these commands:
cd $FWDIR/log
ls -la
ls -la
Verify that the fw.log file is increasing. If it is increasing then the modules are logging locally instead of forwarding the traffic to the MS. This could be a connectivity issue, or it could be the way the logging is setup. Check the FW object to ensure it is setup to send logs to the MS.4. Can you fetch a policy? Verify that you can fetch using the hostname and IP address. If this fails then you probably have a SIC issue. To test this run the following commands:
fw fetch hostname_of_MS
fw fetch IP_Addr_of_MS (fetch by IP address also to ensure it is not a DNS issue)
5. Check the masters file. The hostname or IP address of the management station should be listed in there. To check this run the following commands:
cd $FWDIR/conf
cat masters
It should be look like this:
[Policy]
hostname_of_MS
[Log]
hostname_of_MS
[Alert]
hostname_of_MS
6. Run tcpdumps on the module, listening for port 257 on the interface facing the MS, to see if it is attempting to send logs. To check this run the following command:tcpdump -i eth-facing-MS port 257 (use the Ctrl+C to break out of the dump)
You should see traffic leaving the FW and heading to the IP address of the MS.
You should also see traffic coming back from the MS.
7. The log file may have gotten corrupt. Run a log switch on the MS and reboot the MS to create a new log file. If logswitch does not work, move all contents of the log directory (do not move the directory itself) to a temp folder outside of the log directory. Reboot and see if the logs start again.
8. Delete the $FWDIR/log files and $FWDIR/state directory files on the module; reboot the module.
Reboot and see if the logs start again.
9. Look to see if there is a listening port for logging. Run the following command on the MS and the module:
netstat -na
You should see the *.257 LISTEN for logging connections. You should also see the IP address of the MS :257 associated with the IP address of each module, and showing an ESTABLISHED connection.
10. Check the log settings for the FW object and make sure the 'Log Server' is set to the MS that should be receiving the logs. This is usually done by default, but may have been changed by a user.
If after going through these steps you are still experiencing logging issues, please open a ticket with Nokia TAC for further troubleshooting.
Post a Comment