Adapting Logwatch scripts

Logwatch is a linux tool that usually runs automathicaly through cron parsing through servers logs to creates a customized report analyzing specified areas about the activity of a server.

Recently we got a new server, hereinafter NewServer, and we realized that logwatch was behaving weirdly. We were not getting the desired information as we did on other servers, hereinafter OldServer, so many information given by logwatch in other servers was not present. To make a long story short, it turned out that the /var/log/ files format (secure, syslog,...) was different. We have not identify yet where does this difference come from. It could be because the operative system in NewServer is a newer version, but why the new logwatch is not compatible? Hereinafter, you will find more information about the problem and how did we fixed it.

On one hand the information given by OldServer is shown bellow:

################### Logwatch 7.3 (03/24/06) ####################
 Processing Initiated: Mon Feb 16 04:02:07 2015
  Date Range Processed: yesterday
   ( 2015-Feb-15 )
    Period is day.
 Detail Level of Output: 0
 Type of Output: unformatted
 Logfiles for Host: OldServer
##################################################################
 --------------------- sendmail-largeboxes (large mail spool files) Begin ------------------------
  Large Mailbox threshold: 40MB (41943040 bytes)
  Warning: Large mailbox: bsmuser (51198120)
  Warning: Large mailbox: nagios (51199606)
 ---------------------- sendmail-largeboxes (large mail spool files) End -------------------------
 --------------------- SSHD Begin ------------------------
  Users logging in through sshd:
  user1:
 xxx.xxx.xxx.x.111: 2 times
  user2:
 xxx.xxx.xxx.x.119: 2 times
    Received disconnect:
 11: Bye Bye : 2 Time(s)
 11: Closed due to user request. : 10 Time(s)
 11: disconnected by user : 6 Time(s)
 SFTP subsystem requests: 16 Time(s)
 ---------------------- SSHD End -------------------------
 --------------------- Disk Space Begin ------------------------
   Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup00-LogVol01
 142G  102G   34G  76% /
  /dev/mapper/VolGroup00-LogVol03
   95G   44G   47G  49% /opt
    /dev/mapper/VolGroup00-LogVol05
 98G   35G   58G  38% /usr
  ---------------------- Disk Space End -------------------------

Therefore, the information given by logwatch on the OldServer was just related with disk space, sshd, pam and mail services. In the other hand, in the information given by the NewServer only disk space is shown while other information was missed:

################### Logwatch 7.3.6 (05/19/07) ####################
 Processing Initiated: Sun Feb 15 03:40:02 2015
 Date Range Processed: yesterday
 ( 2015-Feb-14 )
 Period is day.
 Detail Level of Output: 0
 Type of Output: unformatted
 Logfiles for Host: OldServer
 ##################################################################
   --------------------- Disk Space Begin ------------------------
    Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/vg_n0-lv_root
 188G   28G  151G  16% /
 /dev/sda1             485M   81M  379M  18% /boot
 /dev/mapper/vg_n0-lv_opt
 375G  1.2G  355G   1% /opt
 /dev/mapper/vg_n0-lv_usr
 472G   29G  419G   7% /usr
 /dev/mapper/vg_n0-lv_var
 375G   40G  316G  12% /var
  ---------------------- Disk Space End -------------------------
 ###################### Logwatch End #########################

In the sake of clarity we are going to define the variable:

TMP_PATH=/usr/share/logwatch/scripts

We compared the logwatch filters on the different servers (/usr/share/logwatch/scripts), but all were identical, so it did not seem to be a logwatch problem, so, a deeper analysis seemed to be required in order to isolate the problem. Fortunately, this analysis turned to be quite simple because the logwatch’s “debug” option prints almost all the information that processes the command and allows you to execute it
line by line in the command line.

Part of the missed information was that related to ssh, so we choose that for debugging:

logwatch --service sendmail --range=Yesterday --debug 6

This command will print out many information on the screen but the important lines look like this:

cat /var/cache/logwatch/logwatch.aqP9adQd/secure | 
perl $TMP_PATH/shared/onlyservice 'sshd' |
perl $TMP_PATH/shared/removeheaders '' |
perl $TMP_PATH/services/sshd

Knowing this, we can now run those commands on the command line. For comparison purposes, we run it on NewServer and OldServer :

cat /var/log/secure | perl $TMP_PATH/shared/onlyservice 'sshd'

The first command provided some information on OldServer but none on NewServer. onlyservice script was identical on both servers, so obviously the problem was the /var/log/secure. It turned out that the format of this file was different on NewServer and that most of the logwatch scripts where not able to deal with it.

NewServer /var/log/secure looks like this:

1424159876 2015 Feb 17 08:57:56 newserver authpriv info sshd Accepted publickey for ...

while OldServer /var/log/secure:

Feb 17 10:33:47 s_sys@oldserver sshd[14805]: Accepted password for ....

Logwatch scripts are able to process the OldServer format, but not NewServer one. The first step then was to modify the $TMP_PATH/shared/onlyservice filter. It turned out to be a perl script and adding the proper regExp fixed the firsts step. In our case we added this line:

elsif ($ThisLine =~ m/^.......... .... ... .. ..:..:.. [ ]*[^ ]* [^ ]* [^ ]* $ServiceName/io){
    print $ThisLine

After this fix, we continue executing the second filter of the command:

cat /var/log/secure | perl $TMP_PATH/shared/onlyservice 'sshd' |
perl $TMP_PATH/shared/removeheaders

In this case the the filter was not removing the corresponding information, again, because the format was different. In this case, we added the following line to $TMP_PATH/shared/ remove headers

ThisLine =~ s/^.......... .... ... .. ..:..:.. .. ........ [^ ]* [^ ]* //;

Finally, we run the last filter from the command line:

cat /var/log/secure | perl $TMP_PATH/shared/onlyservice 'sshd' |
perl $TMP_PATH/shared/removeheaders |
perl $TMP_PATH/services/sshd

And now, we got a nice sshd logwatch output as the one we where getting on OldServer.

Failed logins from:
xxx.xxx.xxx.41 (xxx.xxx.xxx.es): 1 time
Illegal users from:
xxx.xxx.xxx.178 (xxx.xxx.xxx.es): 3 times
Users logging in through sshd:
   user1:
   xxx.xxx.xxx.254 (xxx.xxx.xxx.es): 14 times

In our case, we also modified $TMP_PATH/services/pam_unix, $TMP_PATH/services/secure/filters and added some strings to be ignored in /etc/logwatch/conf/ignore.conf.