The program is based on a model of independent **parsers** (consisting of Python modules) which analyse logs from a particular service. Logparse comes with a range of these built in, but additional parsers can be written in Python and placed in `/usr/share/logparse/parsers`. At the moment, the built-in parsers are:
-- cron (DEPRECATED) - number of commands, list commands (root user only)
-- cron-journald - number of commands, list commands, list commands per user (requires libsystemd)
-- httpd - list requests, clients, user agents, bytes transferred, no. of errors
+####
+cron
+####
+
+.. automodule:: logparse.parsers.cron
+
+####
+httpd
+####
+
+.. automodule:: logparse.parsers.httpd
+
- mem - get installed/usable/free memory
- postfix - list recipients and bytes sent
- smbd - number of logins, list users & clients
- sudo (DEPRECATED)- number of sessions, list users and commands
- sudo-journald - number of sessions, list users and commands (requires libsystemd)
- sysinfo - hostname, OS, OS version, platform, processors
-- systemctl - system status, running/failed units (requires libsystemd)
+- systemd - system status, running/failed units (requires libsystemd)
- temperature - instantaneous temperatures of motherboard, CPU, cores, disks
- ufw - blocked packets, port and IP data (requires libsystemd)
- zfs - zpool scrub reports, disk usage
Space-separated list of parsers to enable. If empty, all the included parsers are run except for deprecated ones. Analogous to the command line option -l|--logs. Default: empty
ignore-parsers
Space-separated list of parsers to ignore (i.e. not execute). If empty, no parsers are ignored. Analogous to the command line option -nl|--ignore-logs. Default: empty
+datetime-format
+ String representing the format for dates and times in the logfiles (when using plain logfiles, not journald parsers). This should be a standard Python strptime format (see <https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior>). Supported parsers allow the datetime-format to be set in their individual sections, which overrides this global value. Default: %b %d %H:%M:%S
##############################################
HTML specific configuration (`[html]` section)
Each parser has its own set of options in a section with the name of the parser. In the case of multiple versions of the same parser (e.g. sshd and sshd-journald), the configuration section goes by the base name (e.g. sshd). Options defined in individual parser sections override those defined in the global configuration.
-####
-cron
-####
+######################
+cron and cron_journald
+######################
+commands
+ Regular expression string for which commands to include when parsing logs. If `truncate-commands` is set to true, then the truncated command will be compared against the regex pattern, not the full command. Default: `.*`
+datetime-format
+ String representing the format for dates and times in the logfiles (when using cron, not cron_journald). This should be a standard Python strptime format (see <https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior>). Reverts to the global config if empty. Default: empty
+list-users
+ Display a list of the most common commands run by each user (this may be combined with the `summary` option below). Default: true
period
- Maximum age of logs to analyse. Overrides global config. Only used in cron-journald at the moment. See :ref:`period` for more information. Default: empty
+ Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
+summary
+ Show a summary of cron sessions. This consists of the total number of sessions, total number of users, and a list of the most popular commands and who executed them. Default: false
+truncate-commands
+ Whether to remove absolute directory paths in commands. When set to true, a command such as `/usr/bin/cat` will become `cat`. Default: true
+users
+ Regular expression string for which usernames to include when parsing logs. This could be used to exclude cron sessions from trusted users. Default: `.*`
-.. _period:
-####
-sshd
-####
+######################
+sshd and sshd_journald
+######################
period
Maximum age of logs to analyse. Overrides global config. Only used in sshd-journald at the moment. See :ref:`period` for more information. Default: empty
sshd-resolve-domains
DNS lookup configuration for sshd parsers only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty
-####
-smbd
-####
+######################
+smbd and smbd_journald
+######################
shares
Regular expression string for which Samba shares to include when parsing logs. To consider all shares, set this to `.*`. To exclude a certain share, use negative lookaround. Default: `^((?!IPC\$).)*$`
period
Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
-####
-sudo
-####
+######################
+sudo and sudo_journald
+######################
+list-users
+ Display a list of the most common commands initiated by each user (this may be combined with the `summary` option below). Default: true
period
Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
+summary
+ Show a summary of sudo sessions and most popular commands. Default: false
+truncate-commands
+ Whether to remove absolute directory paths in commands. When set to true, a command such as `/usr/bin/cat` will become `cat`. Default: true
-#########
-systemctl
-#########
+#######
+systemd
+#######
period
Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
Whether to include services which are running but okay in the output. Default: true
+.. _period:
========================
Log period configuration
<body>
<table width=100%>
<tr>
- <td><h1>$title</h1></td>
+ <td><h1>$title</h1><p>Parsing logs since $period</p></td>
<td style="text-align: right;">$hostname<br />$date $time</td>
</tr>
</table>
}
# Template for formatting log messages
-FORMAT = ("{bold}%(name)-15s{reset} %(levelname)-18s %(message)s "
+FORMAT = ("{bold}%(module)-15s{reset} %(levelname)-18s %(message)s "
"({bold}%(filename)s{reset}:%(lineno)d)")
'hostname-path': '/etc/hostname',
'parsers': '',
'ignore-parsers': '',
- 'period': '1 minute'
+ 'period': '1 week',
+ 'datetime-format': "%%b %%d %%H:%%M:%%S",
+ 'journald': True
},
'html': {
'header': '/etc/logparse/header.html',
'httpd-error': '/var/log/apache2/error.log'
},
'cron': {
- 'period': ''
+ 'summary': False,
+ 'list-users': True,
+ 'period': '',
+ 'datetime-format': '',
+ 'truncate-commands': True,
+ 'users': '.*',
+ 'commands': '.*'
},
'mail': {
'to': '',
},
'httpd': {
'httpd-resolve-domains': '',
- 'period': ''
+ 'datetime-format': "%%d/%%b/%%Y:%%H:%%M:%%S %%z",
+ 'period': '',
+ 'clients': '.*',
+ 'files': '.*',
+ 'referrers': '.*',
+ 'access-format': "%%h %%l %%u %%t \"%%r\" %%>s %%O \"%%{Referer}i\" \"%%{User-Agent}i\""
},
'du': {
'paths': ['/', '/etc', '/home'],
'period': ''
},
'sudo': {
+ 'journald': '',
+ 'datetime-format': '',
'period': '',
'list-users': True,
'summary': True,
- 'truncate-commands': True
+ 'truncate-commands': True,
+ 'init-users': '.*',
+ 'superusers': '.*',
+ 'commands': '.*',
+ 'directories': '.*'
+
},
- 'systemctl': {
+ 'systemd': {
'period': '',
'show-all': True
}
prefs.read_dict(defaults)
try:
success = prefs.read(configpaths)
- logger.debug("Loaded {0} config file(s): {1}".format(str(len(success)), str(success)))
+ logger.debug("Loaded {0} config file(s): {1}".format(
+ str(len(success)), str(success)))
except Exception as e:
logger.warning("Error processing config: " + str(e))
return prefs
from string import Template
from math import floor, ceil
from tabulate import tabulate
+import textwrap
import logparse
from logparse import interface, util, mail, config
JXNCHARS_SINGLE = ['├', '┤', '┬', '┴', '┼']
BULLET = "• "
INDENT = " "
+SPLIT_CHARS = ['.', '(', ')', '[', ']', '&', r"/", "\\", ',', '-', '_']
global VARSUBST
"hostname": util.hostname(config.prefs.get(
"logparse", "hostname-path")),
"version": logparse.__version__,
- "css": css_path
+ "css": css_path,
+ "period": util.LogPeriod("logparse").startdate.strftime(
+ TIMEFMT + " " + DATEFMT)
}
Print details with some primitive formatting
"""
box = PlaintextBox(content=
- Template("$title $version on $hostname\n\n$time $date")
+ Template("$title $version on $hostname\n\n$time $date"
+ "\nParsing logs since $period")
.safe_substitute(VARSUBST),
vpadding=2, hpadding="\t\t", linewidth=self.linewidth)
- line = PlaintextLine(self.linewidth)
- self.append(box.draw() + line.draw())
+ self.append(box.draw() + "\n"*2)
def append_footer(self):
"""
Append a horizontal line and some details
"""
- self.append(PlaintextLine(self.linewidth, vpadding=1).draw())
+ self.append(PlaintextLine(self.linewidth).draw())
self.append(Template("$hostname $time $date").safe_substitute(VARSUBST))
def append_section(self, section):
logger.warning("No subtitle provided.. skipping section")
return
+ logger.debug("Processing data {}".format(subtitle))
+
if (data == None or len(data) == 0):
- logger.debug("No data provided.. just printing subtitle")
- return subtitle + '\n'
+ # If no list items are provided, just print the subtitle
+ return subtitle + "\n"
+ elif (len(data) == 1):
+ # If only one item is provided, print it inline with subtitle
+ return self._wrap_datum("{}: {}".format(subtitle, data[0]),
+ bullet=False, indent=False) + "\n"
else:
- logger.debug("Received data " + str(data))
- subtitle += ':'
- if (len(data) == 1):
- return subtitle + ' ' + data[0] + '\n'
- else:
- itemoutput = subtitle + '\n'
- for datum in data:
- datum = BULLET + datum
- if len(datum) > self.linewidth - 3:
- words = datum.split()
- if max(map(len, words)) > self.linewidth - len(INDENT):
- continue
- res, part, others = [], words[0], words[1:]
- for word in others:
- if 1 + len(word) > self.linewidth - len(part):
- res.append(part)
- part = word
- else:
- part += ' ' + word
- if part:
- res.append(part)
- datum = ('\n ').join(res)
- datum = INDENT + datum
- itemoutput += datum + '\n'
- return itemoutput
+ # If many items are provided, print them all as a bulleted list
+ itemoutput = subtitle + ":\n"
+ for datum in data:
+ itemoutput += self._wrap_datum(datum) + "\n"
+ return itemoutput
+
+ def _wrap_datum(self, text, bullet=True, indent=True):
+ """
+ Use cpython's textwrap module to limit line width to the value
+ specified in self.linewidth. This is much easier than doing it all
+ from scratch (which I tried to do originally). Note that line
+ continuations are automatically indented even if they don't have a
+ bullet. This is to make it clear which lines are continuations.
+ """
+
+ wrapper = textwrap.TextWrapper(
+ initial_indent=(INDENT if indent else "") \
+ + (BULLET if bullet else ""),
+ subsequent_indent=INDENT + (' '*len(BULLET) if bullet else ""),
+ width=self.linewidth,
+ replace_whitespace=True)
+
+ return wrapper.fill(text)
+
class HtmlOutput(Output):
"""
logger.debug("No data provided.. just printing subtitle")
return tag('p', False, subtitle, cl="severity-" + str(severity))
else:
- logger.debug("Received data " + str(data))
+ logger.debug("Received data {}: {}".format(subtitle, data))
subtitle += ':'
if (len(data) == 1):
return tag('p', False, subtitle + ' ' + data[0],
def truncl(self, limit): # truncate list
"""
Truncate self.items to a specified value and state how many items are
- hidden.
+ hidden. Set limit to -1 to avoid truncating any items.
"""
+ if limit == -1:
+ return self
if (len(self.items) > limit):
more = len(self.items) - limit
if more == 1:
class Column(object):
"""
- Object representing a single table cell. This is somewhat of a misnomer -
- one column object exists for each cell in the table. Columns are children
+ Object representing a single table cell. "Column" is somewhat of a misnomer
+ - one column object exists for each cell in the table. Columns are children
of rows.
"""
Draw a horizontal line for plain text format, with optional padding/styling.
"""
- def __init__(self, linewidth=80, double=True, vpadding=1, hpadding=""):
+ def __init__(self, linewidth=80, double=True, vpadding=0, hpadding=""):
"""
Initialise variables
"""
line = (LINECHARS_DOUBLE[1] if self.double else LINECHARS_SINGLE[1])
return "\n" * self.vpadding + self.hpadding \
+ line * (self.linewidth - 2 * len(self.hpadding)) \
- + self.hpadding + "\n" * self.vpadding
+ + self.hpadding + "\n" * (self.vpadding + 1)
class PlaintextBox:
contentwidth = int((self.linewidth if self.linewidth > 0 else 80)
if self.content.splitlines()
else len(max(contentlines, key=len)))
- logger.debug("Contentwidth is {0}".format(str(contentwidth)))
+ logger.debug("Content width is {0}".format(str(contentwidth)))
logger.debug("Longest line is {0}".format(
len(max(contentlines, key=len))))
contentwidth += -2*(len(self.hpadding)+1)
import logging
import logging.handlers
import os
-from sys import stdin, version
+from sys import exit, stdin, version
from subprocess import check_output
from datetime import datetime
# Set up parsers
loader = load_parsers.ParserLoader()
+
+ try:
+ loader.check_systemd()
+ except Exception as e:
+ logger.error("Failed to check systemd dependencies: ".format(e))
+
if parser_names:
for parser_name in parser_names:
if parser_name not in ignore_logs:
# Execute parsers
+ executed_parsers = []
+
for parser in loader.parsers:
- output.append_section(parser.parse_log())
+ if (argparser.parse_args().verbose
+ or config.prefs.getboolean("logparse", "verbose")):
+ output.append_section(parser.parse_log())
+
+ else:
+ try:
+ output.append_section(parser.parse_log())
+ except Exception as e:
+ logger.error("Uncaught error executing logger {0}: {1}".format(
+ parser.name, e))
+ executed_parsers.append(parser.name)
+
+ if len(executed_parsers) == 0:
+ exit()
# Write footer
output.append_footer()
return
+
def get_argparser():
"""
Initialise arguments (in a separate function for documentation purposes)
"""
import importlib
+from importlib import util
from os.path import dirname
from pkgutil import iter_modules
import inspect
from pathlib import Path
+import subprocess
+from subprocess import Popen, PIPE
from typing import get_type_hints
import logging
"""
raise NotImplementedError("Failed to find an entry point for parser")
+ def check_dependencies(self) -> tuple:
+ """
+ Parsers should check their own requirements here and return a boolean
+ value reflecting whether the parser can run successfully. Typically
+ this method should check for the program whose logs are being parsed,
+ as well as any external dependencies like libsystemd. This method
+ should return a tuple containing a boolean representing whether or not
+ the dependencies are satisfied and list containing the names of any
+ dependencies that are unsatisfied.
+ """
+ return (True, None)
+
+ def _check_dependency_command(self, cmdline) -> tuple:
+ """
+ Runs a shell command (typically something --version) and returns the
+ output and return code as a tuple. The command to run is passed as a
+ string, optionally including arguments, in the `cmdline` argument.
+ """
+ logger.debug("Checking output of command " + cmdline)
+ cmd = subprocess.getstatusoutput(cmdline)
+ if cmd[0] != 0:
+ logger.warning("{0} is not available on this system (`{1}` "
+ "returned code {2}: \"{3}\")".format(
+ cmdline.split()[0], cmdline, *cmd))
+ return cmd
+ else:
+ logger.debug("Command {0} succeeded".format(cmdline))
+ return cmd
+
class ParserLoader:
self.pkg = pkg
self.path = path
self.parsers = []
+ self.has_systemd = False
def search(self, pattern):
"""
- Basic wrapper for the two search functions below.
+ Find a parser and determine its journald attribute. When a user
+ requests a parser of the form .*_journald, this function will use
+ that parser if it exists, but if not it will revert to using the
+ base parser (without the _journald) if it has a journald attribute.
+ If it does not have this error (and the parser as requested does not
+ exist), then no parser is loaded..
"""
+ # Separate into underscore words
+ split_name = pattern.split("_")
+
+ # Check if parser exists with exact name requested by user
+ result = self._search_both(pattern)
+
+ if result == None and split_name[-1] == "journald":
+ # No match for exact name but .*_journald was requested...
+ if self.has_systemd:
+ # Look for base parser with journald attribute
+ result = self._search_both("".join(split_name[:-1]))
+ if result == None:
+ logger.error("Couldn't find a matching parser module "
+ "for {0}".format(pattern))
+ if not hasattr(result, "journald"):
+ logger.error("Found parser {} but it does not support "
+ "journald".format("".join(split_name[:-1])))
+ result = None
+ else:
+ result.journald = True
+ else:
+ logger.error("A parser that requires systemd was requested "
+ "but the dependencies are not installed.")
+ return None
+ if not result.deps_ok:
+ return None
+
+ if result == None:
+ # Still can't find a matching parser
+ logger.error("Couldn't find a matching parser module "
+ "for {0}".format(pattern))
+ else:
+ self.parsers.append(result)
+
+ return result
+
+ def _search_both(self, pattern):
+ """
+ Basic wrapper for the two search functions below.
+ """
default_parser = self._search_default(pattern)
if default_parser != None:
- self.parsers.append(default_parser)
return default_parser
else:
user_parser = self._search_user(pattern)
if user_parser != None:
- self.parsers.append(user_parser)
return user_parser
else:
- logger.warning("Couldn't find a matching parser module "
- "for search term {0}".format(pattern))
return None
def _search_user(self, pattern):
4. Must provide the parse_log() method
5. Must not return None
6. Must not match an already-loaded class
+ 7. Dependencies must exist
"""
logger.debug("Checking validity of module {0} at {1}".format(
parser_module.__name__, parser_module.__file__))
available_parsers = []
+ missing_dependencies = []
clsmembers = inspect.getmembers(parser_module, inspect.isclass)
# Check individual classes
if not (issubclass(c, Parser) & (c is not Parser)):
continue
if c in self.parsers:
- logger.warning("Parser class {0} has already been loaded "
+ logger.error("Parser class {0} has already been loaded "
"from another source, ignoring it".format(
c.__class__.__name__, c.__file__))
if not inspect.isroutine(c.parse_log):
- logger.warning("Parser class {0} in {1} does not contain a "
+ logger.error("Parser class {0} in {1} does not contain a "
"parse_log() method".format(
c.__class__.__name__, c.__file__))
continue
if None in get_type_hints(c):
- logger.warning("Parser class {0} in {1} contains a "
+ logger.error("Parser class {0} in {1} contains a "
"null-returning parse_log() method".format(
c.__class__.__name__, c.__file__))
continue
logger.warning("Parser {0} is deprecated - "
"use {1} instead".format(
parser_obj.name, parser_obj.successor))
+ # Check dependencies
+ deps = parser_obj.check_dependencies()
+ if deps[0]:
+ parser_obj.deps_ok = True
+ else:
+ logger.error("The following dependencies are missing for "
+ "parser {0}: {1}".format(parser_obj.name,
+ ", ".join(deps[1])))
+ missing_dependencies.append(parser_obj)
+ parser_obj.deps_ok = False
+
logger.debug("Found parser {0}.{1}".format(
c.__module__, c.__class__.__name__))
available_parsers.append(parser_obj)
# Check module structure
+ if len(available_parsers) > 1:
+ logger.error("Found multiple valid parser classes in {0} at {1} "
+ "- ignoring this module"
+ .format(parser_module.__name__, parser_module.__file__))
+ return None
+ elif len(available_parsers) == 0:
+ if len(missing_dependencies) > 0:
+ return None
+ logger.error("No valid classes in {0} at {1}".
+ format(parser_module.__name__, parser_module.__file__))
+ return None
if len(available_parsers) == 1:
logger.debug("Parser module {0} at {1} passed validity checks"
.format(parser_module.__name__, parser_module.__file__))
return available_parsers[0]
- elif len(available_parsers) == 0:
- logger.warning("No valid classes in {0} at {1}".
- format(parser_module.__name__, parser_module.__file__))
- return None
- elif len(available_parsers) > 1:
- logger.warning("Found multiple valid parser classes in {0} at {1} "
- "- ignoring this module"
- .format(parser_module.__name__, parser_module.__file__))
- return None
+
+ def check_systemd(self):
+ """
+ Check if the appropriate dependencies are installed for parsing
+ systemd logs.
+
+ Output codes:
+ 0. systemd + libsystemd + systemd-python are installed
+ 1. systemd + libsystemd are installed
+ 2. systemd is installed
+ 3. systemd is not installed, no support required
+ """
+ # Test if systemctl works
+ systemctl_cmd = Popen(["systemctl", "--version"], stdout=PIPE)
+ systemctl_cmd.communicate()
+
+ if systemctl_cmd.returncode == 0:
+ logger.debug("Passed systemctl test")
+
+ # Test if libsystemd exists
+ libsystemd_cmd = Popen(["locate", "libsystemd.so.0"], stdout=PIPE)
+ libsystemd_cmd.communicate()
+
+ if libsystemd_cmd.returncode == 0:
+ logger.debug("Passed libsystemd test")
+
+ # Test if systemd-python exists
+ if util.find_spec("systemd") is not None:
+ logger.debug("Passed systemd-python test")
+ self.has_systemd = True
+ logger.debug("Passed all systemd dependency checks")
+ return 0
+ else:
+ logger.warning("Systemd is running on this system but the "
+ "package systemd-python is not installed. Parsers "
+ "that use journald will not work. For more "
+ "features, install systemd-python from "
+ "<https://pypi.org/project/systemd-python/> or "
+ "`pip install systemd-python`.")
+ return 1
+ else:
+ logger.warning("Systemd is running on this system but "
+ "libsystemd headers are missing. This package is "
+ "required to make use of the journald parsers. "
+ "Libsystemd should be available with your package "
+ "manager of choice.")
+ return 2
+ else:
+ logger.debug("Systemd not installed.. parsers that use journald "
+ "will not work.")
+ return 3
+
def load_pkg(self):
"""
-#
-# cron.py
-#
-# List the logged (executed) cron jobs and their commands (uses syslog file)
-#
-# NOTE: This file is now deprecated in favour of the newer journald mechanism
-# used in cron-journald.py. This parser is still functional but is slower and
-# has less features. Please switch over if possible.
-#
+"""
+Get information about executed cron commands - number of commands, list
+of commands, and list of commands per user. Uses either journald or plain
+logfiles (path specified in config).
+NOTE: This parser supports reading from both journald and plain syslog files.
+By default the plain logfiles will be used, but the journald option is
+preferred for newer systems which support it. To use the journald mode,
+specify the parser as `cron_journald` instead of `cron`.
+
+NOTE: If using journald, the log level for cron.service should be at least 2
+(default is 1). This can be changed with `sudo systemctl edit cron --full`,
+and ammend `-L 2` to the ExecStart command.
+
+TODO: also output a list of scheduled (future) jobs
+"""
+
+import datetime
import re
from logparse.formatting import *
from logparse.util import readlog
from logparse import config
from logparse.load_parsers import Parser
-from logparse.load_parsers import Parser
+
+class CronCommand:
+ """
+ Class representing a single cron session. Assigns its own variables of
+ date, user and cmd when given a `systemd.journal.Record` object or a plain
+ log message string on initialisation. NOTE: This class is used in both
+ `cron.py` and `cron_journald.py`.
+ """
+
+ def __init__(self, record, datefmt=""):
+ """
+ Parse the date, user and command from the logfile string or record
+ """
+ if isinstance(record, str):
+ if not datefmt:
+ logger.error("Date format not provided - cannot parse this "
+ "log message")
+ # Parse from a raw logfile string
+ self.date, self.user, self.cmd = re.search(
+ r"^(?P<time>.+)\s\w+\sCRON"
+ "\[\d+\]:\s\((?P<user>\S*)\)\sCMD\s\(+(\[\d+\]\s)?(?P<cmd>.*)\)+",
+ record).groupdict().values()
+ self.date = datetime.datetime.strptime(self.date, datefmt)
+ if not "Y" in datefmt:
+ self.date = self.date.replace(year=datetime.datetime.now().year)
+ elif isinstance(record, dict):
+ self.date = record["_SOURCE_REALTIME_TIMESTAMP"]
+ self.user, self.cmd = re.search(r"\((?P<user>\S+)\) "
+ "CMD \((\[\d+\] )?(?P<cmd>.*)\)", record["MESSAGE"]) \
+ .groupdict().values()
+ self.cmd = " ".join(self.cmd.split())
+ else:
+ raise TypeError("record should be str or dict")
+
+ def truncate(self):
+ """
+ Hide the full directory path for any scripts or explicit binary
+ references in the command. e.g. `/usr/bin/cat` → `cat`
+ """
+ self.cmd = re.sub(r"(\s|^)/\S*/(\S+)", r"\1\2", self.cmd)
+ return self.cmd
+
+ def match_user(self, pattern):
+ """
+ Check if the user of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ user_match = False
+ for p in pattern:
+ user_match = re.fullmatch(p, self.user) \
+ or user_match
+ return user_match
+
+ def match_cmd(self, pattern):
+ """
+ Check if the command of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ cmd_match = False
+ for p in pattern:
+ cmd_match = re.fullmatch(p, self.user) \
+ or cmd_match
+ return cmd_match
+
class Cron(Parser):
def __init__(self):
super().__init__()
self.name = "cron"
- self.info = "List the logged (executed) cron jobs and their commands (uses static syslog file)"
- self.deprecated = True
- self.successor = "cron_journald"
+ self.info = "List the logged (executed) cron jobs and their commands"
+ self.journald = False
- def parse_log(self):
+ def _get_journald(self, startdate):
+ from systemd import journal
+ j = journal.Reader()
+ j.this_machine()
+ j.log_level(journal.LOG_INFO)
+ j.add_match(_COMM="cron")
+ j.seek_realtime(startdate)
+ return [entry for entry in j if "MESSAGE" in entry
+ and " CMD " in entry["MESSAGE"]]
+
+ def _get_logfile(self, path):
+ from logparse.util import readlog
+ return [x for x in readlog(path).splitlines() if " CMD " in x]
- logger.warning("NOTE: This cron parser is now deprecated. Please use cron-journald if possible.")
+ def parse_log(self):
logger.debug("Starting cron section")
section = Section("cron")
- matches = re.findall('.*CMD\s*\(\s*(?!.*cd)(.*)\)', readlog(config.prefs.get("logs", "cron")))
- num = len(matches)
- commands = []
- for match in matches:
- commands.append(str(match))
- logger.info("Found " + str(num) + " cron jobs")
- jobs_data = Data(str(num) + " cron jobs run")
- section.append_data(jobs_data)
-
- if (num > 0):
- logger.debug("Analysing cron commands")
- cmd_data = Data("Top cron commands")
- cmd_data.items = ("`{0}`".format(x) for x in commands)
- cmd_data.orderbyfreq()
- cmd_data.truncl(config.prefs.getint("logparse", "maxcmd"))
- section.append_data(cmd_data)
+ if not (config.prefs.getboolean("cron", "summary")
+ or config.prefs.getboolean("cron", "list-users")):
+ logger.warning("Both summary and list-users configuration options "
+ "are set to false, so no output will be generated. "
+ "Skipping this parser.")
+ return None
+
+ datefmt = config.prefs.get("cron", "datetime-format")
+ if not datefmt:
+ datefmt = config.prefs.get("logparse", "datetime-format")
+ if not datefmt:
+ logger.error("Invalid datetime-format configuration parameter")
+ return None
+
+ command_objects = []
+ users = {}
+ oldlog_buffer = 0
+
+ if self.journald:
+ logger.debug("Searching for cron commands in journald")
+ messages = self._get_journald(section.period.startdate)
+ else:
+ logger.debug("Searching for matches in {0}".format(
+ config.prefs.get("logs", "cron")))
+ messages = self._get_logfile(config.prefs.get("logs", "cron"))
+
+ if len(messages) < 1:
+ logger.error("Couldn't find any cron log messages")
+ return
+
+ for msg in messages:
+
+ try:
+ cmd_obj = CronCommand(msg, datefmt)
+ except Exception as e:
+ logger.warning("Malformed cron session log: {0}. "
+ "Error message: {1}".format(msg, str(e)))
+ continue
+ else:
+ if cmd_obj.date < section.period.startdate:
+ continue
+ if not (cmd_obj.match_user(config.prefs.get("cron", "users")
+ .split()) and cmd_obj.match_cmd(config.prefs.get(
+ "cron", "commands").split())):
+ logger.debug("Ignoring cron session by {0} with command "
+ "{1} due to config".format(cmd_obj.user, cmd_obj.cmd))
+ continue
+
+ if config.prefs.getboolean("cron", "truncate-commands"):
+ cmd_obj.truncate()
+
+ command_objects.append(cmd_obj)
+ if not cmd_obj.user in users:
+ users[cmd_obj.user] = []
+ users[cmd_obj.user].append(cmd_obj.cmd)
+
+ if len(command_objects) == 0:
+ logger.error("No valid cron commands found")
+ return
+
+ logger.info("Found {0} cron jobs".format(len(command_objects)))
+
+ if config.prefs.getboolean("cron", "summary"):
+ summary_data = Data()
+ summary_data.subtitle = "Total of " + plural("cron session",
+ len(command_objects)) + " for " + plural("user",
+ len(users))
+ summary_data.items = ["{}: `{}`".format(c.user, c.cmd)
+ for c in command_objects]
+ summary_data.orderbyfreq()
+ summary_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(summary_data)
+
+ if config.prefs.getboolean("cron", "list-users"):
+ for user, cmdlist in users.items():
+ user_data = Data()
+ user_data.subtitle = plural("session", len(cmdlist)) \
+ + " for " + user + (" (" + plural("unique command",
+ len(set(cmdlist))) + ")" if len(set(cmdlist)) > 1
+ else "")
+ user_data.items = ["`{}`".format(cmd) for cmd in cmdlist]
+ user_data.orderbyfreq()
+ user_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(user_data)
logger.info("Finished cron section")
return section
-#
-# cron_journald.py
-#
-# List the logged (executed) cron jobs and their commands (uses journald module)
-#
-# TODO: also output a list of scheduled (future) jobs
-#
+# -*- coding: utf-8 -*-
+
+"""
+List the logged (executed) cron jobs and their commands (uses journald/logfile)
+NOTE: If using journald, the log level for cron.service should be at least 2
+(default is 1). This can be changed with `sudo systemctl edit cron --full`,
+and ammend `-L 2` to the ExecStart command.
+TODO: also output a list of scheduled (future) jobs
+"""
from systemd import journal
+import datetime
from logparse import config
from logparse.formatting import *
from logparse.load_parsers import Parser
+from logparse.parsers.cron import CronCommand
+
class CronJournald(Parser):
def __init__(self):
super().__init__()
self.name = "cron_journald"
- self.info = "List the logged (executed) cron jobs and their commands (uses journald module)"
+ self.info = "List the logged (executed) cron jobs and their commands " \
+ "(uses journald module)"
def parse_log(self):
logger.info("Obtaining cron logs")
- messages = [entry["MESSAGE"] for entry in j if "MESSAGE" in entry and " CMD " in entry["MESSAGE"]]
-
- total_jobs = len(messages)
+ records = [entry for entry in j
+ if "MESSAGE" in entry and " CMD " in entry["MESSAGE"]]
- if total_jobs == 0:
+ if len(records) == 0:
logger.warning("Couldn't find any cron commands")
return 1
- logger.info("Found " + str(total_jobs) + " cron jobs")
- section.append_data(Data("Total of " + plural("cron session", total_jobs) + " executed across all users"))
+ logger.info("Found {0} log records".format(len(records)))
logger.debug("Analysing cron commands for each user")
+ command_objects = []
users = {}
+ for record in records:
+ if record["_SOURCE_REALTIME_TIMESTAMP"] < section.period.startdate:
+ logger.warning("Discarding log record from {0} - was "
+ "seek_realtime set properly?".format(
+ record["_SOURCE_REALTIME_TIMESTAMP"]))
+ continue
+ try:
+ cmd_obj = CronCommand(record)
+ if config.prefs.getboolean("cron", "truncate-commands"):
+ cmd_obj.truncate()
+ if not (cmd_obj.match_user(config.prefs.get("cron", "users")
+ .split()) and cmd_obj.match_cmd(config.prefs.get(
+ "cron", "commands").split())):
+ logger.debug("Ignoring cron session by {0} with command "
+ "{1} due to config".format(cmd_obj.user, cmd_obj.cmd))
+ continue
+ command_objects.append(cmd_obj)
+ if not cmd_obj.user in users:
+ users[cmd_obj.user] = []
+ users[cmd_obj.user].append(cmd_obj.cmd)
+ except Exception as e:
+ logger.warning("Malformed cron log message: {0}. "
+ "Error message: {1}".format(record["MESSAGE"], str(e)))
+ continue
+
+ logger.info("Found {0} valid cron sessions".format(len(command_objects)))
+
+ if config.prefs.getboolean("cron", "summary"):
+ summary_data = Data()
+ summary_data.subtitle = "Total of " + plural("cron session",
+ len(command_objects)) + " for " + plural("user",
+ len(users))
+ summary_data.items = ["{}: `{}`".format(c.user, c.cmd)
+ for c in command_objects]
+ summary_data.orderbyfreq()
+ summary_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(summary_data)
+
+ if config.prefs.getboolean("cron", "list-users"):
+ for user, cmdlist in users.items():
+ user_data = Data()
+ user_data.subtitle = plural("session", len(cmdlist)) \
+ + " for " + user + (" (" + plural("unique command",
+ len(set(cmdlist))) + ")" if len(set(cmdlist)) > 1
+ else "")
+ user_data.items = ["`{0}`".format(cmd) for cmd in cmdlist]
+ user_data.orderbyfreq()
+ user_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(user_data)
+ logger.debug("Found {0} cron sessions for user {1} "
+ "({2} unique commands): {3}".format(
+ len(cmdlist), user,
+ len(set(cmdlist)), user_data.items))
- for msg in messages:
- usr_cmd = re.search('\((\S+)\) CMD (.*)', msg) # [('user', 'cmd')]
- if usr_cmd:
- if not usr_cmd.group(1) in users:
- users[usr_cmd.group(1)] = []
- users[usr_cmd.group(1)].append(usr_cmd.group(2))
-
- for usr, cmdlist in users.items():
- user_data = Data()
- user_data.subtitle = plural("cron session", len(cmdlist)) + " for " + usr
- user_data.items = ("`{0}`".format(cmd) for cmd in cmdlist)
- user_data.orderbyfreq()
- user_data.truncl(config.prefs.getint("logparse", "maxcmd"))
- section.append_data(user_data)
logger.info("Finished cron section")
-#
-# httpd.py
-#
-# Analyse Apache (httpd) server logs, including data transferred, requests,
-# clients, and errors. Note that Apache's logs can get filled up very quickly
-# with the default verbosity, leading to logparse taking a very long time to
-# analyse them. In general the default verbosity is good, but logs should be
-# cleared as soon as they are analysed (make sure 'rotate' is set to 'y').
-#
+# -*- coding: utf-8 -*-
+"""
+Analyse Apache (httpd) server logs, including data transferred, requests,
+clients, user agents, and errors. Note that Apache's logs can get filled up
+very quickly with the default verbosity, leading to logparse taking a very
+long time to analyse them. In general the default verbosity is good, but logs
+should be cleared as soon as they are analysed (make sure 'rotate' enabled in
+the logparse config).
+"""
+
+import datetime
import re
+import time
from logparse.formatting import *
from logparse.util import readlog, resolve
from logparse import config
from logparse.load_parsers import Parser
-ACCESS_REGEX = "^\s*(\S+).*\"GET (\S+) HTTP(?:\/\d\.\d)?\" (\d{3}) (\d*) \".+\" \"(.*)\""
+IPv4_ADDR_REGEX = '(?:\d{1,3}\.){3}\d{1,3}'
+IPv6_ADDR_REGEX = "([0-9A-Fa-f]{0,4}:){2,7}([0-9A-Fa-f]{0,4})"
+IP_ADDR_REGEX = "("+IPv4_ADDR_REGEX+"|"+IPv6_ADDR_REGEX+")"
+LOG_VARS = {
+ "%a": "(?P<client>{})?".format(IPv4_ADDR_REGEX), # client IP
+ "%A": "(?P<peer>{})?".format(IP_ADDR_REGEX), # local (peer) IP
+ "%B": "(?P<bytes>(\d+|-))", # bytes
+ "%b": "(?P<clfbytes>(\d+|\"-\"))", # bytes (CLF format)
+ "%{[^}]+?}C": "(?P<cookie>.*)", # contents of cookie
+ "%D": "(?P<serveus>-?\d+)", # time taken to serve request (μs)
+ "%{[^}]+?}e": "(?P<envvar>.*)", # environment variable contents
+ "%f": "(?P<file>.*)", # file name requested
+ "%h": "(?P<hostname>\S+)", # remote hostname or IP
+ "%H": "(?P<protocol>.*)", # request protocol
+ "%{Referer}i": "(?P<referer>.*)", # referrer
+ "%{User-Agent}i": "(?P<useragent>.*)", # user agent string
+ "%{[^}]+?}i": "(?P<header>.*)", # request header
+ "%k": "(?P<keepalive>\d*)", # number of keepalive requests
+ "%l": "(?P<logname>.*)", # remote logname
+ "%m": "(?P<method>.*)", # request method
+ "%{[^}]+?}n": "(?P<note>.*)", # notes
+ "%{[^}]+?}o": "(?P<replyheader>.*)", # reply header
+ "%p": "(?P<cport>\d*)", # canonical port on server
+ "%{[^}]+?}p": "(?P<port>\d*)", # optional port
+ "%P": "(?P<pid>\d*)", # process ID of child
+ "%{[^}]+?}P": "(?P<thread>.*)", # process or thread ID
+ "%q": "(?P<query>.*)", # query string
+ "%r": "(?P<requesthead>.*)", # first line of request
+ "%R": "(?P<handler>.*)", # handler generating response
+ "%s": "(?P<status>(\d+?|-))", # status code
+ "%t": "\[(?P<date>.*?)\]", # request date and time with offset
+ "%{[^}]+?}t": "(?P<fdate>\d+)", # request date and time ()custom format)
+ "%T": "(?P<serves>\d+)", # time taken to serve request (seconds)
+ "%{[^}]+?}T": "(?P<servec>\d+)", # time taken to serve request (custom format)
+ "%u": "(?P<user>.*)", # remote user if authenticated
+ "%U": "(?P<url>.*)", # URL path excluding query string
+ "%v": "(?P<servername>.*)", # server name
+ "%V": "(?P<servernamec>.*)", # server name (custom format)
+ "%X": "(?P<responsestatus>.?)", # status on response completion
+ "%I": "(?P<bytesreceived>\d+)", # bytes received
+ "%O": "(?P<bytessent>\d+)", # bytes sent
+ "%S": "(?P<bytestransferred>\d+)?" # total bytes transferred
+}
+LOG_ESCAPES = {
+ ">": "", # final value
+ "<": "", # initial value
+ "%%": "%" # percent escape
+}
+
+def convert_logformat(format_template):
+ """
+ Convert an Apache LogFormat string to a regex pattern
+ """
+ escape_pattern = re.compile('|'.join(LOG_ESCAPES.keys()))
+ format_template = escape_pattern.sub(lambda x: LOG_ESCAPES[x.group()], format_template)
+ var_pattern = re.compile('|'.join(LOG_VARS.keys()))
+ format_template = var_pattern.sub(lambda x: LOG_VARS[x.group()], format_template)
+ return re.compile(format_template)
+
class AccessLine(object):
+ """
+ Retrieves information from a line of the httpd access log
+ """
+
+ def __init__(self, record, datefmt, pattern):
+ """
+ Assign attributes and verify/cast those than require it. Note that the
+ `pattern` argument must be a pre-compiled regex object (to save time).
+ """
+
+ # Parse from a raw logfile string
+ self.properties = pattern.search(record).groupdict()
+ for field, value in self.properties.items():
+ if value and not (value == "-" or value == "\"-\""):
+ setattr(self, field, value)
+ else:
+ setattr(self, field, None)
+
+ # Verify data transfer metrics
+ for field, value in [x for x in self.properties.items() if "bytes" in x[0]]:
+ if isinstance(value, str) and value.isdigit():
+ setattr(self, field, int(value))
+ else:
+ setattr(self, field, 0)
+
+ # Verify date
+ self.date = datetime.datetime.strptime(self.properties["date"], datefmt)
+
+ # Verify client
+ if (not hasattr(self, "client") or not self.client) \
+ and hasattr(self, "hostname") and self.hostname:
+ self.client = self.hostname
+
+
+ # Verify file
+ if (not hasattr(self, "file") or not self.file) and hasattr(self, "requesthead"):
+ try:
+ self.file = re.search(r"^\w+\s(.*)\s\S+$", self.requesthead).group(1)
+ except:
+ self.file = ""
+
+ def match_client(self, pattern):
+ """
+ Check if the client of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ if hasattr(self, "client") and self.client:
+ return re.fullmatch(pattern, self.client)
+ elif hasattr(self, "hostname") and self.hostname:
+ return re.fullmatch(pattern, self.hostname)
+ else:
+ return True
+
+ def match_file(self, pattern):
+ """
+ Check if the target of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ if hasattr(self, "file") and self.file:
+ return re.fullmatch(pattern, self.file)
+ else:
+ return True
- def __init__(self, line):
- self.line = line
- fields = re.search(ACCESS_REGEX, line)
+ def match_ref(self, pattern):
+ """
+ Check if the referrer of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ if hasattr(self, "referer") and self.referer:
+ return re.fullmatch(pattern, self.referer)
+ else:
+ return True
- self.client = fields.group(1)
- self.file = fields.group(2)
- self.statuscode = int(fields.group(3))
- self.bytes = int(fields.group(4))
- self.useragent = fields.group(5)
class Httpd(Parser):
def __init__(self):
super().__init__()
self.name = "httpd"
- self.info = "Analyse Apache (httpd) server logs, including data transferred, requests, clients, and errors."
+ self.info = "Analyse Apache (httpd) server logs, including data " \
+ "transferred, requests, clients, and errors."
def parse_log(self):
logger.debug("Starting httpd section")
section = Section("httpd")
+ datefmt = config.prefs.get("httpd", "datetime-format")
+ if not datefmt:
+ datefmt = config.prefs.get("logparse", "datetime-format")
+ if not datefmt:
+ logger.error("Invalid datetime-format configuration parameter")
+ return None
+
+ # Initialise patterns
+ logger.debug("Converting pattern from {0}".format(
+ config.prefs.get("httpd", "access-format")))
+ pattern = convert_logformat(config.prefs.get("httpd", "access-format"))
+ logger.debug("Compiled log format {0}".format(pattern))
+
+ logger.debug("Retrieving log data")
+
accesslog = readlog(config.prefs.get("logs", "httpd-access"))
errorlog= readlog(config.prefs.get("logs", "httpd-error"))
total_errors = len(errorlog.splitlines())
- logger.debug("Retrieved log data")
-
- logger.debug("Searching through access log")
+ logger.debug("Parsing access logs")
accesses = []
for line in accesslog.splitlines():
- if "GET" in line:
- accesses.append(AccessLine(line))
+ if not "GET" in line:
+ continue
+ try:
+ ac_obj = AccessLine(line, datefmt, pattern)
+ except Exception as e:
+ logger.warning("Malformed access log: {0}. "
+ "{1}: {2}".format(line, type(e).__name__, e))
+ else:
+ if not section.period.compare(ac_obj.date):
+ continue
+
+ checks = [
+ ac_obj.match_client(
+ config.prefs.get("httpd", "clients")),
+ ac_obj.match_file(
+ config.prefs.get("httpd", "files")),
+ ac_obj.match_ref(
+ config.prefs.get("httpd", "referrers"))
+ ]
+ if not all(checks):
+ logger.debug("Ignoring access log due to config: " + line)
+ continue
+ accesses.append(ac_obj)
+
+ logger.debug("Processed {0} access logs".format(len(accesses)))
total_requests = len(accesses)
- section.append_data(Data("Total of " + plural("request", total_requests)))
+ section.append_data(Data("Total of "
+ + plural("request", total_requests)))
section.append_data(Data(plural("error", total_errors)))
+ logger.debug("Parsing total size")
+
size = Data()
- size.subtitle = "Transferred " + parsesize(sum([ac.bytes for ac in accesses]))
+ size.subtitle = "Transferred " \
+ + parsesize(sum([ac.bytessent for ac in accesses]))
section.append_data(size)
- clients = Data()
- clients.items = [resolve(ac.client, config.prefs.get("httpd", "httpd-resolve-domains")) for ac in accesses]
- clients.orderbyfreq()
- clients.subtitle = "Received requests from " + plural("client", len(clients.items))
- clients.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(clients)
+ logger.debug("Parsing clients")
+
+# clients = Data()
+# clients.items = [resolve(ac.hostname,
+# config.prefs.get("httpd", "httpd-resolve-domains"))
+# for ac in accesses]
+# clients.orderbyfreq()
+# clients.subtitle = "Received requests from " \
+# + plural("client", len(clients.items))
+# clients.truncl(config.prefs.getint("logparse", "maxlist"))
+# section.append_data(clients)
+
+ logger.debug("Parsing files")
files = Data()
- files.items = [ac.file for ac in accesses]
+ files.items = [ac.file for ac in accesses if hasattr(ac, "file")]
files.orderbyfreq()
files.subtitle = plural("file", len(files.items)) + " requested"
files.truncl(config.prefs.getint("logparse", "maxlist"))
section.append_data(files)
+ logger.debug("Parsing user agents")
+
useragents = Data()
useragents.items = [ac.useragent for ac in accesses]
useragents.orderbyfreq()
useragents.truncl(config.prefs.getint("logparse", "maxlist"))
section.append_data(useragents)
- logger.info("httpd has received " + str(total_requests) + " requests with " + str(total_errors) + " errors")
-
+ logger.info("httpd has received " + str(total_requests)
+ + " requests with " + str(total_errors) + " errors")
logger.info("Finished httpd section")
return section
-#
-# mem.py
-#
-# Get instantaneous memory statistics (installed, total, free, available)
-#
+"""
+Get instantaneous memory statistics (installed, total, free, available)
+"""
import re
def __init__(self):
super().__init__()
self.name = "mem"
- self.info = "Get instantaneous memory statistics (installed, total, free, available)"
+ self.info = "Get instantaneous memory statistics "
+ "(installed, total, free, available)"
def parse_log(self):
matches = line_regex.findall(line)
if len(matches) > 0:
- logger.debug("Detected {0} memory of {1} kB".format(matches[0][0].lower(), matches[0][1]))
- table.add_row(Row([Column(matches[0][0]), Column(parsesize(float(matches[0][1])*1000))]))
+ logger.debug("Detected {0} memory of {1} kB".
+ format(matches[0][0].lower(), matches[0][1]))
+ table.add_row(Row([Column(matches[0][0]),
+ Column(parsesize(float(matches[0][1])*1000))]))
table.align_column(0, "right")
section.append_table(table)
-#
-# postfix.py
-#
-# Get message statistics from postfix/sendmail logs
-#
+"""
+Get message statistics from postfix/sendmail logs
+"""
import re
section = Section("postfix")
logger.debug("Starting postfix section")
logger.debug("Searching through postfix logs")
- messages = re.findall('.*from\=<(.*)>, size\=(\d*),.*\n.*to=<(.*)>', readlog(config.prefs.get("logs", "postfix")))
+ messages = re.findall('.*from\=<(.*)>, size\=(\d*),.*\n.*to=<(.*)>',
+ readlog(config.prefs.get("logs", "postfix")))
r = []
s = []
size = 0
-#
-# smbd.py
-#
-# Get login statistics for a samba server.
-#
-# NOTE: This file is now deprecated in favour of the newer journald mechanism
-# used in smbd-journald.py. This parser is still functional but is slower and
-# has less features. Please switch over if possible.
-#
+"""
+Get login statistics for a samba server.
+NOTE: This file is now deprecated in favour of the newer journald mechanism
+used in smbd-journald.py. This parser is still functional but is slower and
+has less features. Please switch over if possible.
+"""
import re
import glob
self.successor = "smbd_journald"
def parse_log(self):
- logger.debug("Starting smbd section")
- section = Section("smbd")
- files = glob.glob(config.prefs.get("logs", "smbd") + "/log.*[!\.gz][!\.old]") # find list of logfiles
- # for f in files:
-
- # file_mod_time = os.stat(f).st_mtime
- # Time in seconds since epoch for time, in which logfile can be unmodified.
- # should_time = time.time() - (30 * 60)
+ # Find list of logfiles
- # Time in minutes since last modification of file
- # last_time = (time.time() - file_mod_time)
- # logger.debug(last_time)
-
- # if (file_mod_time - should_time) < args.time:
- # print "CRITICAL: {} last modified {:.2f} minutes. Threshold set to 30 minutes".format(last_time, file, last_time)
- # else:
+ logger.debug("Starting smbd section")
+ section = Section("smbd")
+ files = glob.glob(config.prefs.get("logs", "smbd")
+ + "/log.*[!\.gz][!\.old]")
- # if (datetime.timedelta(datetime.datetime.now() - datetime.fromtimestamp(os.path.getmtime(f))).days > 7):
- # files.remove(f)
logger.debug("Found log files " + str(files))
+
n_auths = 0 # total number of logins from all users
sigma_auths = [] # contains users
logger.debug("Looking at file " + file)
- # find the machine (ip or hostname) that this file represents
- ip = re.search('log\.(.*)', file).group(1) # get ip or hostname from file path (/var/log/samba/log.host)
+ # Find the machine (IP or hostname) that this file represents
+
+ # Get IP or hostname from file path (/var/log/samba/log.host)
+ ip = re.search('log\.(.*)', file).group(1)
+
+ # If IP has disappeared, fall back to a hostname from logfile
host = resolve(ip, fqdn=config.prefs.get("smbd", "smbd-resolve-domains"))
- if host == ip and (config.prefs.get("smbd", "smbd-resolve-domains") != "ip" or config.prefs.get("logparse", "resolve-domains") != "ip"): # if ip has disappeared, fall back to a hostname from logfile
+ if (host == ip and (
+ config.prefs.get("smbd", "smbd-resolve-domains") != "ip"
+ or config.prefs.get("logparse", "resolve-domains") != "ip"):
newhost = re.findall('.*\]\@\[(.*)\]', readlog(file))
if (len(set(newhost)) == 1): # all hosts in one file should be the same
host = newhost[0].lower()
- # count number of logins from each user-host pair
- matches = re.findall('.*(?:authentication for user \[|connect to service .* initially as user )(\S*)(?:\] .*succeeded| \()', readlog(file))
+ # Count number of logins from each user-host pair
+ matches = re.findall('.*(?:authentication for user \[|connect "
+ "to service .* initially as user )(\S*)(?:\] .*succeeded| \()',
+ readlog(file))
+
for match in matches:
userhost = match + "@" + host
sigma_auths.append(userhost)
- # exists = [i for i, item in enumerate(sigma_auths) if re.search(userhost, item[0])]
- # if (exists == []):
- # sigma_auths.append([userhost, 1])
- # else:
- # sigma_auths[exists[0]][1] += 1
n_auths += 1
+
auth_data = Data(subtitle=plural("login", n_auths) + " from")
- if (len(sigma_auths) == 1): # if only one user, do not display no of logins for this user
+ if (len(sigma_auths) == 1):
+ # If only one user, do not display no of logins for this user
auth_data.subtitle += ' ' + sigma_auths[0][0]
section.append_data(auth_data)
else: # multiple users
class SmbdJournald(Parser):
def __init__(self):
+
super().__init__()
self.name = "smbd_journald"
self.info = "Get login statistics for a samba server."
def parse_log(self):
+
logger.debug("Starting smbd section")
section = Section("smbd")
messages = [entry["MESSAGE"] for entry in j if "MESSAGE" in entry]
- total_auths = 0 # total number of logins for all users and all shares
- shares = {} # file shares (each share is mapped to a list of user-hostname pairs)
+ total_auths = 0 # total no. of logins for all users and all shares
+
+ shares = {} # file shares (each share is mapped to a list of
+ # user-hostname pairs)
logger.debug("Found {0} samba logins".format(str(len(messages))))
+ logger.debug("Parsing data")
for msg in messages: # one log file for each client
if "connect to service" in msg:
- entry = re.search('(\w*)\s*\(ipv.:(.+):.+\) connect to service (\S+) initially as user (\S+)', msg) # [('client', 'ip', 'share', 'user')]
+
+ # Generate list of [('client', 'ip', 'share', 'user')]
+ entry = re.search("(\w*)\s*\(ipv.:(.+):.+\) connect to service"
+ "(\S+) initially as user (\S+)", msg)
+
try:
client, ip, share, user = entry.group(1,2,3,4)
except:
if (not client.strip()):
client = ip
- userhost = user + '@' + resolve(client, fqdn=config.prefs.get("smbd", "smbd-resolve-domains"))
+ userhost = user + '@' + resolve(client,
+ fqdn=config.prefs.get("smbd", "smbd-resolve-domains"))
user_match = False
for pattern in config.prefs.get("smbd", "users").split():
user_match = re.fullmatch(pattern, userhost) or user_match
if not user_match:
- logger.debug("Ignoring login to {0} by user {1} due to config".format(share, userhost))
+ logger.debug("Ignoring login to {0} by user {1} "
+ "due to config".format(share, userhost))
continue
total_auths += 1
else:
shares[share] = [userhost]
- section.append_data(Data(subtitle="Total of {0} authentications".format(str(total_auths))))
+ # Format Data() objects
+
+ section.append_data(Data(subtitle="Total of {0} authentications"
+ .format(str(total_auths))))
for share, logins in shares.items():
share_data = Data()
share_data.items = logins
share_data.orderbyfreq()
share_data.truncl(config.prefs.getint("logparse", "maxlist"))
- share_data.subtitle = share + " ({0}, {1})".format(plural("user", len(share_data.items)), plural("login", len(logins)))
+ share_data.subtitle = share + " ({0}, {1})".format(
+ plural("user", len(share_data.items)),
+ plural("login", len(logins)))
section.append_data(share_data)
- logger.debug("Found {0} logins for share {1}".format(str(len(logins)), share))
+ logger.debug("Found {0} logins for share {1}".format(
+ str(len(logins)), share))
logger.info("Finished smbd section")
return section
-#
-# sshd.py
-#
-# Find number of ssh logins and authorised users (uses /var/log/auth.log)
-#
-# NOTE: This file is now deprecated in favour of the newer journald mechanism
-# used in sshd-journald.py. This parser is still functional but is slower and
-# has less features. Please switch over if possible.
-#
+"""
+Find number of ssh logins and authorised users (uses /var/log/auth.log)
+NOTE: This file is now deprecated in favour of the newer journald mechanism
+used in sshd-journald.py. This parser is still functional but is slower and
+has less features. Please switch over if possible.
+"""
import re
def __init__(self):
super().__init__()
self.name = "sshd"
- self.info = "Find number of ssh logins and authorised users (uses /var/log/auth.log)"
+ self.info = "Find number of ssh logins and authorised users "
+ "(uses /var/log/auth.log)"
self.deprecated = True
self.successor = "sshd_journald"
def parse_log(self):
- logger.warning("NOTE: This sshd parser is now deprecated. Please use sshd-journald if possible.")
-
+ logger.warning("NOTE: This sshd parser is now deprecated. "
+ "Please use sshd-journald if possible.")
logger.debug("Starting sshd section")
section = Section("ssh")
- logger.debug("Searching for matches in {0}".format(config.prefs.get("logs", "auth")))
- matches = re.findall('.*sshd.*Accepted publickey for .* from .*', readlog(config.prefs.get("logs", "auth"))) # get all logins
- logger.debug("Finished searching for logins")
- logger.debug("Searching for matches in {0}".format(config.prefs.get("logs", "auth")))
+ logger.debug("Searching for matches in {0}".format(
+ config.prefs.get("logs", "auth")))
authlog = readlog(config.prefs.get("logs", "auth"))
-
- matches = re.findall('.*sshd.*Accepted publickey for .* from .*', authlog) # get all logins
+ matches = re.findall('.*sshd.*Accepted publickey for .* from .*',
+ authlog) # get all logins
invalid_matches = re.findall(".*sshd.*Invalid user .* from .*", authlog)
- root_matches = re.findall("Disconnected from authenticating user root", authlog)
+ root_matches = re.findall("Disconnected from authenticating user root",
+ authlog)
logger.debug("Finished searching for logins")
- users = [] # list of users with format [username, number of logins] for each item
+ users = [] # list of users with format [username, number of logins]
+ # for each item
data = []
num = len(matches) # total number of logins
+
for match in matches:
- entry = re.search('^.*publickey\sfor\s(\w*)\sfrom\s(\S*)', match) # [('user', 'ip')]
+
+ # [('user', 'ip')]
+ entry = re.search('^.*publickey\sfor\s(\w*)\sfrom\s(\S*)', match)
user = entry.group(1)
ip = entry.group(2)
- userhost = user + '@' + resolve(ip, fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
+ userhost = user + '@' + resolve(ip,
+ fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
users.append(userhost)
+
logger.debug("Parsed list of authorised users")
+ # Format authorised users
auth_data = Data(subtitle=plural('login', num) + ' from', items=users)
-
- if (len(auth_data.items) == 1): # if only one user, do not display no of logins for this user
- logger.debug("found " + str(len(matches)) + " ssh logins for user " + users[0])
- auth_data.subtitle += ' ' + auth_data.items[0]
+ if (len(auth_data.items) == 1):
+ # If only one user, do not display no of logins for this user
+ logger.debug("Found {0} logins for user {1}".format(
auth_data.orderbyfreq()
auth_data.truncl(config.prefs.getint("logparse", "maxlist"))
- logger.debug("Found " + str(len(matches)) + " ssh logins for users " + str(data))
+ logger.debug("Found {0} logins for {1} users".format(
+ len(matches), len(users)))
section.append_data(auth_data)
+ # Format invalid users
invalid_users = []
for match in invalid_matches:
- entry = re.search('^.*Invalid user (\S+) from (\S+).*', match) # [('user', 'ip')]
-
+ # [('user', 'ip')]
+ entry = re.search('^.*Invalid user (\S+) from (\S+).*', match)
try:
user = entry.group(1)
ip = entry.group(2)
userhost = user + '@' + ip
invalid_users.append(userhost)
+
logger.debug("Parsed list of invalid users")
- invalid_data = Data(subtitle=plural("attempted login", len(invalid_matches)) + " from " + plural("invalid user", len(invalid_users), print_quantity=False), items=invalid_users)
- if (len(invalid_data.items) == 1): # if only one user, do not display no of logins for this user
- logger.debug("Found " + str(len(invalid_matches)) + " SSH login attempts for invalid user " + invalid_users[0])
+
+ invalid_data = Data(subtitle=plural("attempted login",
+ len(invalid_matches)) + " from "
+ + plural("invalid user", len(invalid_users), print_quantity=False),
+ items=invalid_users)
+ if (len(invalid_data.items) == 1):
+ # If only one user, do not display no of logins for this user
+ logger.debug("Found {0} login attempts for user {1}"
+ .format(len(invalid_matches), invalid_data.items[0]))
invalid_data.subtitle += ' ' + invalid_data.items[0]
invalid_data.orderbyfreq()
invalid_data.truncl(config.prefs.get("logparse", "maxlist"))
- logger.debug("Found " + str(len(invalid_matches)) + " SSH login attempts for invalid users " + str(data))
+ logger.debug("Found {0} login attempts for invalid users"
+ .format(len(invalid_matches)))
section.append_data(invalid_data)
- logger.debug("Found {0} attempted logins for root".format(str(len(root_matches))))
+ logger.debug("Found {0} attempted logins for root".
+ format(str(len(root_matches))))
- section.append_data(Data(subtitle=plural("attempted login", str(len(root_matches))) + " for root"))
+ section.append_data(Data(subtitle=plural("attempted login",
+ str(len(root_matches))) + " for root"))
logger.info("Finished sshd section")
return section
-#
-# sshd_journald.py
-#
-# Find number of ssh logins and authorised users (uses journald)
-#
+"""
+Find number of ssh logins and authorised users (uses journald)
+"""
import re
from systemd import journal
def __init__(self):
super().__init__()
self.name = "sshd_journald"
- self.info = "Find number of ssh logins and authorised users (uses journald)"
+ self.info = "Find number of ssh logins and authorised users "
+ "(uses journald)"
def parse_log(self):
for msg in messages:
if "Accepted publickey" in msg:
- entry = re.search('^.*publickey\sfor\s(\w*)\sfrom\s(\S*)', msg) # [('user', 'ip')]
+ # [('user', 'ip')]
+ entry = re.search('^.*publickey\sfor\s(\w*)\sfrom\s(\S*)', msg)
user = entry.group(1)
ip = entry.group(2)
- userhost = user + '@' + resolve(ip, fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
+ userhost = user + '@' + resolve(ip,
+ fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
login_data.items.append(userhost)
elif "Connection closed by authenticating user root" in msg:
- entry = re.search('^.*Connection closed by authenticating user (\S+) (\S+)', msg) # [('user', 'ip')]
+ entry = re.search('^.*Connection closed by authenticating user"
+ " (\S+) (\S+)', msg) # [('user', 'ip')]
user = entry.group(1)
ip = entry.group(2)
- userhost = user + '@' + resolve(ip, fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
+ userhost = user + '@' + resolve(ip,
+ fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
failed_data.items.append(userhost)
elif "Invalid user" in msg:
- entry = re.search('^.*Invalid user (\S+) from (\S+).*', msg) # [('user', 'ip')]
+ # [('user', 'ip')]
+ entry = re.search('^.*Invalid user (\S+) from (\S+).*', msg)
user = entry.group(1)
ip = entry.group(2)
- userhost = user + '@' + resolve(ip, fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
+ userhost = user + '@' + resolve(ip,
+ fqdn=config.prefs.get("sshd", "sshd-resolve-domains"))
invalid_data.items.append(userhost)
- login_data.subtitle = plural("successful login", len(login_data.items)) + " from"
+ login_data.subtitle = plural("successful login",
+ len(login_data.items)) + " from"
login_data.orderbyfreq()
login_data.truncl(config.prefs.getint("logparse", "maxlist"))
invalid_data.subtitle = plural("attempted login", len(invalid_data.items))
invalid_data.orderbyfreq()
- invalid_data.subtitle += plural(" from invalid user", len(invalid_data.items), False)
+ invalid_data.subtitle += plural(" from invalid user",
+ len(invalid_data.items), False)
invalid_data.truncl(config.prefs.getint("logparse", "maxlist"))
- failed_data.subtitle = plural("failed login", len(failed_data.items)) + " from"
+ failed_data.subtitle = plural("failed login",
+ len(failed_data.items)) + " from"
failed_data.orderbyfreq()
failed_data.truncl(config.prefs.getint("logparse", "maxlist"))
-#
-# sudo.py
-#
-# Get number of sudo sessions for each user
-#
-# NOTE: This file is now deprecated in favour of the newer journald mechanism
-# used in sudo-journald.py. This parser is still functional but is slower and
-# has less features. Please switch over if possible.
-#
+"""
+Get number of sudo sessions for each user
+NOTE: This parser supports reading from both journald and plain syslog files.
+By default the plain logfiles will be used, but the journald option is
+preferred for newer systems which support it. To use the journald mode,
+specify the parser as `sudo_journald` instead of `sudo`.
+"""
+
+import datetime
import re
+from subprocess import Popen, PIPE
from logparse.formatting import *
-from logparse.util import readlog
from logparse.config import prefs
from logparse.load_parsers import Parser
+
+class SudoCommand():
+ """
+ Class representing a single sudo log entry. Used for both sudo and
+ sudo_journald, so it accepts either a dictionary output by systemd.Journal
+ or a line from a logfile upon initialisation.
+ """
+
+ def __init__(self, record, datefmt):
+ """
+ Get instance variables from log message or record object
+ """
+ if isinstance(record, str):
+ if not datefmt:
+ logger.error("Date format not provided - cannot parse this "
+ "log message")
+ # Parse from a raw logfile string
+ self.date, self.init_user, self.pwd, self.su, self.command \
+ = re.search(r"^(?P<time>.+)\s\w+\ssudo:\s+"
+ "(?P<init_user>\w*) : TTY=.* ; PWD=(?P<pwd>\S*) ;"
+ " USER=(?P<su>\w*) ; COMMAND=(?P<command>\S*)", record)\
+ .groupdict().values()
+ self.date = datetime.datetime.strptime(self.date, datefmt)
+ if not "Y" in datefmt:
+ self.date = self.date.replace(year=datetime.datetime.now().year)
+ elif isinstance(record, dict):
+ self.date = record["_SOURCE_REALTIME_TIMESTAMP"]
+ self.init_user, self.pwd, self.su, self.command = re.search(
+ r"\s+(?P<init_user>\S+) : TTY=.* ; PWD=(?P<pwd>\S*) ;"
+ " USER=(?P<su>\w*) ; COMMAND=(?P<command>\S*)",
+ record["MESSAGE"]).groupdict().values()
+ self.command = " ".join(self.command.split())
+ else:
+ raise TypeError("record should be str or dict")
+
+ def truncate(self):
+ """
+ Hide the full directory path for any scripts or explicit binary
+ references in the command. e.g. `/usr/bin/cat` → `cat`
+ """
+ self.command = re.sub(r"(\s|^)/\S*/(\S+)", r"\1\2", self.command)
+ return self.command
+
+ def match_su(self, pattern):
+ """
+ Check if the user of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ return re.fullmatch(pattern, self.su)
+
+ def match_init_user(self, pattern):
+ """
+ Check if the initialising user of this object matches against a regex
+ string and return a boolean result of this comparison.
+ """
+ return re.fullmatch(pattern, self.init_user)
+
+ def match_cmd(self, pattern):
+ """
+ Check if the command of this object matches against a regex string and
+ return a boolean result of this comparison.
+ """
+ return re.fullmatch(pattern, self.command)
+
+ def match_pwd(self, pattern):
+ """
+ Check if the directory of this object matches against a regex string
+ and return a boolean result of this comparison.
+ """
+ return re.fullmatch(pattern, self.pwd)
+
+
class Sudo(Parser):
def __init__(self):
super().__init__()
self.name = "sudo"
self.info = "Get number of sudo sessions for each user"
- self.deprecated = True
- self.successor = "sudo_journald"
+ self.journald = False
+
+ def _get_journald(self, startdate):
+ from systemd import journal
+ j = journal.Reader()
+ j.this_machine()
+ j.log_level(journal.LOG_INFO)
+ j.add_match(_COMM="sudo")
+ j.seek_realtime(startdate)
+ j.log_level(5)
+ return [entry for entry in j if "MESSAGE" in entry
+ and "COMMAND=" in entry["MESSAGE"]]
+
+ def _get_logfile(self, path):
+ from logparse.util import readlog
+ return re.findall(r".+sudo.+TTY=.+PWD=.+USER=.+COMMAND=.+",
+ readlog(path)) # Initial check to make sure all fields exist
+
def parse_log(self):
+
logger.debug("Starting sudo section")
section = Section("sudo")
- logger.debug("Searching for matches in {0}".format(prefs.get("logs", "auth")))
- umatches = re.findall('.*sudo:session\): session opened.*', readlog(prefs.get("logs", "auth")))
- num = sum(1 for line in umatches) # total number of sessions
- users = []
- data = []
- for match in umatches:
- user = re.search('.*session opened for user root by (\S*)\(uid=.*\)', match).group(1)
- exists = [i for i, item in enumerate(users) if re.search(user, item[0])]
- if (exists == []):
- users.append([user, 1])
- else:
- users[exists[0]][1] += 1
- commands = []
- cmatches = re.findall('sudo:.*COMMAND\=(.*)', readlog(prefs.get("logs", "auth")))
- for cmd in cmatches:
- commands.append(cmd)
- logger.debug("Finished parsing sudo sessions")
-
- auth_data = Data(subtitle=plural("sudo session", num) + " for")
-
- if (len(users) == 1):
- logger.debug("found " + str(num) + " sudo session(s) for user " + str(users[0]))
- auth_data.subtitle += ' ' + users[0][0]
+
+ datefmt = config.prefs.get("sudo", "datetime-format")
+ if not datefmt:
+ datefmt = config.prefs.get("logparse", "datetime-format")
+ if not datefmt:
+ logger.error("Invalid datetime-format configuration parameter")
+ return None
+
+ if not (config.prefs.getboolean("sudo", "summary")
+ or config.prefs.getboolean("sudo", "list-users")):
+ logger.warning("Both summary and list-users configuration options "
+ "are set to false, so no output will be generated. "
+ "Skipping this parser.")
+ return None
+
+
+ if self.journald:
+ logger.debug("Searching for sudo commands in journald")
+ messages = self._get_journald(section.period.startdate)
else:
- for user in users:
- auth_data.items.append(user[0] + ' (' + str(user[1]) + ')')
- logger.debug("found " + str(num) + " sudo sessions for users " + str(data))
- section.append_data(auth_data)
-
- if (len(commands) > 0):
- command_data = Data(subtitle="top sudo commands")
- commands = backticks(commands)
- command_data.items = commands
- command_data.orderbyfreq()
- command_data.truncl(prefs.getint("logparse", "maxcmd"))
- section.append_data(command_data)
+ logger.debug("Searching for matches in {0}".format(
+ prefs.get("logs", "auth")))
+ messages = self._get_logfile(config.prefs.get("logs", "auth"))
+
+
+ commands_objects = [] # list of command objects
+ init_users = {} # keys are users, values are lists of commands
+
+ logger.debug("Parsing sudo log messages")
+
+ for msg in messages:
+
+ try:
+ cmd_obj = SudoCommand(msg, datefmt)
+ except Exception as e:
+ logger.warning("Malformed sudo log message: {0}. "
+ "Error message: {1}".format(msg, str(e)))
+ continue
+ else:
+ if cmd_obj.date < section.period.startdate:
+ continue
+ checks = [
+ cmd_obj.match_init_user(
+ config.prefs.get("sudo", "init-users")),
+ cmd_obj.match_su(
+ config.prefs.get("sudo", "superusers")),
+ cmd_obj.match_cmd(
+ config.prefs.get("sudo", "commands")),
+ cmd_obj.match_pwd(
+ config.prefs.get("sudo", "directories")),
+ ]
+ if not all(checks):
+ logger.debug("Ignoring sudo session by {0} with command "
+ "{1} due to config".format(
+ cmd_obj.init_user, cmd_obj.command))
+ continue
+ if config.prefs.getboolean("sudo", "truncate-commands"):
+ cmd_obj.truncate()
+ commands_objects.append(cmd_obj)
+ if not cmd_obj.init_user in init_users:
+ init_users[cmd_obj.init_user] = []
+ init_users[cmd_obj.init_user].append(
+ cmd_obj.su + ": " + cmd_obj.command)
+
+ logger.debug("Generating output")
+
+ if len(commands_objects) == 0:
+ logger.warning("No sudo commands found")
+ return
+
+ if config.prefs.getboolean("sudo", "summary"):
+
+ summary_data = Data()
+ summary_data.subtitle = plural(
+ "sudo session", len(commands_objects))
+
+ if all(cmd.su == commands_objects[0].su
+ for cmd in commands_objects):
+ # Only one superuser
+ if len(set(cmd.init_user for cmd in commands_objects)) > 1:
+ # Multiple initiating users
+ summary_data.subtitle += " for superuser " \
+ + commands_objects[0].su
+ summary_data.items = ["{}: {}".format(
+ cmd.init_user, cmd.command)
+ for cmd in commands_objects]
+ else:
+ # Only one initiating user
+ summary_data.subtitle += " opened by " \
+ + commands_objects[0].init_user + " for " \
+ + commands_objects[0].su
+ summary_data.items = [cmd.command
+ for cmd in commands_objects]
+ else:
+ # Multiple superusers
+ if len(set(cmd.init_user for cmd in commands_objects)) > 1:
+ # Multiple initiating users
+ summary_data.subtitle += " for " + plural("superuser",
+ len(set(cmd.su for cmd in commands_objects)))
+ summary_data.items = ["{}→{}: {}".format(
+ cmd.init_user, cmd.su, cmd.command)
+ for cmd in commands_objects]
+ else:
+ # Only one initiating user
+ summary_data.subtitle += " by " \
+ + commands_objects[0].init_user \
+ + " for " + plural("superuser",
+ len(set(cmd.su
+ for cmd in commands_objects)))
+ summary_data.items = ["{}: {}".format(
+ cmd.su, cmd.command) for cmd in commands_objects]
+ summary_data.orderbyfreq()
+ summary_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(summary_data)
+
+ if config.prefs.getboolean("sudo", "list-users") \
+ and len(set(cmd.init_user for cmd in commands_objects)) > 1:
+ for user, user_commands in init_users.items():
+ user_data = Data()
+ user_data.subtitle = plural("sudo session",
+ len(user_commands)) + " for user " + user
+ user_data.items = user_commands
+ user_data.orderbyfreq()
+ user_data.truncl(config.prefs.getint("logparse", "maxcmd"))
+ section.append_data(user_data)
logger.info("Finished sudo section")
return section
+
+ def check_dependencies(self):
+
+ # Check if sudo exists
+ sudo_cmdline = "sudo --version"
+ if self._check_dependency_command(sudo_cmdline)[0] != 0:
+ return (False, ["sudo"])
+ else:
+ return (True, None)
+++ /dev/null
-#
-# sudo-journald.py
-#
-# Get number of sudo sessions for each user
-#
-
-import re
-from systemd import journal
-
-from logparse.formatting import *
-from logparse.util import readlog
-from logparse.config import prefs
-from logparse.load_parsers import Parser
-
-class SudoCommand():
-
- def __init__(self, msg):
- self.init_user, self.pwd, self.su, self.command = re.search(
- r"\s*(?P<init_user>\w*) : TTY=.* ; PWD=(?P<pwd>\S*) ;"
- " USER=(?P<su>\w*) ; COMMAND=(?P<command>\S*)", msg)\
- .groupdict().values()
-
-class Sudo(Parser):
-
- def __init__(self):
- super().__init__()
- self.name = "sudo"
- self.info = "Get number of sudo sessions for each user"
-
- def parse_log(self):
-
- logger.debug("Starting sudo section")
- section = Section("sudo")
-
- if not (config.prefs.getboolean("sudo", "summary")
- or config.prefs.getboolean("sudo", "list-users")):
- logger.warning("Both summary and list-users configuration options "
- "are set to false, so no output will be generated. "
- "Skipping this parser.")
- return None
-
- j = journal.Reader()
- j.this_machine()
- j.log_level(journal.LOG_INFO)
- j.add_match(_COMM="sudo")
- j.seek_realtime(section.period.startdate)
- j.log_level(5)
-
- logger.debug("Searching for sudo commands")
-
- messages = [entry["MESSAGE"] for entry in j if "MESSAGE" in entry]
-
- commands_objects = [] # list of command objects
- init_users = {} # keys are users, values are lists of commands
-
- logger.debug("Parsing sudo log messages")
-
- for msg in messages:
-
- if "COMMAND=" in msg:
- try:
- command_obj = SudoCommand(msg)
- except Exception as e:
- logger.warning("Malformed sudo log message: {0}. "
- "Error message: {1}".format(msg, str(e)))
- continue
- if config.prefs.getboolean("sudo", "truncate-commands"):
- command_obj.command = command_obj.command.split("/")[-1]
- commands_objects.append(command_obj)
- if not command_obj.init_user in init_users:
- init_users[command_obj.init_user] = []
- init_users[command_obj.init_user].append(
- command_obj.su + ": " + command_obj.command)
-
- logger.debug("Generating output")
-
- if len(commands_objects) == 0:
- logger.warning("No sudo commands found")
- return
-
- if config.prefs.getboolean("sudo", "summary"):
-
- summary_data = Data()
- summary_data.subtitle = plural("sudo session", len(commands_objects))
-
- if all(cmd.su == commands_objects[0].su for cmd in commands_objects):
- # Only one superuser
- if len(set(cmd.init_user for cmd in commands_objects)) > 1:
- # Multiple initiating users
- summary_data.subtitle += " for superuser " \
- + commands_objects[0].su
- summary_data.items = ["{}: {}".format(cmd.init_user, cmd.command)
- for cmd in commands_objects]
- else:
- # Only one initiating user
- summary_data.subtitle += " opened by " \
- + commands_objects[0].init_user + " for " \
- + commands_objects[0].su
- summary_data.items = [cmd.command
- for cmd in commands_objects]
- else:
- # Multiple superusers
- if len(set(cmd.init_user for cmd in commands_objects)) > 1:
- # Multiple initiating users
- summary_data.subtitle += " for " + plural("superuser",
- len(set(cmd.su for cmd in commands_objects)))
- summary_data.items = ["{}→{}: {}".format(
- cmd.init_user, cmd.su, cmd.command)
- for cmd in commands_objects]
- else:
- # Only one initiating user
- summary_data.subtitle += " by " \
- + commands_objects[0].init_user \
- + " for " + plural("superuser",
- len(set(cmd.su
- for cmd in commands_objects)))
- summary_data.items = ["{}: {}".format(
- cmd.su, cmd.command) for cmd in commands_objects]
- summary_data.orderbyfreq()
- summary_data.truncl(config.prefs.getint("logparse", "maxcmd"))
- section.append_data(summary_data)
-
- if config.prefs.getboolean("sudo", "list-users") \
- and len(set(cmd.init_user for cmd in commands_objects)) > 1:
- for user, user_commands in init_users.items():
- user_data = Data()
- user_data.subtitle = plural("sudo session",
- len(user_commands)) + " for user " + user
- user_data.items = user_commands
- user_data.orderbyfreq()
- user_data.truncl(config.prefs.getint("logparse", "maxcmd"))
- section.append_data(user_data)
-
- logger.info("Finished sudo section")
-
- return section
-#
-# sysinfo.py
-#
-# Get standard system information from basic Unix commands
-#
+"""
+Get standard system information from basic Unix commands
+"""
import platform
import subprocess
section = Section("system")
table = Table()
- table.add_row(Row([Column("Hostname"), Column(util.hostname(prefs.get("logparse", "hostname-path")))]))
+ table.add_row(Row([Column("Hostname"),
+ Column(util.hostname(prefs.get("logparse", "hostname-path")))]))
table.add_row(Row([Column("OS"), Column(platform.platform())]))
table.add_row(Row([Column("OS version"), Column(platform.version())]))
- table.add_row(Row([Column("Platform"), Column(platform.system() + " " + platform.machine())]))
+ table.add_row(Row([Column("Platform"),
+ Column(platform.system() + " " + platform.machine())]))
processors = []
raw_proc = util.readlog(prefs.get("logs", "cpuinfo"))
for line in raw_proc.splitlines():
if "model name" in line:
processor = line_regex.sub("", line, 1)
- processor = " ".join(proc_regex.sub("", processor).split()) # remove extraneous text and whitespace
+ # Remove extraneous text and whitespace:
+ processor = " ".join(proc_regex.sub("", processor).split())
if not processor in processors:
processors.append(processor)
else:
- logger.debug("Found duplicate entry (perhaps multiple cores?) for {0}".format(processor))
+ logger.debug("Found duplicate entry (perhaps multiple "
+ "cores?) for {0}".format(processor))
table.align_column(0, "right")
if len(processors) == 1:
- table.add_row(Row([Column("Processor"), Column("; ".join(processors))]))
+ table.add_row(Row([Column("Processor"),
+ Column("; ".join(processors))]))
section.append_table(table)
elif len(processors) > 1:
section.append_table(table)
logger.debug("Found uptime data " + str(raw_uptime))
uptime_total = float(raw_uptime.split()[0])
- table.add_row(Row([Column("Uptime"), Column("%d d %d h %d m" % (uptime_total // 86400, uptime_total % 86400 // 3600, uptime_total % 3600 // 60))]))
+ table.add_row(Row([Column("Uptime"),
+ Column("%d d %d h %d m" % (
+ uptime_total // 86400,
+ uptime_total % 86400 // 3600,
+ uptime_total % 3600 // 60))]))
idle_time = float(raw_uptime.split()[1]) / cpu_count()
m, s = divmod(idle_time, 60)
h, m = divmod(m, 60)
- table.add_row(Row([Column("Idle time"), Column("%d d %d h %d m per core (avg)" % (idle_time // 86400, idle_time % 86400 // 3600, idle_time % 3600 // 60))]))
+ table.add_row(Row([Column("Idle time"),
+ Column("%d d %d h %d m per core (avg)" % (
+ idle_time // 86400,
+ idle_time % 86400 // 3600,
+ idle_time % 3600 // 60))]))
logger.info("Finished sysinfo section")
return section
+++ /dev/null
-# -*- coding: utf-8 -*-
-
-#
-# systemctl.py
-#
-# Get information about running/failed units and boot process
-#
-
-import re
-import subprocess
-
-from logparse import config
-from logparse.formatting import *
-from logparse.load_parsers import Parser
-from logparse.util import resolve
-
-# The following list changes with each systemd version.
-# Run `systemctl --state=help` to view currently implemented states.
-# The numbers correspond to degrees of severity for later formatting.
-BAD_STATES = {"bad": 4, "failed": 4, "not-found": 4, "bad-setting": 2,
- "error": 3, "masked": 2, "dead": 3, "abandoned": 3}
-SYS_STATUS = {'running': 0, 'initializing': 1, 'starting': 1, 'stopping': 1,
- 'degraded': 3, 'unknown': 4, 'offline': 5}
-
-class Unit():
-
- def __init__(self, name, loaded, active, sub, description):
- self.name = name
- self.loaded = loaded
- self.active = active
- self.sub = sub
- self.description = description
-
- def status():
- try:
- p = subprocess.Popen(["systemctl", "is-active", self.name],
- stdout=subprocess.PIPE)
- (output, err) = p.communicate()
- status = output.decode('utf-8')
- return status
- except Exception as e:
- logger.warning("Failed to get status for unit {0}: {1}".format(
- self.name, str(e)))
-
-
-class Systemctl(Parser):
-
- def __init__(self):
- super().__init__()
- self.name = "systemctl"
- self.info = "Information about running/failed units and boot process"
-
- def parse_log(self):
-
- logger.debug("Starting systemctl section")
- section = Section("systemctl")
-
- try:
- p = subprocess.Popen(["systemctl", "is-system-running"],
- stdout = subprocess.PIPE)
- (output, err) = p.communicate()
- except Exception as e:
- logger.warning("Failed to get system status: " + str(e))
- else:
- status_raw = str(output.decode('utf-8')).split()[0]
- section.append_data(Data("System status", [status_raw], severity=SYS_STATUS[status_raw]))
-
- try:
- p = subprocess.Popen(
- ["systemctl", "list-units"], stdout = subprocess.PIPE)
- (output, err) = p.communicate()
- except Exception as e:
- logger.warning("Failed to get list of unit files: " + str(e))
- units_raw = None
- else:
- units_raw = output.decode('utf-8')
- unit_properties = [Unit(*line.split(maxsplit=4))
- for line in units_raw.replace("●", " ").splitlines()[1:-7]]
- unit_states = {}
-
- for u in unit_properties:
- if not u.sub in unit_states:
- unit_states[u.sub] = []
- unit_states[u.sub].append(u.name)
-
- ok_data = Data()
-
- for state, unit_list in unit_states.items():
- if state in BAD_STATES:
- logger.debug("Found critical unit {0} with status {1}".format(
- u.name, u.sub))
- section.append_data(Data(
- plural(state + " unit", len(unit_list)), unit_list,
- severity=BAD_STATES[state])
- .truncl(config.prefs.getint("logparse", "maxlist")))
- else:
- ok_data.items.append(" ".join([str(len(unit_list)), state]))
-
- if len(ok_data.items) > 0 and config.prefs.getboolean("systemctl", "show-all"):
- ok_data.subtitle = plural("unit", len(ok_data.items)) \
- + " in a non-critical state"
- ok_data.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(ok_data)
-
- logger.info("Finished systemctl section")
- return section
-
--- /dev/null
+# -*- coding: utf-8 -*-
+
+"""
+Get information about running/failed units and boot process
+"""
+
+import re
+import subprocess
+
+from logparse import config
+from logparse.formatting import *
+from logparse.load_parsers import Parser
+from logparse.util import resolve
+
+# The following list changes with each systemd version.
+# Run `systemctl --state=help` to view currently implemented states.
+# The numbers correspond to degrees of severity for later formatting.
+BAD_STATES = {"bad": 4, "failed": 4, "not-found": 4, "bad-setting": 2,
+ "error": 3, "masked": 2, "dead": 3, "abandoned": 3}
+SYS_STATUS = {'running': 0, 'initializing': 1, 'starting': 1, 'stopping': 1,
+ 'degraded': 3, 'unknown': 4, 'offline': 5}
+
+
+class Unit():
+ """
+ Class with some basic variables to represent a systemd unit.
+ """
+
+ def __init__(self, name, loaded, active, sub, description):
+ self.name = name
+ self.loaded = loaded
+ self.active = active
+ self.sub = sub
+ self.description = description
+
+ def status():
+ """
+ Finds the status of a unit on-demand with `systemctl is-active`.
+ Currently not used anywhere.
+ """
+ try:
+ p = subprocess.Popen(["systemctl", "is-active", self.name],
+ stdout=subprocess.PIPE)
+ (output, err) = p.communicate()
+ status = output.decode('utf-8')
+ return status
+ except Exception as e:
+ logger.warning("Failed to get status for unit {0}: {1}".format(
+ self.name, str(e)))
+
+
+class Systemd(Parser):
+
+ def __init__(self):
+ super().__init__()
+ self.name = "systemd"
+ self.info = "Information about running/failed units and boot process"
+
+ def parse_log(self):
+
+ logger.debug("Starting systemd section")
+ section = Section("systemd")
+
+ # Get overall system status
+ try:
+ p = subprocess.Popen(["systemctl", "is-system-running"],
+ stdout = subprocess.PIPE)
+ (output, err) = p.communicate()
+ except Exception as e:
+ logger.warning("Failed to get system status: " + str(e))
+ else:
+ status_raw = str(output.decode('utf-8')).split()[0]
+ section.append_data(Data("System status", [status_raw],
+ severity=SYS_STATUS[status_raw]))
+
+ # Get summary of systemd units
+ try:
+ p = subprocess.Popen(
+ ["systemctl", "list-units"], stdout = subprocess.PIPE)
+ (output, err) = p.communicate()
+ except Exception as e:
+ logger.warning("Failed to get list of unit files: " + str(e))
+ units_raw = None
+ else:
+ units_raw = output.decode('utf-8')
+ unit_properties = [Unit(*line.split(maxsplit=4))
+ for line in units_raw.replace("●", " ") \
+ .splitlines()[1:-7]]
+ unit_states = {}
+
+ for u in unit_properties:
+ if not u.sub in unit_states:
+ unit_states[u.sub] = []
+ unit_states[u.sub].append(u.name)
+
+ ok_data = Data()
+
+ for state, unit_list in unit_states.items():
+ if state in BAD_STATES:
+ logger.debug("Found critical unit {0} with status {1}"
+ .format(u.name, u.sub))
+ section.append_data(Data(
+ plural(state + " unit", len(unit_list)), unit_list,
+ severity=BAD_STATES[state])
+ .truncl(config.prefs.getint("logparse", "maxlist")))
+ else:
+ ok_data.items.append(" ".join([str(len(unit_list)), state]))
+
+ if len(ok_data.items) > 0 \
+ and config.prefs.getboolean("systemd", "show-all"):
+ ok_data.subtitle = plural("unit", len(ok_data.items)) \
+ + " in a non-critical state"
+ ok_data.truncl(config.prefs.getint("logparse", "maxlist"))
+ section.append_data(ok_data)
+
+ logger.info("Finished systemd section")
+ return section
+
+ def check_dependencies(self):
+
+ # Check if systemctl is set up
+ cmd = "systemctl --version"
+ if self._check_dependency_command(cmd)[0] != 0:
+ return (False, ["systemctl"])
+ else:
+ return (True, None)
-#
-# temperature.py
-#
-# Find current temperature of various system components (CPU, motherboard,
-# hard drives, ambient). Detection of motherboard-based temperatures (CPU
-# etc) uses the pysensors library, and produces a similar output to
-# lmsensors. HDD temperatures are obtained from the hddtemp daemon
-# (http://www.guzu.net/linux/hddtemp.php) which was orphaned since 2007. For
-# hddtemp to work, it must be started in daemon mode, either manually or with
-# a unit file. Manually, it would be started like this:
-#
-# sudo hddtemp -d /dev/sda /dev/sdb ... /dev/sdX
-#
+"""
+Find current temperature of various system components (CPU, motherboard,
+hard drives, ambient). Detection of motherboard-based temperatures (CPU
+etc) uses the pysensors library, and produces a similar output to
+lmsensors. HDD temperatures are obtained from the hddtemp daemon
+<http://www.guzu.net/linux/hddtemp.php> which was orphaned since 2007. For
+hddtemp to work, it must be started in daemon mode, either manually or with
+a unit file. Manually, it would be started like this:
+ `sudo hddtemp -d /dev/sda /dev/sdb ... /dev/sdX`
+"""
import re
import sensors
class HddtempClient:
- def __init__(self, host: str='127.0.0.1', port: int=7634, timeout: int=10, sep: str='|') -> None:
+ def __init__(self, host: str='127.0.0.1', port: int=7634, timeout: int=10,
+ sep: str='|') -> None:
self.host = host
self.port = port
self.timeout = timeout
def _parse_drive(self, drive: str) -> Drive:
try:
drive_data = drive.split(self.sep)
- return Drive(drive_data[0], drive_data[1], int(drive_data[2]), drive_data[3])
+ return Drive(drive_data[0], drive_data[1],
+ int(drive_data[2]), drive_data[3])
except Exception as e:
- logger.warning("Error processing drive: {0}".format(str(drive_data)))
+ logger.warning("Error processing drive: " + str(drive_data))
return None
def _parse(self, data: str) -> List[Drive]:
if parsed_drive != None:
parsed_drives.append(parsed_drive)
-# return [self._parse_drive(drive) for drive in drives if drive != None]
-# return list(filter(lambda drive: self._parse_drive(drive), drives))
return parsed_drives
def get_drives(self) -> List[Drive]: # Obtain data from telnet server
raw_data = tn.read_all()
return self._parse(raw_data.decode('ascii')) # Return parsed data
except Exception as e:
- logger.warning("Couldn't read data from {0}:{1} - {2}".format(self.host, self.port, str(e)))
+ logger.warning("Couldn't read data from {0}:{1} - {2}".format(
+ self.host, self.port, str(e)))
return 1
def __init__(self):
super().__init__()
self.name = "temperature"
- self.info = "Find current temperature of various system components (CPU, motherboard, hard drives, ambient)."
+ self.info = "Find current temperature of various system components "
+ "(CPU, motherboard, hard drives, ambient)."
def parse_log(self):
for chip in sensors.iter_detected_chips():
for feature in chip:
if "Core" in feature.label:
- coretemp.items.append([feature.label, float(feature.get_value())])
+ coretemp.items.append([feature.label,
+ float(feature.get_value())])
continue
if "CPUTIN" in feature.label:
- pkgtemp.items.append([feature.label, float(feature.get_value())])
+ pkgtemp.items.append([feature.label,
+ float(feature.get_value())])
continue
if "SYS" in feature.label:
- systemp.items.append([feature.label, float(feature.get_value())])
+ systemp.items.append([feature.label,
+ float(feature.get_value())])
continue
logger.debug("Core data is {0}".format(str(coretemp.items)))
logger.debug("Sys data is {0}".format(str(systemp.items)))
logger.debug("Pkg data is {0}".format(str(pkgtemp.items)))
for temp_data in [systemp, coretemp, pkgtemp]:
- logger.debug("Looking at temp data {0}".format(str(temp_data.items)))
+ logger.debug("Looking at temp data {0}".format(
+ temp_data.items))
if len(temp_data.items) > 1:
- avg = float(sum(feature[1] for feature in temp_data.items)) / len(temp_data.items)
- logger.debug("Avg temp for {0} is {1} {2}{3}".format(temp_data.subtitle, str(avg), DEG, CEL))
- temp_data.subtitle += " (avg {0}{1}{2})".format(str(avg), DEG, CEL)
- temp_data.items = ["{0}: {1}{2}{3}".format(feature[0], str(feature[1]), DEG, CEL) for feature in temp_data.items]
+ avg = (float(sum(feature[1] for feature in temp_data.items))
+ / len(temp_data.items))
+ logger.debug("Avg temp for {0} is {1} {2}{3}".format(
+ temp_data.subtitle, avg, DEG, CEL))
+ temp_data.subtitle += " (avg {0}{1}{2})".format(
+ avg, DEG, CEL)
+ temp_data.items = ["{0}: {1}{2}{3}".format(
+ feature[0], feature[1], DEG, CEL)
+ for feature in temp_data.items]
else:
temp_data.items = [str(temp_data.items[0][1]) + DEG + CEL]
section.append_data(temp_data)
for drive in sorted(drives, key=lambda x: x.path):
if drive.path in config.prefs.get("temperatures", "drives").split():
sumtemp += drive.temperature
- hddtemp_data.items.append(("{0} ({1})".format(drive.path, drive.model) if config.prefs.getboolean("temperatures", "show-model") else drive.path) + ": {0}{1}{2}".format(drive.temperature, DEG, drive.units))
+ hddtemp_data.items.append(("{0} ({1})".format(
+ drive.path, drive.model)
+ if config.prefs.getboolean("temperatures", "show-model")
+ else drive.path) + ": {0}{1}{2}".format(
+ drive.temperature, DEG, drive.units))
else:
drives.remove(drive)
- logger.debug("Ignoring drive {0} ({1}) due to config".format(drive.path, drive.model))
+ logger.debug("Ignoring drive {0} ({1}) due to config".format(
+ drive.path, drive.model))
logger.debug("Sorted drive info: " + str(drives))
if not len(drives) == 0:
- hddavg = '{0:.1f}{1}{2}'.format(sumtemp/len(drives), DEG, drives[0].units) # use units of first drive
- logger.debug("Sum of temperatures: {}; Number of drives: {}; => Avg disk temp is {}".format(str(sumtemp), str(len(drives)), hddavg))
- hddtemp_data.subtitle += " (avg {0}{1}{2})".format(str(hddavg), DEG, CEL)
+ # use units of first drive
+ hddavg = '{0:.1f}{1}{2}'.format(
+ sumtemp/len(drives), DEG, drives[0].units)
+ logger.debug("Sum of temperatures: {}; Number of drives: {}; "
+ "=> Avg disk temp is {}".format(sumtemp, len(drives), hddavg))
+ hddtemp_data.subtitle += " (avg {0}{1}{2})".format(hddavg, DEG, CEL)
section.append_data(hddtemp_data)
logger.debug("Finished processing drive temperatures")
--- /dev/null
+"""
+Get details about packets blocked by ufw (uses journald)
+"""
+
+import datetime
+import re
+from systemd import journal
+
+from logparse import config
+from logparse.formatting import *
+from logparse.load_parsers import Parser
+from logparse.util import resolve
+
+PROTOCOLS = ["TCP", "UDP", "UDP-Lite", "ICMP", "ICMPv6", "AH", "SCTP", "MH"]
+
+class Packet():
+ """
+ Class to hold variables for each packet. Also parses incoming log messages
+ on object initialisation.
+ """
+
+ def __init__(self, msg):
+ """
+ Determine fields in log message. If any of the fields are missing the
+ log is considered malformed and is discarded. Also the protocol can be
+ specified as either an integer or a string - see `man ufw`.
+ """
+ try:
+ self.inif, self.outif, self.mac, self.src, self.dst, self.len, \
+ self.proto, self.spt, self.dpt = \
+ re.search(r"IN=(?P<inif>\w*).*OUT=(?P<outif>\w*).*"
+ "MAC=(?P<mac>\S*).*SRC=(?P<src>\S*).*DST=(?P<dst>\S*)"
+ ".*LEN=(?P<length>\d*).*PROTO=(?P<proto>\S*)"
+ "(?:\sSPT=(?P<spt>\d*))?(?:\sDPT=(?P<dpt>\d*))?", msg
+ ).groupdict().values()
+ if self.proto and self.proto.isdigit():
+ self.proto = PROTOCOLS[int(self.proto)-1]
+ except Exception as e:
+ logger.warning("Malformed packet log: {0}. Error message: {1}"
+ .format(msg, str(e)))
+ return None
+
+class UfwJournald(Parser):
+
+ def __init__(self):
+ super().__init__()
+ self.name = "ufw"
+ self.info = "Get details about packets blocked by ufw"
+
+ def parse_log(self):
+
+ logger.debug("Starting ufw section")
+ section = Section("ufw")
+
+ # Find applicable log entries
+
+ j = journal.Reader()
+ j.this_machine()
+ j.add_match(_TRANSPORT='kernel')
+ j.add_match(PRIORITY=4)
+ j.seek_realtime(section.period.startdate)
+
+ logger.debug("Searching for messages")
+
+ blocked_packets = [Packet(entry["MESSAGE"]) for entry in j
+ if "MESSAGE" in entry and "UFW BLOCK" in entry["MESSAGE"]]
+
+ # Parse messages
+
+ logger.debug("Parsing messages")
+
+ inbound_interfaces = []
+ outbound_interfaces = []
+ n_inbound = n_outbond = 0
+ src_ips = []
+ dst_ips = []
+ src_ports = []
+ dst_ports = []
+ protocols = {'UDP': 0, 'TCP': 0}
+ src_macs = []
+
+ for pkt in blocked_packets:
+ if pkt.inif:
+ inbound_interfaces.append(pkt.inif)
+ elif pkt.outif:
+ outbound_interfaces.append(pkt.outif)
+ if pkt.src: src_ips.append(resolve(pkt.src,
+ config.prefs.get("ufw", "ufw-resolve-domains")))
+ if pkt.dst: dst_ips.append(resolve(pkt.dst,
+ config.prefs.get("ufw", "ufw-resolve-domains")))
+ if pkt.spt: src_ports.append(pkt.spt)
+ if pkt.dpt: dst_ports.append(pkt.dpt)
+ if pkt.proto: protocols[pkt.proto] += 1
+
+ # Format data objects
+
+ section.append_data(Data(subtitle="{} blocked ({} UDP, {} TCP)".format(
+ plural("packet", len(blocked_packets)),
+ protocols['UDP'], protocols['TCP'])))
+
+ src_port_data = Data(items=src_ports)
+ src_port_data.orderbyfreq()
+ src_port_data.subtitle = plural("source port", len(src_port_data.items))
+ src_port_data.truncl(config.prefs.getint("logparse", "maxlist"))
+ section.append_data(src_port_data)
+
+ dst_port_data= Data(items=dst_ports)
+ dst_port_data.orderbyfreq()
+ dst_port_data.subtitle = plural("destination port",
+ len(dst_port_data.items))
+ dst_port_data.truncl(config.prefs.getint("logparse", "maxlist"))
+ section.append_data(dst_port_data)
+
+ src_ips_data= Data(items=src_ips)
+ src_ips_data.orderbyfreq()
+ src_ips_data.subtitle = plural("source IP", len(src_ips_data.items))
+ src_ips_data.truncl(config.prefs.getint("logparse", "maxlist"))
+ section.append_data(src_ips_data)
+
+ dst_ips_data= Data(items=dst_ips)
+ dst_ips_data.orderbyfreq()
+ dst_ips_data.subtitle = plural("destination IP",
+ len(dst_ips_data.items))
+ dst_ips_data.truncl(config.prefs.getint("logparse", "maxlist"))
+ section.append_data(dst_ips_data)
+
+ logger.info("Finished ufw section")
+ return section
+
+ def check_dependencies(self):
+ """
+ Basic dependency check to determine if there are any logs to parse
+ """
+
+ ufw_cmdline = "ufw --version"
+ if self._check_dependency_command(ufw_cmdline)[0] != 0:
+ return (False, ["ufw"])
+ else:
+ return (True, None)
+++ /dev/null
-#
-# ufw_journald.py
-#
-# Get details about packets blocked by ufw (uses journald)
-#
-
-import datetime
-import re
-from systemd import journal
-
-from logparse import config
-from logparse.formatting import *
-from logparse.load_parsers import Parser
-from logparse.util import resolve
-
-PROTOCOLS = ["TCP", "UDP", "UDP-Lite", "ICMP", "ICMPv6", "AH", "SCTP", "MH"]
-
-class Packet():
-
- def __init__(self, msg):
- try:
- self.inif, self.outif, self.mac, self.src, self.dst, self.len, self.proto, self.spt, self.dpt = re.search(r"IN=(?P<inif>\w*).*OUT=(?P<outif>\w*).*MAC=(?P<mac>\S*).*SRC=(?P<src>\S*).*DST=(?P<dst>\S*).*LEN=(?P<length>\d*).*PROTO=(?P<proto>\S*)(?:\sSPT=(?P<spt>\d*))?(?:\sDPT=(?P<dpt>\d*))?", msg).groupdict().values()
- if self.proto and self.proto.isdigit():
- self.proto = PROTOCOLS[int(self.proto)-1]
- except Exception as e:
- logger.warning("Malformed packet log: {0}. Error message: {1}".format(msg, str(e)))
- return None
-
-class UfwJournald(Parser):
-
- def __init__(self):
- super().__init__()
- self.name = "ufw_journald"
- self.info = "Get details about packets blocked by ufw"
-
- def parse_log(self):
-
- logger.debug("Starting ufw section")
- section = Section("ufw")
-
- j = journal.Reader()
- j.this_machine()
- j.add_match(_TRANSPORT='kernel')
- j.add_match(PRIORITY=4)
- j.seek_realtime(section.period.startdate)
-
- logger.debug("Searching for messages")
-
- blocked_packets = [Packet(entry["MESSAGE"]) for entry in j if "MESSAGE" in entry and "UFW BLOCK" in entry["MESSAGE"]]
-
- logger.debug("Parsing messages")
-
- inbound_interfaces = []
- outbound_interfaces = []
- n_inbound = n_outbond = 0
- src_ips = []
- dst_ips = []
- src_ports = []
- dst_ports = []
- protocols = {'UDP': 0, 'TCP': 0}
- src_macs = []
-
- for pkt in blocked_packets:
- if pkt.inif:
- inbound_interfaces.append(pkt.inif)
- elif pkt.outif:
- outbound_interfaces.append(pkt.outif)
- if pkt.src: src_ips.append(resolve(pkt.src, config.prefs.get("ufw", "ufw-resolve-domains")))
- if pkt.dst: dst_ips.append(resolve(pkt.dst, config.prefs.get("ufw", "ufw-resolve-domains")))
- if pkt.spt: src_ports.append(pkt.spt)
- if pkt.dpt: dst_ports.append(pkt.dpt)
- if pkt.proto: protocols[pkt.proto] += 1
-
- section.append_data(Data(subtitle="{} blocked ({} UDP, {} TCP)".format(plural("packet", len(blocked_packets)), protocols['UDP'], protocols['TCP'])))
-
- src_port_data = Data(items=src_ports)
- src_port_data.orderbyfreq()
- src_port_data.subtitle = plural("source port", len(src_port_data.items))
- src_port_data.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(src_port_data)
-
- dst_port_data= Data(items=dst_ports)
- dst_port_data.orderbyfreq()
- dst_port_data.subtitle = plural("destination port", len(dst_port_data.items))
- dst_port_data.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(dst_port_data)
-
- src_ips_data= Data(items=src_ips)
- src_ips_data.orderbyfreq()
- src_ips_data.subtitle = plural("source IP", len(src_ips_data.items))
- src_ips_data.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(src_ips_data)
-
- dst_ips_data= Data(items=dst_ips)
- dst_ips_data.orderbyfreq()
- dst_ips_data.subtitle = plural("destination IP", len(dst_ips_data.items))
- dst_ips_data.truncl(config.prefs.getint("logparse", "maxlist"))
- section.append_data(dst_ips_data)
-
- logger.info("Finished ufw section")
- return section
-#
-# zfs.py
-#
-# Look through ZFS logs to find latest scrub and its output.
-# Note that ZFS doesn't normally produce logs in /var/log, so for this to
-# work, we must set up a cron job to dump `zpool iostat` into a file (hourly
-# is best):
-#
-# zpool iostat > /var/log/zpool.log && zpool status >> /var/log/zpool.log
-#
-# The file gets overwritten every hour, so if more than one scrub occurs
-# between logparse runs, it will only get the latest one.
-#
-# TODO: add feature to specify pools to check in config file
-# TODO: set critical value for scrub data repair
-#
+"""
+Look through ZFS logs to find latest scrub and its output.
+Note that ZFS doesn't normally produce logs in /var/log, so for this to
+work, we must set up a cron job to dump `zpool iostat` into a file (hourly is
+best):
+ `zpool iostat > /var/log/zpool.log && zpool status >> /var/log/zpool.log`
+
+The file gets overwritten every hour, so if more than one scrub occurs
+between logparse runs, it will only get the latest one.
+
+TODO: add feature to specify pools to check in config file
+TODO: set critical value for scrub data repair
+"""
import re
import sys, traceback
def __init__(self):
super().__init__()
self.name = "zfs"
- self.info = "Look through ZFS logs to find latest scrub and its output."
+ self.info = "Look through ZFS logs to find latest scrub and its output"
def parse_log(self):
logger.debug("Analysing zpool log")
pool = re.search('.*---\n(\w*)', zfslog).group(1)
- scrub = re.search('.* scrub repaired (\d+\s*\w+) in .* with (\d+) errors on (\w+)\s+(\w+)\s+(\d+)\s+(\d{1,2}:\d{2}):\d+\s+(\d{4})', zfslog)
+ scrub = re.search(".* scrub repaired (\d+\s*\w+) in .* with (\d+) "
+ "errors on (\w+)\s+(\w+)\s+(\d+)\s+(\d{1,2}:\d{2}):"
+ "\d+\s+(\d{4})", zfslog)
logger.debug("Found groups {0}".format(scrub.groups()))
iostat = re.search('.*---\n\w*\s*(\S*)\s*(\S*)\s', zfslog)
scrubrepairs = scruberrors = scrubdate = None
if (scrubdate != None):
scrub_data = Data("Scrub of " + pool + " on " + scrubdate)
- scrub_data.items = [scrubrepairs + " repaired", scruberrors + " errors", alloc + " used", free + " free"]
+ scrub_data.items = [scrubrepairs + " repaired",
+ scruberrors + " errors", alloc + " used", free + " free"]
else:
scrub_data = Data(pool)
scrub_data.items = [alloc + " used", free + " free"]
- `getlocaldomain`: get the current machine's domain name
- `resolve`: attempt to convert a local/public IP to hostname
- `readlog`: read contents of a log file from disk
+And the following classes:
+ - `LogPeriod`: period to search logs (wrapper for datetime.timedelata)
"""
from datetime import datetime, timedelta
import copy
+from configparser import NoSectionError
import ipaddress
import logging
import os
from pkg_resources import Requirement, resource_filename
import re
import socket
-from systemd import journal
+from sys import exit
from logparse import config, formatting
from logparse.timeparse import timeparse
hn = socket.gethostbyaddr(ip)[0] # resolve ip to hostname
except socket.herror:
# cannot resolve ip
- logger.debug(ip + " cannot be found, might not exist anymore")
+ # LOGGING DISABLED TO AVOID SPAM
+# logger.debug(ip + " cannot be found, might not exist anymore")
return(ip)
except Exception as err:
logger.warning("Failed to resolve hostname for " + ip + ": " + str(err))
try:
return open(path, mode).read()
except IOError or OSError as e:
- logger.warning("Error reading log at {0}: {1}"
+ logger.error("Error reading log at {0}: {1}"
.format(path, e.strerror))
return 1
class LogPeriod:
+ """
+ Represents a time period for which logs should be parsed (this is given to
+ journald.seek_realtime). Uses timeparse.py by Will Roberts.
+ """
def __init__(self, section):
- if config.prefs.get(section.split("_")[0], "period"):
+ """
+ If no period is defined for the section config, it is inherited from
+ the global config. Failing that, the program will die.
+ """
+ try:
self.startdate = datetime.now() \
- timeparse(config.prefs.get(section.split("_")[0], "period"))
logger.debug("Parsing logs for {0} since {1}".format(
section, self.startdate.strftime(formatting.DATEFMT
+ " " + formatting.TIMEFMT)))
self.unique = True
- else:
+ except (NoSectionError, TypeError):
+ logger.debug("No period defined in section {0} - inheriting "
+ "global period".format(section))
self.startdate = datetime.now() \
- timeparse(config.prefs.get("logparse", "period"))
self.unique = False
+ except Exception as e:
+ logger.error("Could not find valid time period for parser {0}"
+ " {1}".format(section, e))
+ return None
+
+ def compare(self, d, ephemeral=False) -> bool:
+ """
+ Compare the datetime `d` of a log record with the starting datetime of
+ this LogPeriod and return a boolean result representing whether the
+ log record is after the starting datetime. If either `self.startdate`
+ or `d` are UTC-naive, an appropriate correction is attempted. The
+ `ephemeral` boolean argument, if set to true, prevents permanent
+ modification of `self.startdate` to match the UTC-awareness of the
+ section's log records, in the case that `d` may represent an anomaly.
+ """
+ try:
+ # First attempt a direct comparison
+ return True if d > self.startdate else False
+
+ except TypeError as e:
+ # If the direct comparison fails, make sure that both objects have
+ # a UTC offset
+
+ if d.tzinfo is None \
+ or (d.tzinfo is not None
+ and not d.tzinfo.utcoffset(d)):
+ # d has no timezone info,
+ # OR d has timezone info but no offset
+
+ if self.startdate.tzinfo is not None \
+ and self.startdate.tzinfo.utcoffset(self.startdate):
+ # If d is naive and self.startdate is aware, assume that
+ # the timezone of d is the same as self.startdate by
+ # making self.startdate naive
+
+ logger.warning("{0} checking date {1}: {2}. Time "
+ "of log record is naive. Inheriting UTC "
+ "offset for logs from system date".format(
+ e.__class__.__name__, d, e))
+ if ephemeral:
+ # Compare with UTC-naive version of self.startdate
+ return True if d > self.startdate.replace(tzinfo=None) \
+ else False
+ else:
+ # Remove UTC awareness for self.startdate
+ self.startdate = self.startdate.replace(tzinfo=None)
+ return True if d > self.startdate else False
+
+ elif self.startdate.tzinfo is None \
+ or (self.startdate.tzinfo is not None
+ and not self.startdate.tzinfo.utcoffset(self.startdate)):
+ # d has timezoneinfo and offset, but self.startdate has either
+ # no timezone info, or timezone info and no offset. In this
+ # case, self.startdate inherits the offset of d.
+
+ logger.warning("{0} checking date {1}: {2}. Time of "
+ "start date is naive. Inheriting UTC offset "
+ "for date comparison from logs".format(
+ e.__class__.__name__, d, e))
+
+ if ephemeral:
+ # Compare with UTC-aware version of self.startdate
+ return True if d > self.startdate.astimezone(d.tzinfo) \
+ else False
+ else:
+ # Add UTC awareness for self.startdate
+ self.startdate = self.startdate.astimezone(d.tzinfo)
+ return True if d > self.startdate else False
+
+ else:
+ # Other errors return false (effectively ignore this record)
+ logger.error("{0} comparing date {1}: {2}".format(
+ e.__class__.__name__, d, e))
+ return False