f824ae98ae76b182f429fd7adfeb808254e4a0bc
   1-----------
   2logparse(8)
   3-----------
   4
   5.. _synopsis:
   6
   7========
   8Synopsis
   9========
  10
  11`logparse` [options]
  12
  13.. _description:
  14
  15===========
  16Description
  17===========
  18
  19Logparse is a simple and extensible log analyser which parses and summaries medium-term server logs (up to around 30 days old). It reports notable events and statistics reported from common server programs, and has a comprehensive API to allow users to write their own parsers for custom services.
  20
  21Logparse is also integrated with systemd's `logrotate` to optionally rotate logs only after they have been summarised, and the user may choose to get an email (requires postfix) or a static HTML/plaintext file with the log summary. As an added feature, IP addresses from ssh/samba/apache logs may be resolved to either hostnames or FQDNs.
  22
  23Configuration is through the file `/etc/logparse.conf`, in INI format. A description of the configuration variables is shown below at :ref:`configuration`.
  24
  25Some features require logparse to be run as root (primarily log rotation). It is recommended to set it up on a cron job on a weekly basis.
  26
  27.. _usage:
  28
  29======
  30Usage
  31======
  32.. argparse::
  33   :module: logparse.interface
  34   :func: get_argparser
  35   :prog: logparse
  36   :nodescription: True
  37
  38.. _parsers:
  39
  40=======
  41Parsers
  42=======
  43
  44The program is based on a model of independent **parsers** (consisting of Python modules) which analyse logs from a particular service. Logparse comes with a range of these built in, but additional parsers can be written in Python and placed in `/usr/share/logparse/parsers`. At the moment, the built-in parsers are:
  45
  46- cron (DEPRECATED) - number of commands, list commands (root user only)
  47- cron-journald - number of commands, list commands, list commands per user (requires libsystemd)
  48- httpd - list requests, clients, user agents, bytes transferred, no. of errors
  49- mem - get installed/usable/free memory
  50- postfix - list recipients and bytes sent
  51- smbd - number of logins, list users & clients
  52- sshd (DEPRECATED) - logins by user/hostname, attempted root logins, invalid users
  53- sshd-journald - logins by user/hostname, attempted root logins, invalid users (requires libsystemd)
  54- sudo (DEPRECATED)- number of sessions, list users and commands
  55- sudo-journald - number of sessions, list users and commands (requires libsystemd)
  56- sysinfo - hostname, OS, OS version, platform, processors
  57- systemctl - system status, running/failed units (requires libsystemd)
  58- temperature - instantaneous temperatures of motherboard, CPU, cores, disks
  59- ufw - blocked packets, port and IP data (requires libsystemd)
  60- zfs - zpool scrub reports, disk usage
  61
  62.. _configuration:
  63
  64=============
  65Configuration
  66=============
  67
  68Logparse can be configured with various options in the file `/etc/logparse/logparse.conf`. Alternatively, the configuration file may be placed elsewhere and referenced with the `-c|--config` command line option.
  69
  70Configuration files are written in INI format and interpreted using the standard Python :py:mod:`configparser` module. The file is divided into sections denoted by a line with the section name in square brackets, e.g. `[logparse]` for the section called logparse (this is the main section where global options are placed). Other sections are defined for each parser, e.g. `[httpd]` for the section of configuration values for the httpd parser. For more information on configuration syntax, see <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>.
  71
  72############################################
  73General configuration (`[logparse]` section)
  74############################################
  75
  76output
  77  File to which the logparse output (HTML or plaintext) is saved. If empty, no output is written. Analogous to the command line option -d|--destination which overrides this config value. Default: empty
  78overwrite
  79  Whether to automatically overwrite an existing output file. Default: false
  80title
  81  Title for HTML and plaintext, shown at the top of the output. This string supports the standard variables which are substituted for various metadata in the program (see :ref:`variables`). Default: logparse
  82maxlist
  83  Maximum number of regular text items to show in bulleted lists, e.g. user agents for the httpd parser. Default: 10
  84maxcmd
  85  Maximum number of command items to show in bulleted lists, e.g. recently executed commands in the cron parser. Default: 6
  86resolve-domains
  87  Global setting for resolution of IP addresses into domains. If set to `ip`, logparse will never resolve IP addresses into domains. If set to `fqdn`, IPs will be resolved to show full the full hostname and domain. If set to `fqdn-implicit`, FQDNs will be shown except for local addresses which just show the hostname. If set to `host-only`, only the first segment of the FQDN is shown (i.e. the hostname). This configuration value is passed to the :py:meth:`logparse.util.resolve` function which can be implemented by parsers. This option can also be set in individual parsers, where it is called `resolve-domains-x` where x is the name of the parser. These parser-specific declerations override the global one. Default: fqdn
  88rotate
  89  Whether to rotate logs using systemd's logrotate utility. Analogous to the command line option -r|--rotate and -nr|--no-rotate which override this configuration value. Default: false
  90verbose
  91  High level of debug output. Analogous to the command line option -v|--verbose. Default: false
  92quiet
  93  No output to stdout except for printing the resulting output (enabled with the `-p|--print` command line option) and critical errors. Analogous to the command line option -q|--quiet. Default: false
  94hostname-path
  95  Path to file containing the hostname of the current machine (FQDN or not). This is used to identify the machine in the log output. Default: /etc/hostname
  96parsers
  97  Space-separated list of parsers to enable. If empty, all the included parsers are run except for deprecated ones. Analogous to the command line option -l|--logs. Default: empty
  98ignore-parsers
  99  Space-separated list of parsers to ignore (i.e. not execute). If empty, no parsers are ignored. Analogous to the command line option -nl|--ignore-logs. Default: empty
 100
 101##############################################
 102HTML specific configuration (`[html]` section)
 103##############################################
 104
 105header
 106  Path to a header template which will be prepended to HTMl files and HTML emails. This template includes the entire <head> section and the <body> tag, and by default it outputs a title and some basic metadata at the top of the page. The template file can include variables for substitution, see :ref:`variables`. Default: /etc/logparse/header.html
 107css
 108  Path to a CSS stylesheet which will be linked or converted to inline tags for all HTML output (files and email). Default: /etc/logparse/main.css
 109embed-styles
 110  Whether to convert the stylesheet to inline tags. This is enforced for all HTML emails, and this option allows this same conversion for standalone files. This is done using the :py:mod:`premailer` package. Default: false
 111css-relpath
 112  If embed-styles is set to false, this option will convert the URL of the stylesheet (supplied by the css option) into a path relative to the output directory (if writing to a file) or the current directory (if outputting to stdout). Otherwise, the stylesheet is linked to exactly as supplied in the css option. Default: true
 113
 114#####################################################
 115Plain-text specific configuration (`[plain]` section)
 116#####################################################
 117plain
 118  Enable plaintext output instead of HTML. Analogous to the -p|--print command line option. Default: false
 119
 120linewidth
 121  Limit regular text lines to this number of characters. Note tables do not obey this variable (this is on the todo list). Default: 80
 122
 123#####################################
 124Log file locations (`[logs]` section)
 125#####################################
 126
 127This section defines paths to log files used for each parser. These paths may vary depending on the operating system and service configuration. Any value can be added to this section, so user-supplied parsers can get their logfile locations from here as well. However, there is currently no mechanism of setting default values for user-supplied parsers, so defaults must be set in the parser files themselves.
 128
 129auth
 130  Path to the system's authentication log. Default: /var/log/auth.log
 131cron
 132  Path to the cron log. Default: /var/log/cron.log
 133cpuinfo
 134  Path to the system's CPU info file in procfs. Default: /proc/cpuinfo
 135meminfo
 136  Path to the system's memory info file in procfs. Default: /proc/meminfo
 137sys
 138  Path to the system's primary syslog. Default: /var/log/syslog
 139smbd
 140  Path to the directory for Samba logfiles. Default: /var/log/samba
 141zfs
 142  Path to a file which contains the output of `zpool status`. Default: /var/log/zpool.log
 143postfix
 144  Path to the postfix log. Default: /var/log/mail.log
 145httpd-access
 146  Path to Apache's access log. Default: /var/log/apache2/access.log
 147httpd-error
 148  Path to Apache's error log. Default: /var/log/apache2/error.log
 149
 150######################################
 151Email configuration (`[mail]` section)
 152######################################
 153
 154Email is sent using the default mail transfer agent (usually Postfix).
 155
 156to
 157  Recipient address. If empty, no email is sent. Analogous to the -t|--to command line option. Default: empty
 158from
 159  Sender address. If empty, default sender address is determined by the MTA. Default: empty
 160subject
 161  Text to use as the subject of the message. Variables may be substituted with metadata (see :ref:`variables`). Default: logparse from $hostname
 162mailbin
 163  Path to the MTA binary (usually Postfix). Default: /usr/bin/mail
 164
 165======================
 166Default parser options
 167======================
 168
 169Each parser has its own set of options in a section with the name of the parser. In the case of multiple versions of the same parser (e.g. sshd and sshd-journald), the configuration section goes by the base name (e.g. sshd). Options defined in individual parser sections override those defined in the global configuration.
 170
 171####
 172cron
 173####
 174
 175period
 176  Maximum age of logs to analyse. Overrides global config. Only used in cron-journald at the moment. See :ref:`period` for more information. Default: empty
 177
 178.. _period:
 179
 180####
 181sshd
 182####
 183
 184period
 185  Maximum age of logs to analyse. Overrides global config. Only used in sshd-journald at the moment. See :ref:`period` for more information. Default: empty
 186sshd-resolve-domains
 187  DNS lookup configuration for sshd parsers only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty
 188
 189####
 190smbd
 191####
 192
 193shares
 194  Regular expression string for which Samba shares to include when parsing logs. To consider all shares, set this to `.*`. To exclude a certain share, use negative lookaround. Default: `^((?!IPC\$).)*$`
 195users
 196  Regular expression string for which user@hostname values to include when parsing logs. This could be used to exclude logins from a trusted user or hostname. Default: `.*`
 197smbd-resolve-domains
 198  DNS lookup configuration for smbd parsers only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty
 199period
 200  Maximum age of logs to analyse. Overrides global config. Only used in smbd-journald at the moment. See :ref:`period` for more information. Default: empty
 201
 202#####
 203httpd
 204#####
 205
 206httpd-resolve-domains
 207  DNS lookup configuration for httpd parser only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty
 208period
 209  Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
 210
 211###
 212ufw
 213###
 214
 215ufw-resolve-domains
 216  DNS lookup configuration for ufw parser only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty
 217period
 218  Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
 219
 220####
 221sudo
 222####
 223
 224period
 225  Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
 226
 227#########
 228systemctl
 229#########
 230
 231period
 232  Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty
 233show-all
 234  Whether to include services which are running but okay in the output. Default: true 
 235
 236
 237
 238========================
 239Log period configuration
 240========================
 241
 242Some parsers support custom time periods to be searched for logs. This period is specified as a string in the configuration section of supported parsers, and is a timespan relative to the time when the parser is initialised. The time parsing functionality uses a modified version of `timeparse.py` originally written by Will Roberts under the MIT License. The following excerpt is taken from the documentation of `timeparse.py`:
 243
 244.. autofunction:: logparse.timeparse.strseconds
 245
 246.. _variables:
 247
 248=====================
 249Variable substitution
 250=====================
 251
 252In some configuration options, variables may be used to substitute values at runtime. Variable syntax is simply a dollar sign ($) followed by the variable name. The following variables are supported:
 253
 254`$css`
 255  Path to the CSS stylesheet as determined by the css configuration option
 256`$date`
 257  Date when processing was started
 258`$time`
 259  Time when processing was started
 260`$hostname`
 261  Hostname as specified by the hostname configuration option
 262`$title`
 263  Value of the title variable as determined by the title option
 264`$version`
 265  Version of logparse
 266
 267
 268.. _api:
 269
 270=============
 271API structure
 272=============
 273
 274Due to the extensible nature of logparse, many of the core functions can be used in user-supplied parsers. Each parser is written in Python as a single-file module which defines a derivative of the Parser class (see `logparse.load_parsers.Parser` below). Therefore parser classes which inherit the base class have access to all the normal formatting and I/O functions which are documented below.
 275
 276If logparse is to be imported as a Python package into another project for some reason, you can control the general operations with the `logparse.interface` module.
 277
 278##################
 279logparse.interface
 280##################
 281
 282.. automodule:: logparse.interface
 283   :members:
 284
 285#####################
 286logparse.load_parsers
 287#####################
 288
 289.. automodule:: logparse.load_parsers
 290   :members:
 291
 292###################
 293logparse.formatting
 294###################
 295
 296.. automodule:: logparse.formatting
 297   :members:
 298
 299================
 300More information
 301================
 302| Readme    https://git.lorimer.id.au/logparse.git/about
 303| Source    https://git.lorimer.id.au/logparse.git/tree
 304| Contact   mailto:bugs@lorimer.id.au