1----------- 2logparse(8) 3----------- 4 5.. _synopsis: 6 7======== 8Synopsis 9======== 10 11`logparse` [options] 12 13.. _description: 14 15=========== 16Description 17=========== 18 19Logparse is a simple and extensible log analyser which parses and summaries medium-term server logs (up to around 30 days old). It reports notable events and statistics reported from common server programs, and has a comprehensive API to allow users to write their own parsers for custom services. 20 21Logparse is also integrated with systemd's `logrotate` to optionally rotate logs only after they have been summarised, and the user may choose to get an email (requires postfix) or a static HTML/plaintext file with the log summary. As an added feature, IP addresses from ssh/samba/apache logs may be resolved to either hostnames or FQDNs. 22 23Configuration is through the file `/etc/logparse.conf`, in INI format. A description of the configuration variables is shown below at :ref:`configuration`. 24 25Some features require logparse to be run as root (primarily log rotation). It is recommended to set it up on a cron job on a weekly basis. 26 27.. _usage: 28 29====== 30Usage 31====== 32.. argparse:: 33 :module: logparse.interface 34 :func: get_argparser 35 :prog: logparse 36 :nodescription: True 37 38.. _parsers: 39 40======= 41Parsers 42======= 43 44The program is based on a model of independent **parsers** (consisting of Python modules) which analyse logs from a particular service. Logparse comes with a range of these built in, but additional parsers can be written in Python and placed in `/usr/share/logparse/parsers`. At the moment, the built-in parsers are: 45 46#### 47cron 48#### 49 50.. automodule:: logparse.parsers.cron 51 52#### 53httpd 54#### 55 56.. automodule:: logparse.parsers.httpd 57 58- mem - get installed/usable/free memory 59- postfix - list recipients and bytes sent 60- smbd - number of logins, list users & clients 61- sshd (DEPRECATED) - logins by user/hostname, attempted root logins, invalid users 62- sshd-journald - logins by user/hostname, attempted root logins, invalid users (requires libsystemd) 63- sudo (DEPRECATED)- number of sessions, list users and commands 64- sudo-journald - number of sessions, list users and commands (requires libsystemd) 65- sysinfo - hostname, OS, OS version, platform, processors 66- systemd - system status, running/failed units (requires libsystemd) 67- temperature - instantaneous temperatures of motherboard, CPU, cores, disks 68- ufw - blocked packets, port and IP data (requires libsystemd) 69- zfs - zpool scrub reports, disk usage 70 71.. _configuration: 72 73============= 74Configuration 75============= 76 77Logparse can be configured with various options in the file `/etc/logparse/logparse.conf`. Alternatively, the configuration file may be placed elsewhere and referenced with the `-c|--config` command line option. 78 79Configuration files are written in INI format and interpreted using the standard Python :py:mod:`configparser` module. The file is divided into sections denoted by a line with the section name in square brackets, e.g. `[logparse]` for the section called logparse (this is the main section where global options are placed). Other sections are defined for each parser, e.g. `[httpd]` for the section of configuration values for the httpd parser. For more information on configuration syntax, see <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>. 80 81############################################ 82General configuration (`[logparse]` section) 83############################################ 84 85output 86 File to which the logparse output (HTML or plaintext) is saved. If empty, no output is written. Analogous to the command line option -d|--destination which overrides this config value. Default: empty 87overwrite 88 Whether to automatically overwrite an existing output file. Default: false 89title 90 Title for HTML and plaintext, shown at the top of the output. This string supports the standard variables which are substituted for various metadata in the program (see :ref:`variables`). Default: logparse 91maxlist 92 Maximum number of regular text items to show in bulleted lists, e.g. user agents for the httpd parser. Default: 10 93maxcmd 94 Maximum number of command items to show in bulleted lists, e.g. recently executed commands in the cron parser. Default: 6 95resolve-domains 96 Global setting for resolution of IP addresses into domains. If set to `ip`, logparse will never resolve IP addresses into domains. If set to `fqdn`, IPs will be resolved to show full the full hostname and domain. If set to `fqdn-implicit`, FQDNs will be shown except for local addresses which just show the hostname. If set to `host-only`, only the first segment of the FQDN is shown (i.e. the hostname). This configuration value is passed to the :py:meth:`logparse.util.resolve` function which can be implemented by parsers. This option can also be set in individual parsers, where it is called `resolve-domains-x` where x is the name of the parser. These parser-specific declerations override the global one. Default: fqdn 97rotate 98 Whether to rotate logs using systemd's logrotate utility. Analogous to the command line option -r|--rotate and -nr|--no-rotate which override this configuration value. Default: false 99verbose 100 High level of debug output. Analogous to the command line option -v|--verbose. Default: false 101quiet 102 No output to stdout except for printing the resulting output (enabled with the `-p|--print` command line option) and critical errors. Analogous to the command line option -q|--quiet. Default: false 103hostname-path 104 Path to file containing the hostname of the current machine (FQDN or not). This is used to identify the machine in the log output. Default: /etc/hostname 105parsers 106 Space-separated list of parsers to enable. If empty, all the included parsers are run except for deprecated ones. Analogous to the command line option -l|--logs. Default: empty 107ignore-parsers 108 Space-separated list of parsers to ignore (i.e. not execute). If empty, no parsers are ignored. Analogous to the command line option -nl|--ignore-logs. Default: empty 109datetime-format 110 String representing the format for dates and times in the logfiles (when using plain logfiles, not journald parsers). This should be a standard Python strptime format (see <https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior>). Supported parsers allow the datetime-format to be set in their individual sections, which overrides this global value. Default: %b %d %H:%M:%S 111 112############################################## 113HTML specific configuration (`[html]` section) 114############################################## 115 116header 117 Path to a header template which will be prepended to HTMl files and HTML emails. This template includes the entire <head> section and the <body> tag, and by default it outputs a title and some basic metadata at the top of the page. The template file can include variables for substitution, see :ref:`variables`. Default: /etc/logparse/header.html 118css 119 Path to a CSS stylesheet which will be linked or converted to inline tags for all HTML output (files and email). Default: /etc/logparse/main.css 120embed-styles 121 Whether to convert the stylesheet to inline tags. This is enforced for all HTML emails, and this option allows this same conversion for standalone files. This is done using the :py:mod:`premailer` package. Default: false 122css-relpath 123 If embed-styles is set to false, this option will convert the URL of the stylesheet (supplied by the css option) into a path relative to the output directory (if writing to a file) or the current directory (if outputting to stdout). Otherwise, the stylesheet is linked to exactly as supplied in the css option. Default: true 124 125##################################################### 126Plain-text specific configuration (`[plain]` section) 127##################################################### 128plain 129 Enable plaintext output instead of HTML. Analogous to the -p|--print command line option. Default: false 130 131linewidth 132 Limit regular text lines to this number of characters. Note tables do not obey this variable (this is on the todo list). Default: 80 133 134##################################### 135Log file locations (`[logs]` section) 136##################################### 137 138This section defines paths to log files used for each parser. These paths may vary depending on the operating system and service configuration. Any value can be added to this section, so user-supplied parsers can get their logfile locations from here as well. However, there is currently no mechanism of setting default values for user-supplied parsers, so defaults must be set in the parser files themselves. 139 140auth 141 Path to the system's authentication log. Default: /var/log/auth.log 142cron 143 Path to the cron log. Default: /var/log/cron.log 144cpuinfo 145 Path to the system's CPU info file in procfs. Default: /proc/cpuinfo 146meminfo 147 Path to the system's memory info file in procfs. Default: /proc/meminfo 148sys 149 Path to the system's primary syslog. Default: /var/log/syslog 150smbd 151 Path to the directory for Samba logfiles. Default: /var/log/samba 152zfs 153 Path to a file which contains the output of `zpool status`. Default: /var/log/zpool.log 154postfix 155 Path to the postfix log. Default: /var/log/mail.log 156httpd-access 157 Path to Apache's access log. Default: /var/log/apache2/access.log 158httpd-error 159 Path to Apache's error log. Default: /var/log/apache2/error.log 160 161###################################### 162Email configuration (`[mail]` section) 163###################################### 164 165Email is sent using the default mail transfer agent (usually Postfix). 166 167to 168 Recipient address. If empty, no email is sent. Analogous to the -t|--to command line option. Default: empty 169from 170 Sender address. If empty, default sender address is determined by the MTA. Default: empty 171subject 172 Text to use as the subject of the message. Variables may be substituted with metadata (see :ref:`variables`). Default: logparse from $hostname 173mailbin 174 Path to the MTA binary (usually Postfix). Default: /usr/bin/mail 175 176====================== 177Default parser options 178====================== 179 180Each parser has its own set of options in a section with the name of the parser. In the case of multiple versions of the same parser (e.g. sshd and sshd-journald), the configuration section goes by the base name (e.g. sshd). Options defined in individual parser sections override those defined in the global configuration. 181 182###################### 183cron and cron_journald 184###################### 185 186commands 187 Regular expression string for which commands to include when parsing logs. If `truncate-commands` is set to true, then the truncated command will be compared against the regex pattern, not the full command. Default: `.*` 188datetime-format 189 String representing the format for dates and times in the logfiles (when using cron, not cron_journald). This should be a standard Python strptime format (see <https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior>). Reverts to the global config if empty. Default: empty 190list-users 191 Display a list of the most common commands run by each user (this may be combined with the `summary` option below). Default: true 192period 193 Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty 194summary 195 Show a summary of cron sessions. This consists of the total number of sessions, total number of users, and a list of the most popular commands and who executed them. Default: false 196truncate-commands 197 Whether to remove absolute directory paths in commands. When set to true, a command such as `/usr/bin/cat` will become `cat`. Default: true 198users 199 Regular expression string for which usernames to include when parsing logs. This could be used to exclude cron sessions from trusted users. Default: `.*` 200 201 202###################### 203sshd and sshd_journald 204###################### 205 206period 207 Maximum age of logs to analyse. Overrides global config. Only used in sshd-journald at the moment. See :ref:`period` for more information. Default: empty 208sshd-resolve-domains 209 DNS lookup configuration for sshd parsers only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty 210 211###################### 212smbd and smbd_journald 213###################### 214 215shares 216 Regular expression string for which Samba shares to include when parsing logs. To consider all shares, set this to `.*`. To exclude a certain share, use negative lookaround. Default: `^((?!IPC\$).)*$` 217users 218 Regular expression string for which user@hostname values to include when parsing logs. This could be used to exclude logins from a trusted user or hostname. Default: `.*` 219smbd-resolve-domains 220 DNS lookup configuration for smbd parsers only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty 221period 222 Maximum age of logs to analyse. Overrides global config. Only used in smbd-journald at the moment. See :ref:`period` for more information. Default: empty 223 224##### 225httpd 226##### 227 228httpd-resolve-domains 229 DNS lookup configuration for httpd parser only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty 230period 231 Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty 232 233### 234ufw 235### 236 237ufw-resolve-domains 238 DNS lookup configuration for ufw parser only (overrides global config). Accepted values are `ip`, `fqdn`, `fqdn-implicit`, and `host-only`. See the global setting `resolve-domains` for more information. Default: empty 239period 240 Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty 241 242###################### 243sudo and sudo_journald 244###################### 245 246list-users 247 Display a list of the most common commands initiated by each user (this may be combined with the `summary` option below). Default: true 248period 249 Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty 250summary 251 Show a summary of sudo sessions and most popular commands. Default: false 252truncate-commands 253 Whether to remove absolute directory paths in commands. When set to true, a command such as `/usr/bin/cat` will become `cat`. Default: true 254 255####### 256systemd 257####### 258 259period 260 Maximum age of logs to analyse. Overrides global config. See :ref:`period` for more information. Default: empty 261show-all 262 Whether to include services which are running but okay in the output. Default: true 263 264 265.. _period: 266 267======================== 268Log period configuration 269======================== 270 271Some parsers support custom time periods to be searched for logs. This period is specified as a string in the configuration section of supported parsers, and is a timespan relative to the time when the parser is initialised. The time parsing functionality uses a modified version of `timeparse.py` originally written by Will Roberts under the MIT License. The following excerpt is taken from the documentation of `timeparse.py`: 272 273.. autofunction:: logparse.timeparse.strseconds 274 275.. _variables: 276 277===================== 278Variable substitution 279===================== 280 281In some configuration options, variables may be used to substitute values at runtime. Variable syntax is simply a dollar sign ($) followed by the variable name. The following variables are supported: 282 283`$css` 284 Path to the CSS stylesheet as determined by the css configuration option 285`$date` 286 Date when processing was started 287`$time` 288 Time when processing was started 289`$hostname` 290 Hostname as specified by the hostname configuration option 291`$title` 292 Value of the title variable as determined by the title option 293`$version` 294 Version of logparse 295 296 297.. _api: 298 299============= 300API structure 301============= 302 303Due to the extensible nature of logparse, many of the core functions can be used in user-supplied parsers. Each parser is written in Python as a single-file module which defines a derivative of the Parser class (see `logparse.load_parsers.Parser` below). Therefore parser classes which inherit the base class have access to all the normal formatting and I/O functions which are documented below. 304 305If logparse is to be imported as a Python package into another project for some reason, you can control the general operations with the `logparse.interface` module. 306 307################## 308logparse.interface 309################## 310 311.. automodule:: logparse.interface 312 :members: 313 314##################### 315logparse.load_parsers 316##################### 317 318.. automodule:: logparse.load_parsers 319 :members: 320 321################### 322logparse.formatting 323################### 324 325.. automodule:: logparse.formatting 326 :members: 327 328================ 329More information 330================ 331| Readme https://git.lorimer.id.au/logparse.git/about 332| Source https://git.lorimer.id.au/logparse.git/tree 333| Contact mailto:bugs@lorimer.id.au