SAVELOGS(1) User Contributed Perl Documentation SAVELOGS(1)NAMEsavelogs - save/rotate/delete log files nicely
SYNOPSISsavelogs saves your log files in a nice way (by default).
savelogs --postmovehook='/usr/local/bin/restart_apache' \
--apacheconf=/www/conf/httpd.conf /var/log/messages
savelogs `cat list_of_logs_to_process.txt`
savelogs --loglevel=2 /var/log/maillog /var/log/cronlog \
/var/log/messages
savelogs --config=/etc/savelogs.conf
savelogs --period=15 /var/log/messages
savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com
DESCRIPTIONsavelogs is a flexible and robust log file archival system. Its logic
is simple: move (rename) the log file, filter data from the log file,
store the log file in an archive (via tar or gtar), and compress the
archive (via gzip or compress). After successful compression, the
original log file is deleted.
All of the above phases are optional. This means that you may simply
delete files if you wish. Or you may simply compress existing log
files. Or you may move files and add them to a tar file but leave the
tar file uncompressed, etc. You pick ;o)
(If you just want to cut to the chase and don't care how savelogs
works, see the "EXAMPLES" section near the bottom of this document.)
Savelogs Phases
The processing order may be abbreviated into these five phases:
move -> filter -> archive -> compress -> delete
any of which may be subtracted from the process order. In addition to
these phases are some intermediate 'hooks' where you may supply an
external program or shell command to invoke. This is useful if you're
rotating web server log files and you need to HUP (restart) your web
server after moving the logs (for example).
Subtracting phases is done in one of two possible ways. The first way
is to specify it in the configuration file:
Process move,archive,delete
which will move log files and archive them (but not filter or compress
them). After successful archival, the original log files will be
deleted.
The second way is to specify it on the command-line:
--process=compress,delete
which will simply compress log files (but not move, filter, or archive
them).
In addition to the five phase processing options above, you may also
employ the following abbreviations:
(no option specified)
If you specify no process option, the default is move,compress.
none
Do none of the phases. This isn't a very useful option.
all Do all of the phases.
An Overview
A typical savelogs session might begin by typing this command:
savelogs /var/log/messages
After which the following phases will execute:
move
The log file is renamed:
/var/log/messages --> /var/log/messages.010523
compress
The log file is compressed
/var/log/messages.010523 --> /var/log/messages.010523.gz
A Word About Paths
All paths you specify to savelogs should be relative to your home
directory (if you're the root user--uid 0--your home directory is set
to '/'). You do not need to use a tilde (~). You may assume that
savelogs runs from your home directory and knows how to handle absolute
paths.
If my real home directory were located in /usr/home/joe and I wanted to
rotate the log file located in /usr/home/joe/var/log/messages, I would
do something like this:
savelogs /var/log/messages
and savelogs would Do What I Mean.
The only exception to this are external commands given to postmovehook,
postfilterhook and other such options. Paths you specify here really
are full paths.
CONFIGURATION
Configuration file option format
savelogs will read its configuration options from a configuration file
(which you must supply) or from the command-line. Creating a
configuration file is easy: it is a simple Apache-style plaintext file
that has options specified in this format:
Option value
where Option is one of the options below and value is either a
true/false; yes/no; on/off combination or some string value, such as a
pathname and file (depending on the nature of the option).
Your distribution of savelogs may have included a sample configuration
file for you to edit as you wish in ~/etc/savelogs.conf.sample.
Option processing order
Configuration options are first read from savelogs internal defaults,
which are sprinkled throughout this document. Next savelogs reads its
configuration file, if any is specified. Lastly, savelogs uses options
from the command-line.
For example, the default value for the period directive is 10. If you
ran savelogs like this:
savelogs--period /var/log/messages
every day for 10 days, you would have 10 archived log files.
If you have in your configuration file:
Period 5
and ran the same command above every day for 10 days, you'd only have 5
archived log files because the configuration file overrides savelogs
internal defaults. Finally, if you had a configuration file with the
previously mentioned value for period and ran this command:
savelogs --period=7 /var/log/messages
every day for 10 days, you would have 7 archived log files because
command-line options override configuration file options (which
override internal default values).
Available options
All options you may specify on the command-line you may also sepcify in
a configuration file (except the configuration file directive itself).
For example, if you had a cronjob that did this:
savelogs --process=delete \
--postmovehook="/usr/local/apache/bin/apachectl graceful" \
--apacheconf=/www/conf/httpd.conf
(which deletes all apache logs) you could make a nice configuration
file (call it ~/etc/savelogs1.conf) that would do the same thing:
Process delete
PostMoveHook /usr/local/apache/bin/apachectl graceful
ApacheConf /www/conf/httpd.conf
and then invoke your cron like this:
savelogs --config=/etc/savelogs1.conf
Case-sensitivity
A sample configuration file may be located in
~/etc/savelogs.conf.sample which you may copy and edit as you wish.
Configuration file directives are case-insensitive:
Process move,compress
PROCESS move,compress
process move,compress
pROceSs move,compress
are all the same directive to savelogs. Configuration file values ARE
case-sensitive.
ApacheConf /www/conf/httpd.conf
ApacheConf /WWW/conf/httpd.conf
ApacheConf /www/CONF/httpd.conf
ApacheConf /www/conf/Httpd.conf
are four distinct files, depending on the case-sensitivity of your
operating system.
Testing configurations
When you are testing new configuration directives, use the dry-run
option and watch the log output using the loglevel and logfile
directives. This will help you avoid losing data unnecessarily.
You may also specify the settings option which will show you all the
current settings after defaults, configuration file, and command-line
options have been processed.
OPTIONS
Options given below are configuration directives that may appear in a
configuration file or on the command-line. Options are case-
insensitive, i.e., ApacheLog is the same as apachelog, though the
values associated with the options are often case-sensitive (e.g.,
paths, filenames, etc.)
Options specified on the command-line should be prefixed with two
hyphens. Some options do not make sense in a configuration file or need
to occur before the configuration file is parsed such as config, help,
or home.
Running Savelogs
help
Shows a brief usage statement and exits.
Example:
savelogs--help
version
Shows the current version of savelogs and exits.
Example:
savelogs--version
settings
Shows the current settings of savelogs and exits.
Example:
savelogs--settings --apacheconf=/www/conf/httpd.conf \
/var/log/messages
dry-run
When used with the logfile and loglevel settings, dry-run will show
you what will happen if you were to run savelogs with the current
settings without actually doing it. This is a useful option to
specify if you want to see what effect some changes might have, or
to see which files are going to get archived with the current
settings.
Note that savelogs running under the dry-run directive will
sometimes produce errors that wouldn't occur during normal use.
This can happen for a variety of reasons, mostly related to
savelogs looking for files that don't yet exist, or archives that
don't yet exist because they weren't actually created. In this
respect, dry-run doesn't give you precisely what will happen, but
it does give you a good idea. Use it with a grain of salt.
Example:
savelogs--dry-run --loglevel=2 /var/log/foo
home
Changes the default base location of where savelogs runs from. This
is mostly a debugging utility. Consider this an advanced feature
which should probably be ignored. Defaults to the process owner's
home directory (which is almost always what you want).
Example:
Home /usr/home/joe/usr/home/bob
config
Changes the default configuration file savelogs reads at startup.
This should be done from the command-line or it won't have any
effect.
Example:
savelogs --config=/etc/my_other_savelogs.conf
process=[move],[filter],[archive],[compress],[delete]
Tells savelogs which phases to execute. If you just want to move
(rename) logs, do this:
savelogs --process=move /var/log/messages
and ~/var/log/messages will become ~/var/log/messages.yymmdd (where
yymmdd are today's date).
For just removing logs, specify the delete option. You can get
fancy:
savelogs --process=move,compress /var/log/messages
which renames the log file ~/var/log/messages to
~/var/log/messages.yymmdd and then compresses it, not filtering,
archiving (i.e. putting into a separate tar file), or deleting it
(the compress option also renames the file so that delete becomes
useless since the file as it was no longer exists).
move,compress is the default value for the process option, so the
above directive could have also been specified:
savelogs /var/log/messages
You may also specify all or none as shortcuts for
savelogs --process=move,filter,archive,compress,delete
and
savelogs --process=
respectively.
Savelogs Logging
savelogs has an internal logging facility that helps you diagnose
problems, or just see what's going on. By default, savelogs writes to
stdout (i.e., your screen if you're running this from a tty).
loglevel=#
Determines how verbose savelogs is when it's writing its own
internal messages. Valid values are between 0 and 5 inclusive.
Example:
LogLevel 3
The general rule of thumb for savelogs logging is this:
Level What will be logged
===== ===================
0 no output except fatal errors
1 Level 0 + start/finish stats, errors
2 Level 1 + warnings, logfiles to process
3 Level 2 + chdir, filter, phase completion
4 Level 3 + phase core actions, phase beginning
5 Level 4 + everything else
The first few times you run savelogs, try a higher loglevel value
to see what's happening with your log files. Once you're
comfortable with how savelogs works, you may turn it down a few
notches (level 1 is usually fine) so at least you can check to see
if your cronjob actually ran ;o)
logfile=[stdout|stderr|/path/to/log]
savelogs, depending on the loglevel you've specified, writes what
it's doing, such as moving, archiving, or deleting, etc. The
logfile directive tells savelogs where to write all these messages.
The default value for logfile is stdout which means that your
output will go to the screen unless you've redirected stdout. You
may also specify stderr or the path to a file where you'd like
messages to be appended.
Example:
LogFile /var/log/savelogs.log
How Savelogs Finds Logs to Process
savelogs processes the logs you specify on the command-line (items on
the command-line that are not recognized as options are assumed to be
log files to process).
If no logs are specified (either on the command-line or in a
configuration file using the following directives), savelogs will
complain and show a 'usage' statement. To turn off the usage statement,
use the gripe directive (to gripe is the default behavior):
savelogs--nogripe /no/such/log
To save wear on your finger tips and phosphor in your monitor, we
recommend liberal use of the following configuration directives.
Log=/path/to/log
This works just like adding a file on the command-line, but is
included so that you can put log files you want processed in a
configuration file. It will also work on the command-line:
Example:
savelogs --log=/var/log/messages
is the same as:
savelogs /var/log/messages
which is also equivalent to a config file named
'~/etc/savelogs.conf' with this single line:
Log /var/log/messages
and invoked like this:
savelogs --config=/etc/savelogs.conf
The log directive also accepts any standard csh-ish wildcard (e.g.,
*, ?, [n-m], etc.) for globbing. Globbing is where you specify a
wilcard pattern and the argument list is expanded to all filenames
that match the pattern. In savelogs, this pattern implicitly
excludes files whose names end in '.tar', '.gz', or '.tgz' (so you
don't have to worry about compressing already-compressed files).
This is useful if you have log files that are created dynamically
and whose names you may not know precisely. For example, say you
have a list of files in a directory:
somelog.010909.gz
somelog.01090a.gz
somelog.01090b.gz
somelog.01090c.gz
somelog.01090d.gz
somelog.01090e.gz
somelog.01090f.gz
somelog.011001
somelog.011002
somelog.011003
somelog.011004
somelog.011005
somelog.011006
somelog.011007
somelog.011008
somelog.011009
somelog.01100a
somelog.01100b
somelog.01100c
To compress these files (the ones that have not already been
compressed) you can simply do this:
savelogs --log='/path/to/somelog.*'
The files that end in '.gz' are skipped (savelogs skips them
internally).
Be sure to protect the asterisk (*) with quotes so that the current
shell doesn't try to expand them. You could also do this in a
configuration file:
Log /path/to/somelog.*
NoLog=/path/to/log
This is the compliment to the Log directive: it removes logs from
the list of logs to process. This is useful if you have a log or
set of logs that is handled by a separate rotation program or needs
special treatment at another time.
For example, if you have many log files listed in your Apache
configuration file, you'll want to take advantage of the ApacheConf
directive (see below). This will make your savelogs configuration
file small and easy to understand:
## rename and compress all logs found in httpd.conf
ApacheConf /www/conf/httpd.conf
This is great, except that there's this one log that you don't want
savelogs to process. Before savelogs version 1.40, the only option
you had was to list each log individually with the Log directive
(i.e., you couldn't use the ApacheConf directive at all in such
cases). Now, however, you can use the NoLog directive to exclude
logs that have already been added to the list:
## rename and compress all logs found in httpd.conf
ApacheConf /www/conf/httpd.conf
## ... and exclude joe's logs (joe-*_log matches
## joe-access_log and joe-error_log)
NoLog /www/logs/joe-*_log
You may use full paths or you may use shell wildcard patterns, just
like the Log directive.
If you have both Log and NoLog directives, the NoLog directive is
processed last. This means that:
Log /var/log/messages
NoLog /var/log/messages
is the same as:
NoLog /var/log/messages
Log /var/log/messages
and that /var/log/messages will not be processed in either case.
Gripe|NoGripe
NoGripe tells savelogs not to complain about not finding any log
files to process. By default, savelogs gripes about not finding any
log files: if you forget to specify any logs (or any directives
such as ApacheConf that find logs for you) savelogs will complain.
Also, if you specify log files to process but none of them exist,
savelogs will similarly complain.
When you turn on NoGripe, the complaining is stopped and savelogs
exits happily.
Example:
savelogs--nogripe
or in your configuration file:
Gripe no
ApacheConf=/path/to/httpd.conf
If you specify this option, giving it a valid httpd.conf file,
savelogs will parse your Apache configuration file looking for
standard log file directives. Any files found will be processed.
Example:
savelogs --apacheconf=/www/conf/httpd.conf
or in your configuration file:
ApacheConf /usr/local/etc/httpd/conf/httpd.conf
Using the ApacheConf directive will tell savelogs to search through
httpd.conf looking for all files associated with TransferLog,
ErrorLog, AgentLog, etc. (all those listed in the ApacheLog
directive) and process them.
ApacheHost
This option tells savelogs which logs to select out of the Apache
configuration file (as specified by the ApacheConf directive) based
on the Apache ServerName directive in the VirtualHost block. If
this option is set, only logs for matching hosts will be rotated
(this applies only to logs found in the Apache configuration file;
other logs specified in other ways (e.g., on the command-line or
via the Log directive) will be processed as usual).
Example:
savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com
or in your configuration file:
ApacheConf /www/conf/httpd.conf
ApacheHost foo.com
The ApacheHost directive may be specified multiple times to process
logs for multiple virtual hosts. If no logs are found in the Apache
VirtualHost block, no logs will be rotated for that virtual host.
ApacheLog
This option allows you to tell savelogs which logs to process in
the httpd.conf file specified by apacheconf. The default value for
the apachelog directive is:
TransferLog|ErrorLog|AgentLog|RefererLog|CustomLog
You may do something clever like this:
savelogs --apacheconf=/www/conf/httpd.conf --apachelog=TransferLog
which would archive all of your access_log files. Then after
running this command, you could do this:
savelogs --process=delete --apacheconf=/www/conf/httpd.conf \
--apachelog=ErrorLog
which would delete your error_log files.
In general, the fewer values you specify in the apachelog directive
the faster savelogs will find your log files (though the speedup
really is negligible, it may also save you from rotating logs you
didn't want to).
ApacheLog TransferLog|ErrorLog
would be sufficient for most people using combined Apache logs.
ApacheLogExclude
For those who want somewhat finer control of which logs get
processed in their Apache configuration file, the apachelogexclude
directive allows you to specify a Perl regular expression (which
may simply be a string like 'error') of log files to exclude when
processing logs. This way you could do something like this:
savelogs --apacheconf=/www/conf/httpd.conf --apachelogexclude=logs/bob
which would process all logs found in your httpd.conf file except
log files whose names contain the string 'logs/bob'. Maybe Bob
likes to rotate his logs using another program or system
(conceivable, though unlikely).
Multiple occurances of apachelogexclude are allowed:
ApacheLogExclude /dev/null
ApacheLogExclude \|
ApacheLogExclude logs/bob
which would exclude log files whose names contained '/dev/null' or
'logs/bob' from being processed. Any valid Perl regular expression
will work, so:
ApacheLogExclude ^/dev/null$
is not the same as the previous example. This example will only
match log files whose name is exactly /dev/null, no more, no less.
By default, savelogs uses the following patterns to determine logs
to exclude:
^/dev/null$
\| (this is a literal pipe character)
This means that by default savelogs will not attempt to archive a
log whose name is '/dev/null' or whose name contains a pipe (|). If
for some bizarre reason you wish to remove these defaults when you
run savelogs, you can give an empty apachelogexclude option on the
command-line:
% savelogs --apachelogexclude= --config=/etc/savelogs.conf
Logs that Apache writes via a pipe must be specified separately
using the Log directive or on the command-line.
ApacheInclude
When specified, this directive tells savelogs to read the Apache
configuration file (httpd.conf) and follow Inlcude directives (see
http://httpd.apache.org/docs/mod/core.html#include for details).
It will work just like Apache does: it will look in directories
(when a directory is given as the Include option), it will expand
path wildcards (e.g., "httpd_[bcd].conf" will expand to
httpd_b.conf, httpd_c.conf, and httpd_d.conf), and it will work
with simple include files as well (e.g., "virtual_hosts.conf").
Log files found in Include'ed configuration files will also be
processed. savelogs has internal consistency checks to ensure that
logs are not processed twice, neither are configuration files read
twice (thus avoiding those annoying infinite loops).
Example:
% savelogs --apacheconf=/www/conf/httpd.conf --apacheinclude
or in your configuration file:
ApacheConf /www/conf/httpd.conf
ApacheInclude yes
Grouping Log File Directives
Beginning with savelogs version 1.90, you can apply directives to
groups of log files with the <Group> directive in a savelogs
configuration file (the directive has no command-line equivalent). The
following directives apply in a Group block:
period, ext, sep, datefmt, hourly, touch, chown, chmod, clobber, apachehost, log, disabled
As of version 1.90, the following directives are not honored, but may
be at some point in the future:
size, filter, stem, stemhook, stemlink, nolog, apachelog, apachelogexclude
Group settings override any other settings found in the file, but are
applied only to log files found within the Group block.
Example:
ApacheConf /www/conf/httpd.conf
PostMoveHook apachectl restart
Period 30
<Group>
ApacheHost www.sample1.tld
ApacheHost www.sample2.tld
Period 10
Chown roger:staff
</Group>
<Group>
ApacheHost www.sample3.tld
ApacheHost www.sample4.tld
Chown fonzie
</Group>
Explanation:
The first Group block will rotate Apache logs for "sample1.tld" and
"sample2.tld", but it will only rotate 10 days worth of logs, since the
Period directive inside the block is set to 10.
The second Group block will rotate Apache logs for "sample3.tld" and
"sample4.tld", and the full 30 day Period setting will apply to them.
Both groups will search /www/conf/httpd.conf and restart with
"apachectl restart".
Group blocks have a special directive Disabled that when set to true,
will skip this block. Griping is also disabled when Disabled is set to
true:
## this group will be completely skipped.
<Group>
Disabled 1
ApacheHost www.foo.com
Chown roberto
</Group>
The purpose of the Disabled directive is to prevent ApacheConf from
reverting to its behavior in the absence of any ApacheHost directives
(i.e., it will parse the entire configuration file and process all
hosts found). If this is what you desire, then remove or comment out
your Group blocks instead of Disabled'ing them.
Known bugs related to the Group directive as of 1.91:
If a log file name found inside of a Group block also matches a NoLog
directive outside of the Group block, it will be skipped as if it were
outside the block.
If a log referenced inside of a Group block is processed outside of it
also (found by some other directive, for example), the first one found
will have precedence and the settings inside the Group block may be
ignored.
If there are many files being processed, it is possible that in rare
circumstances (namely, when processing spans midnight) that one set of
logs may be named using one date and another set using another date.
This only applies to non-periodic rotation.
Moving (Renaming) Files
touch
Touches the original file, creating it if necessary. This is useful
for programs that log in "append-only" mode and do not create the
log file if it is missing. Once savelogs has renamed a log file,
touch will create the file if it does not exist, or reset the
timestamp if it does.
size=kbytes
Logs smaller than kbytes will not be included in any processing.
To override a default setting, specify --size with no arguments on
savelogs command-line.
Size 5000
will skip all log files smaller than 5 megs, regardless of other
settings.
datefmt=string
Allows you to change how dates are formatted using the standard
strftime system call. See strftime(1) for format string options.
The default string is '%y%m%d'.
## renames logs like this: access_log.02-12-25
## Merry Christmas!
DateFmt %y-%m-%d
Some popular options are:
## 20020626 (26 June 2002)
DateFmt %Y%m%d
## 1.Mar.2002
DateFmt %e.%b.%Y
ext=string
Set the filename extension to 'string'. When a file is moved, it is
renamed to the original filename plus the extension you specify. If
no extension is specified, today's date is used in 'yymmdd' format.
Options include today and yesterday.
You may not use ext in a configuration file with a value in
backticks (e.g., the line:
## this directive will not work: don't use it!
Ext `/bin/date`
in a configuration file will not work). Any other (static) value
for ext in a configuration file will work.
See also hourly below for information on how to modify ext even
further. See datefmt above if you want to format your dates
differently. ext is provided chiefly for completeness; its
usefulness is limited except in special circumstances where
savelogs can't offer a reasonable name for your log.
sep=char
The separator character to use when moving files. By default this
character is a dot ('.'). Other favorites are underscore ('_') and
hyphen/minus sign ('-').
Example:
Use an underscore character:
savelogs --sep='_' --process=move /var/log/foo
will rename ~/var/log/foo to ~/var/log/foo_yymmdd (where yymmdd is
today's date).
Use no separator (just concatenate the extension with the
filename):
savelogs --sep= --process=move /var/log/foo
will rename ~/var/log/foo to ~/var/log/fooyymmdd.
hourly
Adds a letter of the alphabet to the back of the filename
extension. This is useful if you are rotating logs several times a
day. For example, if you specified your extension (via ext) as
'foo', any log files rotated in the 10am hour will be named
'log.fook'. At 11am, all logs will be called 'log.fool' and so
forth.
Example:
savelogs /path/to/log_file
This will rename the log file to log_file.yymmdd where yymmdd are
today's year, month, and day of month.
savelogs--hourly /path/to/log_file
If you specify the hourly option (or in your configuration file,
the Hourly directive), the log will be renamed to log_file.yymmddz
where the z represents the current hour as a letter of the alphabet
(0 = a, 1 = b, 2 = c, etc.).
You could also use the datefmt directive to get similar results
(with much more flexibility to boot). If you rotate more often than
once and hour, use the datefmt directive or use period to rotate
logs with a unique number as the extension. period, discussed
below, renames log files with a simple integer: log becomes log.0,
the former log.0 becomes log.1 and so forth.
period[=count]
Renames the file based on a period, which is how frequently you run
savelogs. If you specify period you may also optionally specify a
'count', which is how many log files to save using the period
option:
savelogs --period=8 /var/log/messages
You may also use the count option, which is deprecated for backward
compatibility and some possible future enhancements:
savelogs--period --count=8 /var/log/messages
If you do not specify a count value (either in period or count), a
count of 10 is assumed.
The period option will rename the current log to logfile.0, the log
that was previously named logfile.0 to logfile.1 and so on, much
like newsyslog(8).
When you specify the period option, the process phases move and
compress are assumed. If you also specify a process phase of
filter, that will be honored also.
The period option will override any other sep and ext options
specified, using the default dot ('.') for the separator and an
integer for the extension.
The author also recommends you don't try to mix period named log
files with other log files in the same directory, since savelogs
may not be able to tell which logs are oldest based on the filename
extension and destroy the wrong files. You can safely use any
default savelogs extensions (e.g., the default 'today' or
'yesterday' extensions) or your own extension if your own extension
contain at least one non-digit (0-9) or your own extension has 5 or
more digits. If you meet either of these criteria in your own
extension, you may feel confident about mixing logs.
For those familiar with Unix system adminstration, period works
like newsyslog(8) with the B option specified in the newsyslog
configuration file (the B option tells newsyslog to treat logs as
binary files and not append the status message to the log).
An example use of the period option:
savelogs--touch --period=15 /var/log/maillog /var/log/messages
will move any existing ~/var/log/maillog and ~/var/log/messages to
~/var/log/maillog.0 and ~/var/log/messages.0 and compress them. By
specifying the touch option, the original ~/var/log/maillog and
~/var/log/messages will be 'touched', recreating the files. When
this command is run again, ~/var/log/maillog.0 is moved to
~/var/log/maillog.1 and ~/var/log/maillog is moved to
~/var/log/maillog.0.
count
Limits the number of logs saved using the period option. The
internal default value for count is 10.
If you are using the period option, as of version 1.21 you may now
simply specify the count as part of the period option:
savelogs --period=5 /var/log/messages
postmovehook=command
Runs an arbitrary system command you specify after moving files. If
you are rotating Apache log files, you should use a command that
will tell your web server to close its log file descriptors and re-
open them (e.g., 'restart_apache').
Example:
savelogs --apacheconf=/www/conf/httpd.conf \
--postmovehook='/usr/local/bin/restart_apache'
or even nicer in your configuration file:
PostMoveHook /usr/local/bin/restart_apache
Paths in postmovehook are NOT relative to your home directory, as
most other paths are. You should specify the full path to the file
if the file is not in your environment's $PATH. The exception to
this rule is the $LOG macro which, specified, will automatically be
replaced with the current log file path. See Variables below for
details.
postmovehook is one of two ideal phases to analyze your data before
it is archived away (the other phase is the postfilterhook phase).
Variables
Some internal savelogs variables are available during the
postmovehook phase. These variables are automatically interpolated
by savelogs during execution and are guaranteed to contain some
useful value.
$APACHE_CONF
Contains the full path to the Apache configuration file as
specified with the apacheconf option.
savelogs --apacheconf=/www/conf/httpd.conf \
--postmovehook='touch $APACHE_CONF'
$HOME
Contains the path to your home directory.
savelogs --postmovehook='$HOME/bin/myprogram'
$LOG
Contains the current log file being processed. If you wanted to
run a command on each log file after it is moved, you may enter
that command here (on one line). If the line is really long,
consider putting it into a shell script that "wraps" all of
your options.
PostMoveHook /usr/local/bin/do_something_with_every $LOG
force-pmh
Executes the postmovehook command even if there are no logs to
process. By default, savelogs will not execute the postmovehook
command if there are no logs.
chown=uid:gid
After savelogs has renamed your log(s), you may wish to chown them
to a new user or group. Use the chown option for this. If you don't
specify a user, the user will not change. If you don't specify a
group, the group won't change.
## chown the logs to joe:joegroup
Chown joe:joegroup
## make the log files owned by the group with gid 500, keep the
## current owner
Chown :500
## change the owner to root, leave the group alone
Chown root:
Notice that the colon is necessary at all times.
The chown option executes after the PostMoveHook phase has
completed.
chmod
After savelogs has renamed your log(s), you may wish to change the
permissions of the logfile. Use the chmod option for this.
Permissions should be specified in octal.
## make readable only by the owner
Chmod 0600
The chmod option executes after the PostMoveHook phase has
completed.
stem
Like ext except the stem is used in addition to ext. After the move
and postmovehook phases have completed, savelogs checks to see if
you have defined a stemhook. If a stemhook has been defined, a
symbolic link to the log is made using stem.
As an example, say you were processing ~/var/log/messages. During
the stem phase, savelogs would do this:
messages.today -> messages.<ext>
(where <ext> is the extension you specified or today's date by
default)
Once the stemhook command has executed, the symbolic link (or hard
link or copy, as specified by stemlink) is removed.
The stem related options were added in version 1.28.
stemhook=command
A command to execute, much like postmovehook, except that this
phase is suited for log file analysis tools that require a
predictably named, dead (i.e., no logging is currently being done
to it) log file. urchin and analog are good examples of programs
that require such logs.
An example use for stemhook would be:
% savelogs --stemhook="$HOME/usr/local/urchin/urchin" \
/www/logs/access_log /www/logs/error_log
urchin should be instructed to operate on
/www/logs/access_log.today and /www/logs/error_log.today. urchin
should also be instructed to do nothing to the log files since
we're allowing savelogs to manage them for us. Any changes
resulting from the stemhook command to the log file will occur in
the original log file.
The same variables for postmovehook are available for stemhook.
stemlink=linktype
Specify the type of link that should be made during the stem phase.
The default is symlink. Other options are hard which creates hard
links and copy which creates a copy of the original file. hard
links are useful for stemhook commands that cannot process symbolic
links. If your stemhook command modifies the log file, you may wish
to choose the copy option which will be discarded after the
stemhook command is executed.
Filtering
A filter is simply a program that generates something on STDOUT. They
may be pipelines or other programs.
filter=filter_command
If the filter process option is specified (via the process
directive), you should supply a program to filter your log (via the
filter command), such as egrep or perl (or a pipeline or a shell
script containing your commands, etc.).
If no filter command is given, the filter phase will be skipped
(even if you specify --process=filter as the process directive).
Consider following filter command:
--filter='/usr/bin/egrep -v "/images/" \$LOG'
When savelogs gets to its filter phase, it will open a pipe to the
above command. Output from this command will be saved to a
temporary file, then the temporary file will be renamed to replace
the original log file.
Notice the strange $LOG variable. This is an internal savelogs
variable that refers to the location of the log file savelogs is
currently working on. It is automatically replaced with the right
file during execution.
If you are supplying a filter command on the command-line, the
backslash (\) in front of $LOG is necessary to tell the shell to
not interpolate <$LOG> as a shell variable, but instead pass it
along untouched to savelogs. The backslash is not necessary if you
are specifying a filter directive in the configuration file:
Filter /usr/bin/egrep -v "/images/" $LOG
For the sake of completeness, you can also chain filters via a
pipeline, like this:
Filter egrep -v "/images/" $LOG | egrep -v "(root|cmd)\.exe" -
The final '-' tells egrep to use stdin from the previous pipe for
its input.
postfilterhook=command
Runs an arbitrary system command you specify after filtering files.
See postmovehook for examples, including the Variables section.
force-pfh
Like force-pmh, this forces execution of the postfilterhook command
even if there are no logs (and assuming savelogs has reached this
phase).
Archiving and Compressing
gtar
tar Specifies the location of the tar program to use. This defaults to
whatever it can find on your system. You usually don't need to
modify this option unless your tar or gtar program is not in your
path.
Example:
Gtar /usr/sbin/gtar
If both gtar and tar are specified, gtar will be used.
archive
Specifies the name of the archive to which files will be appended.
This directive is somewhat tricky to understand. Under normal use,
savelogs uses the name of the file being archived as the archive
name. For example, if you were archiving a file named
~/var/log/messages, the name of the archive would be
~/var/log/messages.tar.
If you are archiving multiple files (which is common), each file
will be stored in its own archive by name in the directory where
the file is located. For example, if you had several files located
in ~/var/log, each file would be stored in its own archive named
filename.tar in the ~/var/log directory.
If you want to lump together all files in a particular directory
into one archive, use the archive directive without any path
information:
savelogs --archive=system.tar \
/var/log/messages /var/log/proftpd /var/log/foo \
/www/logs/access_log /www/logs/error_log
This will archive ~/var/log/messages, ~/var/log/proftpd, and
~/var/log/foo in a single file named ~/var/log/system.tar (which
may later be compressed if you've so specified).
~/www/logs/access_log and ~/www/logs/error_log will also be lumped
together in a file called ~/www/logs/system.tar.
If you want to lump together all files for this savelogs session
into one archive, use the archive directive and specify the full
path to the archive:
savelogs --archive=/var/tmp/logs.tar /var/log/messages \
--apacheconf=/www/conf/httpd.conf
This will archive ~/var/log/messages and all log files found in the
Apache configuration file into a single archive named
~/var/tmp/logs.tar (which may later be compressed).
If you wish to place the archive in your home directory, you may be
tempted to just do this:
savelogs --archive=/logs.tar
This won't work. Because of the way savelogs tries to simplify
things relative to your home directory, the leading slash is
dropped and savelogs doesn't find any path information (and
therefore places the archive in where the files are).
To put files in your home directory, preceed the archive command
with a dot-slash:
savelogs --archive=./logs.tar
This won't put the archive in your current working directory, as
some are wont to assume, but in your home directory. savelogs has
no notion of a current working directory because it is always
changing directories from your home to the directory where the log
files are and back.
full-path
Specifies whether files stored in archives are full paths relative
to your home directory or relative paths relative to the directory
in which the file is found.
If full-path is not specified, paths are stored in the tar file
relative to their parent directory.
Example:
savelogs /var/log/messages
will create an archive with the following file in it:
messages
while
savelogs--full-path /var/log/messages
will create an archive with the following file in it:
var/log/messages
When you extract this file later on, the paths will be created for
you if they don't exist, which may not be what you want (but, of
course, it may be what you want which is why we have this
directive).
gzip
compress
uncompress
Specifies the location of the gzip/compress/uncompress binaries for
decompressing files (files ending with '.gz' or '.Z'). Use of
compress and uncompress is deprecated. If your system has a gzip
program in a directory that is not in your $PATH variable, specify
its location with this directive. If gzip and compress are
specified, gzip will be used.
By default (if none of the above options are specified), savelogs
will search for a gzip binary in your path and use it.
clobber
If a compressed archive already exists along side a non-compressed
archive (e.g., archive.tar and archive.tar.gz), and you've
instructed savelogs to compress archive.tar, some compression
programs (like gzip) will ask you for confirmation before
overwriting existing files.
To get around this, savelogs by default enables the 'force' option
on compression programs (usually -f). This way, if you're running
savelogs from a cron job or another method where there is no
controlling terminal, savelogs keeps running.
If you're running savelogs interactively (i.e., from a tty) and
want savelogs to prompt you to overwrite existing compressed files,
specify the noclobber option:
Example:
savelogs--noclobber
or in your configuration file:
Clobber no
EXAMPLES
The author recommends liberal use of the dry-run option when testing
these examples or when making big changes to your configuration file or
command-line options. Doubly-so when you have the delete process option
enabled. There's no 'undelete' for UNIX.
Archiving a single log file
savelogs /path/to/log_file
By default, savelogs will move, and compress a log file. This is its
simplest use. If this command is run daily, the result will be a file
name like the old file with a yymmdd extension. This file will be
compressed.
Nuking logs
You can use savelogs (contrary to its name) to just wipe out log files
and reclaim the disk space. If you've got a couple of files that from
time to time just get too big and there's really no valuable
information in them, do something like this:
savelogs --process=delete /path/to/log_file1 /path/to/log_file2
If you want to nuke all Apache log files, do something like this:
savelogs --process=delete --postmovehook=/usr/local/bin/restart_apache \
--apacheconf=/www/conf/httpd.conf
When you specify only the delete as a process option, no logs are
moved, archived, or compressed. They're just deleted.
Compressing logs daily
Compressing logs daily is easy:
savelogs /var/log/messages
will make:
-rw-r--r-- 1 test vuser 751327 Jul 6 12:48 messages
become:
-rw-r--r-- 1 test vuser 84625 Jul 6 12:48 messages.010706.gz
Compressing logs daily if needed
savelogs --size=5000 /var/log/messages
will only compress the log if the size of the file is 5000 kilobytes (5
megabytes) or larger.
Using periodic log rotation
You want to save 3 days worth of Apache log files and 5 days worth of
system log files. You might try the following lines in your crontab:
1 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
--postmovehook=/usr/local/bin/restart_apache --period=3 \
--apacheconf=/www/conf/httpd.conf
5 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
--period=5 /var/log/messages /var/log/ftp.log
Most crontab files require that no lines wrap, so you'd need to make
sure to keep everything on one line.
Your ~/www/logs directory may look something like this after a week:
access_log
access_log.0.gz
access_log.1.gz
access_log.2.gz
error_log
error_log.0.gz
error_log.1.gz
error_log.2.gz
and your system logs directory:
messages
messages.0.gz
messages.1.gz
messages.2.gz
messages.3.gz
messages.4.gz
ftp.log
ftp.log.0.gz
ftp.log.1.gz
ftp.log.2.gz
ftp.log.3.gz
ftp.log.4.gz
Archiving all logs
Most people want to group their log files in an archive. This makes
storing them and retrieving them later for post-processing simple and
efficient. Your directory tree might look like this:
usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log
usr/local/etc/httpd/logs/bar/error_log
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log
usr/local/etc/httpd/logs/baz/error_log
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log
usr/local/etc/httpd/logs/biz/error_log
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log
usr/local/etc/httpd/logs/buz/error_log
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log
usr/local/etc/httpd/logs/foo/error_log
Issue this command:
savelogs --process=all --apacheconf=/www/conf/httpd.conf
and your directory tree now looks like this:
usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log.tar.gz
usr/local/etc/httpd/logs/bar/error_log.tar.gz
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log.tar.gz
usr/local/etc/httpd/logs/baz/error_log.tar.gz
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log.tar.gz
usr/local/etc/httpd/logs/biz/error_log.tar.gz
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log.tar.gz
usr/local/etc/httpd/logs/buz/error_log.tar.gz
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log.tar.gz
usr/local/etc/httpd/logs/foo/error_log.tar.gz
Inside of each compressed archive is a single file:
access_log.010523
which is the old log renamed with today's date. When you run this
command again (e.g., from a cron job) tomorrow, you'll see the same
list of files above, except that inside each compressed archive is an
additional file:
access_log.010523
access_log.010524
Archiving all logs per directory
Say you want to group together all logs in a single directory in one
archive. Your directory tree might look like this:
usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log
usr/local/etc/httpd/logs/bar/error_log
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log
usr/local/etc/httpd/logs/baz/error_log
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log
usr/local/etc/httpd/logs/biz/error_log
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log
usr/local/etc/httpd/logs/buz/error_log
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log
usr/local/etc/httpd/logs/foo/error_log
Try this:
savelogs --process=all --archive=logs.tar --apacheconf=/www/conf/httpd.conf
and your directory tree now looks like this:
usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/logs.tar.gz
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/logs.tar.gz
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/logs.tar.gz
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/logs.tar.gz
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/logs.tar.gz
Inside each 'logs.tar.gz' are two files:
access_log.010523
error_log.010523
Archiving all logs per session
Say you want to lump all logs in a savelogs session into a single
archive. Specify the archive option with a full path, like this:
savelogs --process=all --archive=/tmp/all_logs.tar \
--apacheconf=/www/conf/httpd.conf /var/log/messages
This will create a single file /tmp/all_logs.tar.gz that contains all
logs found in httpd.conf as well as /var/log/messages.
Filtering logs in place
Maybe your log file fills up with garbage entries that you want to
clean out daily. You can use the filter option to trim your log in
place:
savelogs --process=filter \
--filter='/usr/bin/egrep -v "(root|cmd)\.exe" \$LOG' \
--postfilterhook='/usr/local/bin/restart_apache' \
--apacheconf=/www/conf/httpd.conf
This will clean out some Windows worm requests from your Apache log
files.
BUGS/CAVEATS
· savelogs will not properly interpolate backticks inside the
configuration file. Because backticks are shell operators, they
must be visible to the shell (meaning they have to be done on the
command-line). An example is the ext command where the command-line
option might look like this:
savelogs --ext=`smalldate -m`
there is no corresponding configuration file entry.
· You must obey shell escaping rules when you're doing filter
commands on the command-line. Sometimes these can be tricky. For
example, double quotes will do different things than single quotes
(I recommend single quotes because you can then use the $LOG
variable without big headaches). Using a config file sometimes
helps.
· You can't have multiline commands in the configuration file. This
is on the "to do" list.
WISHLIST/TODO
· Would be nice to use the VirtualHost 'User' directive for the chown
option. Maybe a 'ApacheChown' option.
· Optimize the ApacheHost directive to jump out of the Apache config
parsing once all hosts have been found (can you have a VirtualHost
block appear twice with the same ServerName directive?)
ACKNOWLEDGEMENTS
· Thanks to Jeroen Latour (cpantester@calaquendi.net) for working
with me on getting all the tests to run cleanly on his Cobalt box.
It should now test cleanly on many other platforms because of his
patience.
AUTHOR
Scott Wiersdorf, <scott@perlcode.org>
COPYRIGHT
Copyright (c) 2001-2004 Scott Wiersdorf. All rights reserved.
POD ERRORS
Hey! The above document had some coding errors, which are explained
below:
Around line 2465:
=back doesn't take any parameters, but you said =back 4
perl v5.20.2 2011-10-15 SAVELOGS(1)
