rwflowpack(8) SiLK Tool Suite rwflowpack(8)NAMErwflowpack - Collect flow data and store it in binary SiLK Flow files
SYNOPSISrwflowpack [--input-mode=MODE] INPUT_MODE_SPECIFIC_SWITCHES
[--output-mode=MODE] OUTPUT_MODE_SPECIFIC_SWITCHES
{ --log-destination=DESTINATION
| --log-pathname=FILE_PATH
| --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
[--log-post-rotate=COMMAND] }
[--no-file-locking] [--flush-timeout=VAL]
[--file-cache-size=VAL] [--pack-interfaces]
[--byte-order=ENDIAN] [--compression-method=COMP_METHOD]
[--error-directory=DIR_PATH] [--archive-directory=DIR_PATH]
[--flat-archive] [--post-archive-command=COMMAND]
[--site-config-file=FILENAME] [--log-level=LEVEL]
[--log-sysfacility=NUMBER] [--pidfile=FILE_PATH]
[--no-chdir] [--no-daemon]
To collect flow data over the network or directory polling (default):
rwflowpack [--input-mode=stream] --sensor-configuration=FILE_PATH
[--packing-logic=PLUGIN] [--sensor-name=SENSOR]
[--polling-interval=NUMBER] ...
To collect from local files containing flows created by flowcap(8):
rwflowpack --input-mode=fcfiles --incoming-directory=DIR_PATH
--sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
[--polling-interval=NUMBER] ...
To collect from a single file containing NetFlow v5 PDUs:
rwflowpack --input-mode=pdufile --netflow-file=FILE_PATH
--sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
[--sensor-name=SENSOR] ...
To respool SiLK Flows without modifying the class, type, or sensor:
rwflowpack --input-mode=respool --incoming-directory=DIR_PATH
[--polling-interval=NUMBER] ...
To store the SiLK Flow files on the local machine (default):
rwflowpack ... [--output-mode=local-storage]
--root-directory=DIR_PATH ...
To create incremental files to be processed by rwflowappend(8):
rwflowpack ... --output-mode=incremental-files
--incremental-directory=DIR_PATH ...
To create incremental files to be processed by rwflowappend
(deprecated):
rwflowpack ... --output-mode=sending --sender-directory=DIR_PATH
--incremental-directory=DIR_PATH ...
Help options:
rwflowpack --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
{ --verify-sensor-config | --verify-sensor-config=VERBOSE }
rwflowpack--help
rwflowpack--version
DESCRIPTIONrwflowpack is a daemon that runs as part of the SiLK flow collection
and packing tool-chain. The primary job of rwflowpack is to convert
each incoming flow record to the SiLK Flow format, categorize each
incoming flow record (e.g., as incoming or outgoing), set the sensor
value for the record, and determine which hourly file will ultimately
store the record.
The settings that rwflowpack uses to categorize each flow record are
determined by two textual configuration files and compiled code that is
referred to as packing logic.
The first of the configuration files is silk.conf(5) which specifies
the classes, types, and sensors to use at your site. There are several
different ways to specify the location of this file as detailed in the
"FILES" section below.
The second configuration file is the sensor.conf(5) file, whose
location is specified via the --sensor-configuration switch. This file
contains multiple sensor blocks, where each block contains information
used to categorize flow records captured at that particular sensor.
This file also contains probe blocks which specify how to collect
NetFlow v5, IPFIX, and/or NetFlow v9 flow records, and a mapping of
probes to sensors. See the sensor.conf(5) manual page for details.
The packing logic uses the information from the silk.conf and
sensor.conf files to categorize each flow record. By categorizing a
flow record, we mean determine whether the record is incoming or
outgoing and assign a class/type pair (also called a flowtype) to the
record. The flowtype along with the starting hour of the record and
the sensor where the record was collected form a triple which
determines into which file a flow record is stored. The files that
rwflowpack produces have the form flowType-sensorName_YYYYMMDD.HH where
flowType encodes the class/type pair, sensorName is the sensor where
the flows were collected, and YYYYMMDD.HH is the year, month, day, and
hour when the flow started.
For more information on how rwflowpack categorizes each flow record and
converts data to the SiLK Flow format, see the SiLK Installation
Handbook, the sensor.conf(5) manual page, and the manual page for the
packing logic plug-in that rwflowpack is using (e.g.,
packlogic-twoway(3) is the default, packlogic-generic(3)).
The compiled code for the packing-logic is normally loaded into
rwflowpack as a run-time. (The administrator may choose to compile the
packing logic into rwflowpack when building SiLK. See the SiLK
Installation Handbook for details.) The name of this plug-in is
specified in the packing-logic statement in the silk.conf file. A
different location may be specified using the --packing-logic switch.
Input Modes
There are several ways to input data to rwflowpack, and the method to
use is determined by the --input-mode switch, with "stream" being the
default when the switch is not provided.
"stream"
In "stream" input-mode, rwflowpack processes the probe statements
in the sensor.conf(5) file which specify how rwflowpack is to
capture data from one or more sources. The data is assigned to a
sensor based on the probe-sensor mapping in the sensor.conf file.
rwflowpack then categorizes the records, converts them to the SiLK
Flow format, and writes them to files.
The sources of data that rwflowpack supports are:
· listening for NetFlow v5 packets on a UDP socket
· listening for IPFIX (Internet Protocol Flow Information eXport)
packets on a TCP or a UDP socket
· listening for NetFlow v9 packets on a UDP socket
· listening for sFlow v5 packets on a UDP socket
· polling a directory for files containing NetFlow v5 records
(see the description of the "pdufile" input-mode for the
required format of these files)
· polling a directory for files containing IPFIX records (as
generated by yaf(1))
· polling a directory for files containing SiLK Flow records
(compare to the "respool" input-mode)
Multiple sources may be specified.
Processing of IPFIX, NetFlow v9, or sFlow is only available when
SiLK is compiled with support for libfixbuf-1.6.0 or later.
Processing of sFlow records was added in SiLK 3.9.0. libfixbuf is
available from <http://tools.netsa.cert.org/fixbuf/>.
Configuration of "stream" input-mode is specified in the "Stream
Collection Switches (--input-mode=stream)" section below.
"fcfiles"
Instead of having rwflowpack capture data itself, you may configure
rwflowpack to work in conjunction with one or more flowcap(8)
daemons by specifying "fcfiles" as the input-mode.
In this configuration, each flowcap uses the probe statements in
the sensor.conf(5) file to determine how to collect the data.
flowcap supports the network-based capture methods specified for
the "stream" input-mode---i.e., flowcap does not support directory
polling. flowcap writes the data it captures into small files and
includes the probe name in each file's header.
Typically, the flowcap processes run on separate machines near the
router or flow meter that is generating the records. The
rwsender(8) and rwreceiver(8) daemons are often used to move the
files from the flowcap machines to the rwflowpack machine.
rwflowpack polls a directory for the files created by flowcap.
Once it finds a file, it assigns those records a sensor based on
the probe-sensor mapping in the sensor.conf file, it categorizes
the records, and it writes the records to one or more output files.
Since rwflowpack does not capture data in "fcfiles" input-mode,
rwflowpack does not use the probe statements in the sensor.conf
file, and the statements do not need to be present.
The switches that rwflowpack uses in "fcfiles" input-mode are given
below ("Flowcap Files Collection Switches (--input-mode=fcfiles)").
"pdufile"
Setting the input-mode to "pdufile" tells rwflowpack to read a
single file containing NetFlow v5 PDU records and then exit.
rwflowpack will not become a daemon in this input-mode; instead it
will remain in the foreground, process the NetFlow file, and exit.
The file must be in the format created by NetFlow Collector: The
file's size must be an integer multiple of 1464, where each 1464
byte chunk contains a 24-byte NetFlow v5 header and space for
thirty 48-byte NetFlow records. The number of valid records per
chunk is specified in the chunk's header. (This is also the format
that rwflowpack requires in "stream" input-mode when it is polling
a directory for NetFlow v5 files.)
To convert single PDU file to a stream of SiLK Flow records,
consider using rwpdu2silk(1).
In "pdufile" input-mode, the sensor.conf file must define a sensor
that maps to a probe that uses the read-from-file statement.
However, the argument to that statement is ignored, and the
argument is typically set to /dev/null. The NetFlow v5 file's
location must be specified with the --netflow-file switch. If
sensor.conf contains multiple sensor blocks, the --sensor-name
switch is required to tell rwflowpack which sensor to use.
See the "PDU File Collection Switches (--input-mode=pdufile)"
section below for additional configuration information.
"respool"
Sometimes it is desirable to pull existing SiLK Flow records from
one data repository and use them to create a "mini" data repository
(for example, for testing). The "respool" input-mode is one way to
accomplish this.
In this configuration, rwflowpack polls a directory for SiLK flow
files and it uses the existing class/type pair and sensor values to
determine where to store the flow record. That is, rwflowpack puts
the data into appropriate hourly file, but it does not change any
other settings on the flow records.
To contrast "respool" input-mode with rwflowpack processing SiLK
Flow files in "stream" input-mode: In "respool" input-mode, the
existing class, type, and sensor values are used to determine each
record's destination. In the latter, rwflowpack treats the records
as it would any other newly arrived data, assigning the data to a
sensor and re-categorizing the records to assign a class/type pair
to them.
Since no categorization occurs in "respool" input-mode, the
--sensor-configuration and --packing-logic switches are not
required, and their use will cause rwflowpack to exit with an error
code.
Output Modes
As mentioned above, after rwflowpack categorizes a flow record (that
is, determines the sensor, class/type, and starting hour for the
record), it uses those values to generate the name of the hourly file
that will contain that record, and it writes the record to that file.
In order for the record in that file to be retrievable by rwfilter(1),
the file must be stored in a SiLK data repository, which is a directory
tree of binary SiLK Flow files. The root of this directory tree is
called the SILK_DATA_ROOTDIR. The structure of the tree under the root
is determined by the path-format setting in the silk.conf(5) file.
There are two ways to get the files into the SILK_DATA_ROOTDIR; which
method is used is determined by rwflowpack's --output-mode switch.
This switch supports the following values:
"local-storage"
In "local-storage" output-mode, rwflowpack creates the hourly SiLK
Flow files directly in the data repository, and it writes the
records into these files. rwflowpack uses "local-storage" output-
mode when the --output-mode switch is not provided.
"incremental-files"
When the output-mode is "incremental-files", rwflowpack does not
create hourly data files directly. Instead, rwflowpack creates
smaller files (called incremental files), and rwflowpack relies on
the rwflowappend(8) daemon to combine the incremental files into
hourly files in the final data repository.
In "incremental-files" output-mode, rwflowpack uses a single
destination directory whose location is specified by the
--incremental-directory switch. In this directory, rwflowpack
creates a zero-byte place holder file and a corresponding working
file. The name of the place holder file has a unique, random
suffix, and the name of the working file is a dot (".") followed by
the name of the place holder file.
rwflowpack first writes records to the working files. Periodically
(as determined by the value specified to --flush-timeout),
rwflowpack closes all the working files and moves them over the
place holder files. Once rwflowpack has closed and moved a working
file, it no longer takes responsibility for it.
The rwflowappend process can poll the incremental-directory itself,
or another process can poll that directory and pass the files to
rwflowappend. If rwflowpack and rwflowappend are on different
machines, an rwsender(8)/rwreceiver(8) pair can move the files
between the machines.
Additional reasons one may want to use "incremental-files" output-
mode are to support having multiple data repositories or to allow
additional processing of the SiLK Flow records (such as by the
Analysis Pipeline
(<http://tools.netsa.cert.org/analysis-pipeline/>).
Note: This output-mode was introduced in SiLK 3.6.0. This mode is
similar to the "sending" output-mode and is meant to replace it.
In "incremental-files" output-mode, all writing occurs within the
incremental-directory, while the "sending" output-mode uses two
directories.
"sending"
This output-mode works similarly to "incremental-files". The
difference is that when rwflowpack flushes the open files, it moves
the files from the incremental-directory and to the directory
specified by the --sender-directory switch. Once a file is moved
to the sender-directory, rwflowpack no longer takes responsibility
for it.
As of SiLK 3.6.0, the "sending" output-mode is deprecated.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
Input and Output Mode
As described in the "Input Modes" section above, rwflowpack has
multiple ways of getting data, and that data may be stored in one of
two methods (c.f., "Output Modes"). Choosing the modes and configuring
each mode are described in the sections below.
--input-mode=MODE
Determine how rwflowpack will gather data. The default input MODE
is "stream". The available input-modes are "stream", "fcfiles",
"pdufile", and "respool".
--output-mode=MODE
Determines what rwflowpack will do with the data as it is packed
into SiLK binary files. The default output MODE is
"local-storage". The available output-modes are "local-storage",
"incremental-files" and "sending".
Stream Collection Switches (--input-mode=stream)
In "stream" input-mode, rwflowpack uses the probe statements in the
sensor.conf file to capture data, and then rwflowpack categorizes the
data. The "stream" input-mode is the default when the --input-mode
switch is not provided. This input-mode accepts the following
switches; the --sensor-configuration switch is required, and all other
switches are optional.
--sensor-configuration=FILE_PATH
Give the path to the configuration file that specifies how
rwflowpack is to capture data and that tells rwflowpack whether a
record represents an incoming or outgoing flow. The complete
syntax of the configuration file is described in the sensor.conf(5)
manual page; see also the SiLK Installation Handbook.
--packing-logic=PLUGIN
Specify the plug-in that rwflowpack should load, where the plug-in
provides functions that determine into which class and type each
flow record will be categorized and the format of the files that
rwflowpack will write. When SiLK has been configured with hard-
coded packing logic (i.e., when --enable-packing-logic was
specified to the configure script), this switch will not be present
on rwflowpack. A default value for this switch may be specified in
the silk.conf(5) site configuration file (see the description of
the --site-config-file switch). When PLUGIN does not contain a
slash ("/"), rwflowpack will attempt to find a file named PLUGIN in
the directories listed in the "FILES" section. If rwflowpack finds
the file, it uses that path. If PLUGIN contains a slash or if
rwflowpack does not find the file, rwflowpack relies on your
operating system's dlopen(3) call to find the file. When the
SILK_PLUGIN_DEBUG environment variable is non-empty, rwflowpack
prints status messages to the standard error as it attempts to find
and open each of its plug-ins. rwflowpack will exit if it does not
have access to the packing logic functions.
--sensor-name=SENSOR
Cause rwflowpack to ignore all probes in the sensor configuration
file except the probes for SENSOR. Only data for this sensor will
be collected. This allows a common configuration file to be used
by multiple rwflowpack invocations, yet also allow each rwflowpack
instance only collect data for a single sensor. There must be a
sensor definition for SENSOR in the configuration file. When this
switch is not present, rwflowpack will collect and pack data for
all sensors.
--polling-interval=NUMBER
Specify the number of seconds rwflowpack will wait between queries
of the "poll-directory"s. This defaults to 15 seconds.
Flowcap Files Collection Switches (--input-mode=fcfiles)
As described in the "Input Modes" section above, in "fcfiles" input-
mode, rwflowpack processes files created by the flowcap(8) daemon. In
"fcfiles" input-mode, the --sensor-configuration and
--incoming-directory switches are required.
--sensor-configuration=FILE_PATH
Give the path to the configuration file that rwflowpack will
consult to determine whether a record represents an incoming or
outgoing flow. The complete syntax of the configuration file is
described in the sensor.conf(5) manual page; see also the SiLK
Installation Handbook.
--incoming-directory=DIR_PATH
Name the full path of the directory which rwflowpack will monitor
for files created by flowcap. Once processed by rwflowpack, files
are moved from this directory to the archive-directory, if it has
been specified.
--packing-logic=PLUGIN
Specify the plug-in that rwflowpack should load for the packing
logic. For more detail, see the description above.
--polling-interval=NUMBER
Specify the number of seconds rwflowpack will wait between polls of
the incoming-directory for new files created by flowcap. If not
given, the default value is 15 seconds.
PDU File Collection Switches (--input-mode=pdufile)
In this input-mode, rwflowpack stays in the foreground, processes a
single file of NetFlow v5 data, and exits. The --sensor-configuration
and --netflow-file switches are required. The --sensor-name switch is
also required when the sensor.conf file contains more than one sensor.
The following switches are available in "pdufile" input-mode:
--sensor-configuration=FILE_PATH
Give the path to the configuration file that rwflowpack will
consult to determine whether a record represents an incoming or
outgoing flow.
--netflow-file=FILE_PATH
Name the full path of the file from which rwflowpack reads NetFlow
v5 PDUs. This switch is required in PDU File input-mode.
--sensor-name=SENSOR
Cause rwflowpack to ignore all probes in the sensor configuration
file except the probes for SENSOR. There must be a sensor
definition for SENSOR in the configuration file. This switch is
required in this input-mode unless the sensor.conf file only
defines a single sensor.
--packing-logic=PLUGIN
Specify the plug-in that rwflowpack should load for the packing
logic. For more detail, see the description of this switch in the
"stream" input-mode.
Respooling Switches (--input-mode=respool)
When the --input-mode="respool" switch is provided, rwflowpack polls a
directory for SiLK Flow files, and writes the records it finds into new
hourly files, leaving the sensor and class/type values unchanged in the
records. (See "Input Modes" above for additional details.) The first
of the following switches is required:
--incoming-directory=DIR_PATH
Name the full path of the directory which rwflowpack will monitor
for SiLK Flow files. Once processed by rwflowpack, files are moved
from this directory to the archive-directory, if it has been
specified.
--polling-interval=NUMBER
Specify the number of seconds rwflowpack will wait between polls of
the incoming-directory. If not given, the default value is 15
seconds.
Local Storage Switches (--output-mode=local-storage)
In "local-storage" output-mode, rwflowpack stores SiLK Flow records
directly in the data repository. This is the default when the
--output-mode switch is not provided. When operating in this output-
mode, the following switch is required:
--root-directory=DIR_PATH
Name the full path of the SILK_DATA_ROOTDIR; that is, the directory
under which the files containing the packed SiLK Flow records will
be stored. rwflowpack will create subdirectories below DIR_PATH
based on the data received and the path-format setting in the
silk.conf(5) file.
Incremental-Files Switches (--output-mode=incremental-files)
As described in the "Output Modes" section above, the
"incremental-files" output-mode tells rwflowpack to write incremental-
files. The rwflowappend(8) daemon combines these incremental-files to
create a SiLK data repository. When running in "incremental-files"
output-mode, the following switch must be provided:
--incremental-directory=DIR_PATH
Name the full path of the destination directory where incremental-
files are both created and stored to await action by another
process such as rwflowappend or rwsender. It is recommended that
only a single rwflowpack process write to this directory.
Sending Mode Switches (--output-mode=sending)
This output-mode is deprecated as of SiLK 3.6.0. This output-mode
works similarly to the "incremental-files" output-mode, except
rwflowpack moves the files to a second directory periodically. Both
the following switches are required in "sending" output-mode:
--incremental-directory=DIR_PATH
Name the full path of the working directory under which
incremental-files are initially created. Periodically (as
determined by the --flush-timeout switch), rwflowpack closes the
files in this directory and moves them to the sender-directory. An
rwflowpack invocation assumes it has full control over the files in
this directory. When rwflowpack starts, any files in this
directory are moved to the sender-directory.
--sender-directory=DIR_PATH
Name the full path of the destination directory where incremental-
files are moved to await action by another process such as
rwflowappend or rwsender. Once rwflowpack moves files to this
directory, it no longer takes responsibility for them. The other
process (e.g., rwsender) is responsible for removing files from
this directory. Multiple rwflowpack invocations may use a single
sender-directory.
General Configuration
The following switches are optional:
--no-file-locking
Do not use advisory write locks. Normally, rwflowpack obtains a
write lock on an data file prior to writing records to it. The
write lock prevents two instances of rwflowpack from writing to the
same data file simultaneously. However, attempting to use a write
lock on some file systems causes rwflowpack to exit with an error,
and this switch can be use when writing data to these file systems.
--flush-timeout=VAL
Set the timeout for flushing any in-memory records to disk to VAL
seconds. If not specified, the default is 2 minutes (120 seconds).
When the output-mode is "local-storage", this value specifies how
often the files are flushed to disk to ensure that any records in
memory are written to disk. When using the "incremental-files" or
"sending" output-mode, this value specifies how often to close and
move the incremental files. See the "Output Modes" section for
details.
--file-cache-size=VAL
Set the maximum number of data files to have open for writing at
any one time to VAL. If not specified, the default is 128 files.
The minimum file cache size is 4. This switch also determines how
many files rwflowpack will read from simultaneously when using
probes that poll directories for files (see sensor.conf(5)). The
maximum number of input files open at any one time is limited to
one eighth of VAL (with a minimum of 2), and the number of
directory polling operations to perform simultaneously is limited
to one sixteenth of VAL (minimum is 1).
--pack-interfaces
Allow one to override the default file output formats of the packed
SiLK Flow files that rwflowpack writes. When this switch is
present, rwflowpack writes additional information into the packed
files: the router's SNMP input and output interfaces and the next-
hop IP address. (When the sensor.conf file contains an
"interface-values" attribute whose value is "vlan", the input and
output fields contain the vlan IDs instead of SNMP interface
values.) The extra data produced by this switch is useful for
determining why traffic is being stored in certain files. Note
that this switch will only affect newly created files. New records
will always be appended to an existing file in the file's current
output format to maintain file integrity.
--byte-order=ENDIAN
Set the byte order for newly created SiLK Flow files. When
appending records to an existing file, the byte order of the file
is maintained. The argument is one of the following:
"native"
Use the byte order of the machine where rwflowpack is running.
This is the default.
"big"
Use network byte order (big endian) for the flow files.
"little"
Write the flow files in little endian format.
--compression-method=COMP_METHOD
Specify how to compress newly created files. When this switch is
not given, newly created files are compressed using the default
chosen when SiLK was compiled. When appending records to an
existing file, the compression method of the file is maintained.
The valid values for COMP_METHOD are determined by which external
libraries were found when SiLK was compiled. To see the available
compression methods and the default method, use the --help or
--version switch. SiLK can support the following COMP_METHOD values
when the required libraries are available.
none
Do not compress the output using an external library.
zlib
Use the zlib(3) library for compressing the output. Using zlib
produces the smallest output files at the cost of speed.
lzo1x
Use the lzo1x algorithm from the LZO real time compression
library for compression. This compression provides good
compression with less memory and CPU overhead.
best
Use lzo1x if available, otherwise use zlib.
--site-config-file=FILENAME
Read the SiLK site configuration from the named file FILENAME.
When this switch is not provided, rwflowpack searches for the site
configuration file in the locations specified in the "FILES"
section.
Disposal of Input Flow Files
The following switches determine how rwflowpack handles input files
once it has processed them. These switches have no effect when
rwflowpack reads all of its data directly from the network. Otherwise,
the switches affect the named --netflow-file in "pdufile" input-mode,
the files read from the --incoming-directory in "fcfiles" and "respool"
input-mode, and files read from probes that have a "poll-directory"
statement (c.f. sensor.conf(5)) in "stream" input-mode.
--error-directory=DIR_PATH
Move input files that cannot be opened, have an unexpected format,
contain an unrecognized probe name in "fcfiles" input-mode, or are
not successfully processed into the directory DIR_PATH. DIR_PATH
must be a complete directory path. If this switch is not provided,
problem files remain in place and cause rwflowpack to exit.
--archive-directory=DIR_PATH
Move input files that rwflowpack processes successfully into the
directory DIR_PATH. DIR_PATH must be a complete directory path.
When this switch is not provided and the input-mode is "pdufile",
the original NetFlow source file is not modified, moved, or
deleted. In all other input-modes, no --archive-directory switch
causes rwflowpack to delete each input file after successfully
processing it. When the --flat-archive switch is also provided,
incoming files are moved into the top of DIR_PATH; when
--flat-archive is not given, each file is moved to a subdirectory
based on the current UTC time: DIR_PATH/YEAR/MONTH/DAY/HOUR/.
Removing files from the archive-directory is not the job of
rwflowpack; the system administrator should implement a separate
process to clean this directory. This switch is required when the
--post-archive-command switch is present.
--flat-archive
When archiving input files via the --archive-directory switch, move
the files into the top of the archive-directory, not into
subdirectories of the archive-directory. This switch has no effect
if --archive-directory is not also specified. This switch can be
used to allow another process to watch for new files appearing in
the archive-directory.
--post-archive-command=COMMAND
Run COMMAND on each input file after rwflowpack has successfully
processed the file and moved the file into the archive-directory.
Each occurrence of the string %s in COMMAND will be replaced with
the full path to the input file in the archive-directory, and each
occurrence of "%%" will be replaced with "%". If any other
character follows "%", rwflowpack exits with an error. When using
this feature, the --archive-directory switch must be specified.
See also the rwpollexec(8) daemon.
Logging and Daemon Configuration
One of the following mutually-exclusive switches is required:
--log-destination=DESTINATION
Specify the destination where logging messages are written. When
DESTINATION begins with a slash "/", it is treated as a file system
path and all log messages are written to that file; there is no log
rotation. When DESTINATION does not begin with "/", it must be one
of the following strings:
"none"
Messages are not written anywhere.
"stdout"
Messages are written to the standard output.
"stderr"
Messages are written to the standard error.
"syslog"
Messages are written using the syslog(3) facility.
"both"
Messages are written to the syslog facility and to the standard
error (this option is not available on all platforms).
--log-directory=DIR_PATH
Use DIR_PATH as the directory where the log files are written.
DIR_PATH must be a complete directory path. The log files have the
form
DIR_PATH/LOG_BASENAME-YYYYMMDD.log
where YYYYMMDD is the current date and LOG_BASENAME is the
application name or the value passed to the --log-basename switch
when provided. The log files are rotated: At midnight local time,
a new log is opened, the previous file is closed, and the command
specified by --log-post-rotate is invoked on the previous day's log
file. (Old log files are not removed by rwflowpack; the
administrator should use another tool to remove them.) When this
switch is provided, a process-ID file (PID) is also written in this
directory unless the --pidfile switch is provided.
--log-pathname=FILE_PATH
Use FILE_PATH as the complete path to the log file. The log file
is not rotated.
The following set of switches is optional:
--log-level=LEVEL
Set the severity of messages that will be logged. The levels from
most severe to least are: "emerg", "alert", "crit", "err",
"warning", "notice", "info", "debug". The default is "info".
--log-sysfacility=NUMBER
Set the facility that syslog(3) uses for logging messages. This
switch takes a number as an argument. The default is a value that
corresponds to "LOG_USER" on the system where rwflowpack is
running. This switch produces an error unless
--log-destination=syslog is specified.
--log-basename=LOG_BASENAME
Use LOG_BASENAME in place of the application name in the name of
log files in the log directory. See the description of the
--log-directory switch. This switch does not affect the name of
the process-ID file.
--log-post-rotate=COMMAND
Run COMMAND on the previous day's log file after log rotation.
When this switch is not specified, the previous day's log file is
compressed with gzip(1). When the switch is specified and COMMAND
is the empty string, no action is taken on the log file. Each
occurrence of the string %s in COMMAND will be replaced with the
full path to the log file, and each occurrence of "%%" will be
replaced with "%". If any other character follows "%", rwflowpack
exits with an error. Specifying this switch without also using
--log-directory is an error.
--pidfile=FILE_PATH
Set the complete path to the file in which rwflowpack writes its
process ID (PID) when it is running as a daemon. No PID file is
written when --no-daemon is given. When this switch is not
present, no PID file is written unless the --log-directory switch
is specified, in which case the PID is written to
LOGPATH/rwflowpack.pid.
--no-chdir
Do not change directory to the root directory. When rwflowpack
becomes a daemon process, it changes its current directory to the
root directory so as to avoid potentially running on a mounted file
system. Specifying --no-chdir prevents this behavior, which may be
useful during debugging. The application does not change its
directory when --no-daemon is given.
--no-daemon
Force rwflowpack to run in the foreground---it does not become a
daemon process. This may be useful during debugging.
Help Options
--verify-sensor-config
--verify-sensor-config=VERBOSE
Verify that the syntax of the sensor configuration file is correct
and then exit rwflowpack. If the file is incorrect or if it does
not define any sensors, an error message is printed and rwflowpack
exits abnormally. If the file is correct and no argument is
provided to the --verify-sensor-config switch, rwflowpack simply
exits with status 0. If an argument (other than the empty string
and 0) is provided to the switch, the names of the probes and
sensors found in the sensor configuration file are printed to the
standard output, and then rwflowpack exits.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
ENVIRONMENT
SILK_IPFIX_PRINT_TEMPLATES
When set to 1, rwflowpack writes messages to the log file
describing each IPFIX and NetFlow v9 template it receives. The
first message includes the domain, the template identifier, the
number of information elements in the template, and the name of
this environment variable. Next, a message is printed for each
information element in the template where the message contains the
domain id, the template id, and the element's position in the
template, length in octets, numeric information element identifier,
and name. For elements defined by a private enterprise, the IE
number has two parts: the private enterprise number and the
information element number, separated by a slash (/). (Requires
libfixbuf 1.4.0 or later.) Since SiLK 3.8.2.
SILK_LIBFIXBUF_SUPPRESS_WARNINGS
When set to 1, rwflowpack disables all warning messages generated
by libfixbuf. These warning messages include out-of-sequence
packets, data records not having a corresponding template, record
count discrepancies, and issues decoding list elements. Since SiLK
3.10.0.
SILK_CONFIG_FILE
This environment variable is used as the value for the
--site-config-file when that switch is not provided.
SILK_DATA_ROOTDIR
This environment variable specifies the root directory of data
repository. When the output-mode is "sending", rwflowpack may use
this environment variable when searching for the SiLK site
configuration file. See the "FILES" section for details.
SILK_PATH
This environment variable gives the root of the install tree. When
searching for configuration files, rwflowpack may use this
environment variable. See the "FILES" section for details.
SILK_PLUGIN_DEBUG
When set to 1, rwflowpack print status messages to the standard
error as it tries to open the packing logic plug-in.
FILES
sensor.conf
The location of this file must be specified by the
--sensor-configuration switch. This file specifies probe blocks
that tell rwflowpack how to capture data when the --input-mode is
"stream". The file also contains sensor blocks that map probes to
sensors and that the packing-logic uses to determine the category
of each flow record. The syntax of this file is described in the
sensor.conf(5) manual page.
${SILK_CONFIG_FILE}
ROOT_DIRECTORY/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/local/share/silk/silk.conf
/usr/local/share/silk.conf
Possible locations for the SiLK site configuration file which are
checked when the --site-config-file switch is not provided. When
rwflowpack is running in "local-storage" output-mode,
ROOT_DIRECTORY/ is the directory specified to the --root-directory
switch. When the output-mode is "sending", ROOT_DIRECTORY/ is
either the value specified in the SILK_DATA_ROOTDIR environment
variable or the default data repository directory compiled into
rwflowpack (/data).
${SILK_PATH}/lib64/silk/
${SILK_PATH}/lib64/
${SILK_PATH}/lib/silk/
${SILK_PATH}/lib/
/usr/local/lib64/silk/
/usr/local/lib64/
/usr/local/lib/silk/
/usr/local/lib/
Directories that rwflowpack checks when attempting to load the
packing-logic plug-in.
SEE ALSO
SiLK Installation Handbook, sensor.conf(5), silk.conf(5),
packlogic-twoway(3), packlogic-generic(3), flowcap(8), rwfilter(1),
rwflowappend(8), rwreceiver(8), rwsender(8), rwpollexec(8),
rwpdu2silk(1), rwpackchecker(8), silk(7), gzip(1), yaf(1), dlopen(3),
zlib(3), syslog(3)NOTES
As SiLK 3.6.0, the "incremental-files" output-mode should be used in
place of the "sending" output-mode that existed in prior releases of
rwflowpack. See "Output Modes" for details.
For administrators that use the "sending" output-mode in SiLK 3.5 or
older and upgrade to SiLK 3.6 or later: Any incremental files that the
older version of rwflowpack leaves in the incremental-directory will
not be moved to the sender-directory by the new version of rwflowpack.
Those files will need to be moved by hand.
rwflowpack does not check the integrity of the data file before
appending records to it.
When the disk becomes full or other write errors occur, rwflowpack may
leave partially written records or partially written compressed blocks
in the files it has open. For each file where a partially written
compressed block remains and additional compressed blocks are appended,
the newly appended compressed blocks are not readable by other SiLK
tools. For each file where a partially written record remains and
additional records are appended, other SiLK tools will read the
unaligned data as if it were aligned and produce garbage records.
Partially writes may also occur if rwflowpack is suddenly killed (e.g.,
by "kill -9").
When a write error occurs, rwflowpack may leave a zero byte file in the
data repository. Such files do affect the exit status of rwfilter(1),
though rwfilter warns about being unable to read the header from the
file.
rwflowpack obtains an advisory write lock on the hourly file it is
writing, allowing multiple rwflowpack processes to write to the same
hourly file. File locking may be disabled by using the
--no-file-locking switch. If this switch is enabled, the administrator
must ensure that multiple rwflowpack processes do not attempt to write
to the same hourly file simultaneously.
SiLK 3.11.0.1 2016-02-19 rwflowpack(8)
