sensor.conf(5) SiLK Tool Suite sensor.conf(5)NAMEsensor.conf - Sensor Configuration file for rwflowpack and flowcap
DESCRIPTION
As part of collecting flow data, the rwflowpack(8) and flowcap(8)
daemons need to know what type of data they are collecting and how to
collect it (e.g., listen on 10000/udp for NetFlow v5; listen on
4740/tcp for IPFIX). In addition, the rwflowpack daemon needs
information on how to categorize the flow: for example, to label the
flows collected at a border router as incoming or outgoing. The Sensor
Configuration file, sensor.conf, contains this information, and this
manual page describes the syntax of the file (see "SYNTAX" below) and
provides some example configurations (see "EXAMPLES").
The sensor.conf file may have any name, and it may reside in any
location. The name and location of the file is specified by the
--sensor-configuration switch to rwflowpack and flowcap.
The Sensor Configuration File defines the following concepts:
probe
A probe specifies a source for flow data. The source could be a
port on which flowcap or rwflowpack collects NetFlow or IPFIX data
from a flow generator such as a router or the yaf software
(<http://tools.netsa.cert.org/yaf/>). In rwflowpack, the source
can be a directory to periodically poll for files containing
NetFlow v5 PDUs, IPFIX records, or SiLK Flow records. When
defining a probe, you must specify a unique name for the probe and
the probe's type.
group
A group is a named list that contains one of the following: CIDR
blocks, the names of IPset files, or integers representing SNMP
interfaces or VLAN identifiers. The use of groups is optional; the
primary purpose of a group is to allow the administrator to specify
a commonly used list (such as the IP space of the network being
monitored) in a single location.
sensor
A sensor represents a logical collection point for the purposes of
analysis. The sensor contains configuration values that rwflowpack
uses to categorize each flow record depending on how the record
moves between networks at the collection point. Since the sensors
and the categories (known as flowtypes or as class/type pairs) are
also used for analysis, they are defined in the Site Configuration
file, described in silk.conf(5). The Sensor Configuration file
maps sensors to probes and specifies the rules required to
categorize the data. Usually one sensor corresponds to one probe;
however, a sensor may be comprised of multiple probes, or the flow
data collected at a single probe may be handled by multiple
sensors.
The next section of this manual page describes the syntax of the
sensor.conf file.
Using the syntax to configure a sensor requires knowledge of the
packing logic that rwflowpack is using. The packing logic is the set
of rules that rwflowpack uses to assign a flowtype to each record it
processes. The default packing logic is for the twoway site, which is
described in the packlogic-twoway(3) manual page. Additional packing
logic rules are available (e.g., packlogic-generic(3)).
The last major section of this document is "EXAMPLES" where several
common configurations are shown. These examples assume rwflowpack is
using the packing logic from the twoway site.
SYNTAX
When parsing the Sensor Configuration file, blank lines are ignored.
At any location in a line, the character "#" indicates the beginning of
a comment, which continues to the end of the line. These comments are
ignored.
All other lines begin with optional leading whitespace, a command name,
and one or more arguments to the command. Command names are a sequence
of non-whitespace characters, not including the character "#".
Arguments are textual atoms: any sequence of non-whitespace, non-"#"
characters, including numerals and punctuation.
There are four contexts for commands: top-level, probe block, group
block, and sensor block. The probe block, group block, and sensor
block contexts are used to describe individual features of probes,
groups, and sensors, respectively.
The valid commands for each context are described below.
Top-Level Commands
In addition to the commands to begin a probe, group, or sensor block,
the top-level context supports the following command:
include "path"
The include command is used to include the contents of another file
whose location is path. This may be used to separate large
configurations into logical units.
Probe Block
With the exception of the probe command, the commands listed below are
accepted within the probe context. Note that one and only one of
listen-on-port, listen-on-unix-socket, read-from-file, or poll-
directory must be specified.
probe probe-name probe-type
The probe command is used in the top-level context to begin a new
probe block which continues to the end probe command. The
arguments to the probe command are the name of the probe being
defined and the probe type. The probe-name must be unique among
all probes. It must begin with a letter, and it may not contain
whitespace characters or the slash character ("/"). When a probe
is associated with a single sensor, it is good practice to give the
probe the same name as the sensor. The probe-type must be one of
the following:
netflow-v5
This probe processes NetFlow v5 protocol data units (PDU) that
the daemon reads from a UDP port or from a file. NetFlow may
be generated by a router or by software that reads packet
capture (pcap(3)) data and generates NetFlow v5 records.
netflow
This is an alias for netflow-v5 for backwards compatibility.
This alias is deprecated, and it may be removed in a future
release.
ipfix
An IPFIX probe processes Internal Protocol Flow Information
eXchange records that the daemon reads over the network from an
IPFIX source such as yaf(1). An IPFIX probe can also poll a
directory for files generated by the yaf program. To support
IPFIX probes, SiLK must be built with support for the external
library libfixbuf, version 1.6.0 or later. Both yaf and
libfixbuf are available from <http://tools.netsa.cert.org/>.
netflow-v9
This probe processes NetFlow v9 protocol data units (PDU) that
the daemon reads from a UDP port from a router. To support
NetFlow v9 probes, SiLK must be built with support for the
external library libfixbuf, version 1.6.0 or later.
sflow
This probe processes sFlow v5 records that the daemon reads
from a UDP port. To support sFlow probes, SiLK must be built
with support for the external library libfixbuf, version 1.6.0
or later. Since SiLK 3.9.0.
silk
A SiLK probe processes the records contained in SiLK Flow files
created by previous invocations of rwflowpack. The flows will
be completely re-packed, as if they were just received over the
network. The sensor and flowtype values in each flow will be
ignored. Note that SiLK usually removes the SNMP interfaces
from its flow records, and it is likely that you will be unable
to use the SNMP interfaces to pack the flows.
end probe
The end probe command ends the definition of a probe. Following an
end probe command, top-level commands are again accepted.
listen-on-port port
This command specifies the network port where the probe should
collect flow data. When listening to NetFlow from a Cisco router,
this is the port that was specified to the Cisco ISO command
ip flow-export [ip-address] [port]
When listening to IPFIX data from yaf, this is the value specified
to yaf's --ipfix-port switch.
protocol { tcp | udp }
This command, required when listen-on-port is given, specifies
whether the port is a "tcp" or "udp" port. IPFIX probes support
both types; the only permitted value for all other probe types is
"udp". When listening to IPFIX data from yaf, this is the value
specified to yaf's --ipfix switch.
accept-from-host host-name [host-name...]
This optional command specifies the IPs or names of the hosts that
are allowed to connect to the port where the probe is listening.
When this command is not present, any host may connect. The
command may only be specified when the listen-on-port command is
also present. When listening for NetFlow, this attribute would be
the IP address of the router as seen from the machine running
rwflowpack or flowcap. (Prior to SiLK 3.10.1, the accept-from-host
command accepted only a single argument.)
listen-as-host host-name
This optional command is used on a multi-homed machine to specify
the address the probe should listen on (bind(2) to). Its value is
the name of the host or its IP address. If not present, the
program will listen on all the machine's addresses. The command
may only be specified when the listen-on-port command is also
present. For listening to NetFlow, the value would be the ip-
address that was specified to the Cisco ISO command
ip flow-export [ip-address] [port]
listen-on-unix-socket path-to-unix-socket
The value contains the path name to a UNIX domain socket where the
flow generator writes its data. The parent directory of path-to-
unix-socket must exist.
poll-directory directory-path
When this command is given, rwflowpack will periodically poll the
directory-path to look for files to process. flowcap will exit
with an error if you attempt to use probes that contain this
command since flowcap does not support reading data from files.
When polling the directory, zero length files and files whose name
begin with a dot (".") are ignored. This command may be used with
the following probe types:
· For SiLK probes, each file must be a valid SiLK Flow file.
· IPFIX probes can process files created by the yaf program.
· A NetFlow v5 probe will process files containing NetFlow v5
PDUs. The format of these files is specified in the
description of the read-from-file command.
read-from-file dummy-value
When this command is given, rwflowpack will read NetFlow v5 records
from the file specified by the --netflow-file command line switch.
The value to the read-from-file command is completely ignored, and
we recommend you use "/dev/null" as the value. flowcap will exit
with an error if you attempt to use probes that contain this
command since flowcap does not support reading data from files.
The format of a NetFlow v5 file is that the file's length should be
an integer multiple of 1464 bytes, where 1464 is the maximum length
of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte
NetFlow v5 header and space for thirty 48-byte flow records, even
if fewer NetFlow records are valid. rwflowpack will accept NetFlow
v5 files that have been compressed with the gzip(1) program.
log-flags { none | { all | bad | default | firewall-event | missing |
record-timestamps | sampling | ... } }
This optional command gives specific logging instructions for the
probe. If you wish to reduce the verbosity of the log, you may use
the log-flag command to adjust the information logged. The
possible values are:
all Log everything.
bad Write messages about an individual NetFlow v5 record where the
packet or octet count is zero, the packet count is larger than
the octet count, or the duration of the flow is larger than 45
days.
default
Enable the following values: bad, missing, sampling. This is
the default value. Since SiLK 3.10.0. (Prior to SiLK 3.10.0,
all was the default.)
firewall-event
When the "firewall-event" quirks flag is set and the probe is
processing NetFlow v9 or IPFIX, write messages about records
that are ignored because the firewall event information element
on the record is something other than flow deleted or flow
denied. Since SiLK 3.8.1.
missing
Examine the sequence numbers of NetFlow v5 packets and write
messages about missing and out-of-sequence packets. (You may
suppress messages regarding out-of-sequence NetFlow v9 or IPFIX
packets for all probes by setting the
SILK_LIBFIXBUF_SUPPRESS_WARNINGS environment variable.)
none
Log nothing. It is an error to combine this value with any
other.
record-timestamps
Log the timestamps that appear on each record. This produces a
lot of output, and it is primarily used for debugging. Since
SiLK 3.10.0.
sampling
Write messages constructed by parsing the NetFlow v9 Options
Templates that specify the sampling algorithm (when
samplingAlgorithm and samplingInterval IEs are present) or flow
sampler mode (when flowSamplerMode and
flowSamplerRandomInterval IEs are present). Requires
libfixbuf-1.4.0 or later. Since SiLK 3.8.0.
interface-values { snmp | vlan }
This optional command specifies the values that should be stored in
the "input" and "output" fields of the SiLK Flow records that are
read from the probe. If this command is not given, the default is
snmp. Note that NetFlow v5 probes only support snmp.
snmp
Store the index of the network interface card (ifIndex) where
the flows entered and left the router, respectively.
vlan
Store the VLAN identifier for the source and destination
networks, respectively. If only one VLAN ID is available,
"input" is set to that value and "output" is set to 0.
This setting does not affect whether rwflowpack(8) stores the
"input" and "output" fields to its output files. Storage of those
fields is controlled by rwflowpack's --pack-interfaces switch.
quirks { none | { firewall-event | zero-packets | missing-ips ... } }
This optional command is used to indicate that special (or quirky)
handling of the incoming data is desired. The value none disables
all quirks, and that is the default setting. If the value is not
none, it may be a list of one or more of the values specified
below. Since SiLK 3.8.0.
firewall-event
Enable checking for firewall event information elements (IEs)
when processing IPFIX or NetFlow v9 flow records. This quirk
must be enabled when collecting data from a Cisco ASA. The
IPFIX firewallEvent IE is 233. The Cisco elements are
NF_F_FW_EVENT (IE 40005) and NF_F_FW_EXT_EVENT (IE 33002).
When this quirk is active, firewall events that match the value
2 (flow deleted) are categorized as normal flows, firewall
events that match the value 3 (flow denied) are usually put
into one of non-routed types (e.g., innull, outnull, see
packlogic-twoway(3) and packlogic-generic(3) for details), and
all other firewall events values are dropped. (Note that a log
message is generated for these dropped records; to suppress
these messages, use the log-flags command.) When this quirk is
not provided, SiLK handles these records normally, which may
result in duplicate flow records. (Prior to SiLK 3.8, SiLK
dropped all flow records that contained a firewall event IE.)
zero-packets
Enable support for flow records either that do not contain a
valid packets field, such as those from the Cisco ASA series of
routers, or that have an unusually large bytes-per-packet
ratio. When this quirk is active, SiLK sets the packet count
to 1 when the incoming IPFIX or NetFlow v9 flow record has a
the packet count if 0. This quirk may modify the file format
used by rwflowpack for IPv4 records in order to support large
byte-per-packet ratios.
missing-ips
Store a flow record even when the record's NetFlow v9/IPFIX
template does not contain IP addresses. One change in SiLK
3.8.0 was to ignore flow records that do not have a source
and/or destination IP address; this quirk allows one to undo
the effect of that change. Since SiLK 3.8.1.
priority value
This optional command is deprecated. It exists for backwards
compatibility and will be removed in the next major release.
To summarize the probe types and the input they can accept:
Probe Type Berkeley Directory UnixDomain Single
Socket Polling Socket File
========== ========== ========== ========== ==========
ipfix tcp/udp yes
netflow-v5 udp yes yes
netflow-v9 udp
sflow udp
silk yes
Group Block
The use of group blocks is optional. They are a convenience to define
a list of commonly used CIDR blocks, IPset files, or integer values
that are treated as SNMP interfaces or VLAN identifiers. Groups are
used in sensor blocks as described in the descriptions for the discard-
when, discard-unless, network-name-ipblocks, network-name-ipsets and
network-name-interfaces commands, below.
The commands in a group definition must all be of the same type. For
example, you cannot mix ipblocks and ipsets commands in a single group
definition, even though both contain IP addresses.
The contents of an existing group may be added to the current group
block by using a group reference after the appropriate keyword. A
group reference is the name of the group prefixed by the at ("@")
character.
The group command is used at top-level to begin a group definition
block, and the remaining commands are accepted within the group block.
group group-name
The group command begins a new group definition block which
continues to the end group command. The argument to the group
command is the name of the group being defined. The group-name
must be unique among all groups. It must begin with a letter, and
it may not contain whitespace characters or the slash character
("/").
end group
The end group command ends the definition of a group. Following an
end group command, top-level commands are again accepted.
interfaces {integer | group-ref} [integer group-ref...]
The interfaces command adds integer values to a group, where each
integer is treated as an SNMP interface number or VLAN identifier.
The interfaces command may appear multiple times in a group block.
Each integer value may be between 0 and 65535 inclusive.
ipblocks {cidr-block | group-ref} [cidr-block group-ref...]
The ipblocks command adds CIDR block values to a group. The
ipblocks command may appear multiple times in a group block. For
groups containing more than a couple of CIDR blocks, consider using
an IPset instead.
ipsets {filename | group-ref} [filename group-ref...]
The ipset command adds the IP addresses specified in an IPset file
(such as that created by rwsetbuild(1)) a group. The ipsets
command may appear multiple times in a group block. When multiple
IPset files are specified, the group maintains a single IPset that
is the union of files. rwflowpack exits with an error if the IPset
file does not exist or does not contain any IP addresses. Since
SiLK 3.10.0.
Specifying the at ("@") character and the name of an existing group
within an interfaces, ipblocks, or ipsets command causes the contents
of the existing group to be added to the current group as long as the
existing group contains interfaces, ipblocks, or ipsets, respectively.
A group does not reference other groups; the contents of the existing
groups are copied into the current group.
For example group blocks, see "Group definitions" below.
Sensor Block
The information from the sensor block is used by rwflowpack to
determine how to categorize a flow; that is, in which file the flow
record is stored. The packlogic-twoway(3) manual page describes how
rwflowpack may use the sensor blocks to determine a record's category.
When the Sensor Configuration file is used with flowcap, no sensors
need to be defined. In fact, flowcap completely ignores all text
inside each sensor block.
The sensor block works with the packing logic to determine where
rwflowpack stores flow records. The packing logic plug-in specifies a
list of network names, and you will refer to these networks when you
configure the sensor block. Most plug-ins provide the "external",
"internal", and "null" names, where internal refers to network being
monitored, null are flows that were blocked by the router's access
control list, and external is everything else.
Several of the commands in the sensor block require as an argument a
list of CIDR blocks, a list of IPset files, or a list of integers.
Instead of specifying a list of values, you may specify a group
reference to a group (see "Group Block") containing ipblocks, ipsets,
or interfaces, respectively. (A group reference the at ("@") character
followed by group's name.) These lists are defined as follows:
cidr-block-list
A cidr-block-list contains one or more CIDR blocks or group
references that represent an address space. As part of determining
how to process a flow record, rwflowpack may check whether the
record's source or destination IP address is in the list. When
comparing an IP address to a cidr-block-list, note the following:
· the IP address is compared to each element in the cidr-block-
list, stopping once a match is made
· when comparing an IPv4 address to a cidr-block-list element
that is IPv6, the IPv4 address is converted to IPv6 by mapping
it into the ::ffff:0:0/96 prefix for purposes of the comparison
· when comparing an IPv6 address to a cidr-block-list element
that is IPv4, an IPv6 address in the ::ffff:0:0/96 prefix will
be converted to IPv4 for purposes of the comparison and any
other IPv6 address will fail the comparison
ipset-list
An ipset-list contains the names of one or more IPset files or
group references. These files represent an address space, and
rwflowpack may check whether a flow record's source or destination
IP address is in this address space. When multiple IPset files are
specified, the contents of the files are merged into a single
IPset. rwflowpack exits with an error if the IPset file does not
exist or does not contain any IP addresses. The rules for
comparing IPv4 and IPv6 addresses are the same as those for the
cidr-block-list. Since SiLK 3.10.0.
interface-list
An interface-list contains one or more group references or integers
(ranging from 0 to 65535) that represent SNMP interface index(es)
or VLAN identifiers. As part of determining how to process a flow
record, rwflowpack may check whether the record's "input" or
"output" fields are in the list. Whether the "input" and "output"
fields contain SNMP interfaces or VLAN identifiers is determined by
the interface-values command in the probe block (c.f. "Probe
Block").
The sensor command is used in the top-level context to begin a sensor
configuration block, and the remaining commands are accepted within the
sensor block.
sensor sensor-name
The sensor command begins a new sensor configuration block. It
takes as an argument the name of the sensor being configured, and
that sensor must be defined in the Site Configuration file (see
silk.conf(5)). A sensor block is closed with the end sensor
command. You may have multiple sensor blocks that have the same
sensor-name.
end sensor
The end sensor command ends the configuration of a sensor.
Following an end sensor command, top-level commands are again
accepted.
probe-type-probes probe-name [probe-name ...]
This command associates the listed probe names of the given probe
type with the sensor. The probes do not have to be defined before
they are used. (Note this also means that a mistyped probe-name
will not be detected.) For example, netflow-v5-probes S1 says that
S1 is a netflow-v5 probe; whenever flow data arrives on the S1
probe, the sensor associated with the probe notices that data is
available and processes it.
source-network network-name
This command causes the sensor to assume that all flows originated
from the network named network-name. For example, if a sensor is
associated a probe that only monitors incoming traffic, you could
use "source-network external" to specify that all traffic
originated from the external network.
destination-network network-name
This command causes the sensor to assume that all flows were sent
to the network named network-name.
network-name-ipblocks {cidr-block-list | remainder}
This command specifies the IP-space that is assigned to the network
named network-name. The value of the command can be the keyword
remainder or a list containing CIDR blocks and/or group references
to groups containing CIDR blocks. When the value is the keyword
remainder, the IP-space for network-name is conceptually all IPs
not assigned to other networks on this sensor. The remainder
keyword may only appear one time within a sensor block.
network-name-ipsets {ipset-list | remainder}
This command specifies the IP-space that is assigned to the network
named network-name. The value of the command can be the keyword
remainder or a list containing the names of IPset files and/or
group references to groups containing IPset files. When the value
is the keyword remainder, the IP-space for network-name is
conceptually all IPs not assigned to other networks on this sensor.
The remainder keyword may only appear one time within a sensor
block.
network-name-interfaces {interface-list | remainder}
This command specifies the SNMP interface index(es) or VLAN
identifiers that are assigned to the network named network-name.
The value of the command can be the keyword remainder or a list
containing interface numbers and/or group references to groups
containing interfaces. When the value is the keyword remainder,
the interface list is computed by finding all interface values not
assigned to other networks on this sensor. The remainder keyword
may only appear one time within a sensor block.
isp-ip ip-address [ip-address ...]
This optional command may be used for a sensor that processes
NetFlow data. The value to the command is a list of IP addresses
in dotted-decimal notation, where the IPs are the addresses of the
NICs on the router. For traffic that doesn't leave the router (and
thus was sent to the router's null-interface), some packing-logic
plug-ins use these IPs to distinguish legitimate traffic for the
router (e.g., routing protocol traffic, whose destination address
would be in this list) from traffic that violated the router's
access control list (ACL).
The following optional sensor block commands provide a way to filter
the flow records that rwflowpack packs for a sensor. Each filter
begins with either discard-when or discard-unless, mentions a flow
record field, and ends with a list of values.
The discard-when command causes the sensor to ignore the flow record if
the property matches any of the elements in the list. When a match is
found, rwflowpack immediately stops processing the record for the
current sensor and the flow is not packed for this sensor.
The discard-unless command causes the sensor to ignore the flow record
unless the property matches one of the elements in the list. That is,
the flow record is packed only if its property matches one of the
values specified in the list, and, when multiple discard-unless
commands are present, if the flow record matches the values specified
in each.
For each individual property, only one of discard-when or discard-
unless may be specified.
discard-when source-interfaces interface-list
Instructs rwflowpack to discard a flow record for this sensor if
the value in the flow's "input" field is listed in interface-list.
When paired with VLAN tagging (see the interface-values command in
the probe block), this allows the administrator to discard flows
that have a specific VLAN tag. The commands discard-when source-
interfaces and discard-unless source-interfaces may not be
specified on the same sensor, but other discard- commands are
permitted.
discard-unless source-interfaces interface-list
Instructs rwflowpack to discard the flow record for this sensor
unless the flow's "input" field is in interface-list. That is, the
flow record may be packed only if its "input" field matches one of
the values specified in interface-list. When paired with VLAN
tagging, this allows one to discard flows that do not have a
specific VLAN tag. The commands discard-when source-interfaces and
discard-unless source-interfaces may not be specified on the same
sensor, but other discard- commands are permitted.
discard-when destination-interfaces interface-list
Discards a flow for this sensor when the flow's "output" field
matches a value in interface-list. May not appear in the same
sensor block with discard-unless destination-interfaces.
discard-unless destination-interfaces interface-list
Discards a flow for this sensor unless the flow's "output" field
matches a value in interface-list. May not appear in the same
sensor block with discard-when destination-interfaces.
discard-when any-interfaces interface-list
Discards a flow for this sensor when either the flow's "input" or
its "output" field matches a value in interface-list. May not
appear in the same sensor block with discard-unless any-interfaces.
discard-unless any-interfaces interface-list
Discards a flow for this sensor unless either the flow's "input" or
its "output" field matches a value in interface-list. May not
appear in the same sensor block with discard-unless any-interfaces.
discard-when source-ipblocks cidr-block-list
Discards a flow for this sensor when the flow's source IP address,
"sIP", matches one of the CIDR blocks in cidr-block-list. May not
appear in the same sensor block with discard-unless source-
ipblocks.
discard-unless source-ipblocks cidr-block-list
Discards a flow for this sensor unless the flow's source IP
address, "sIP", matches one of the CIDR blocks in cidr-block-list.
May not appear in the same sensor block with discard-when source-
ipblocks.
discard-when destination-ipblocks cidr-block-list
Discards a flow for this sensor when the flow's destination IP
address, "dIP", matches one of the CIDR blocks in cidr-block-list.
May not appear in the same sensor block with discard-unless
destination-ipblocks.
discard-unless destination-ipblocks cidr-block-list
Discards a flow for this sensor unless the flow's destination IP
address, "dIP", matches one of the CIDR blocks in cidr-block-list.
May not appear in the same sensor block with discard-when
destination-ipblocks.
discard-when any-ipblocks cidr-block-list
Discards a flow for this sensor when either the flow's source IP or
its destination IP address matches one of the CIDR blocks in cidr-
block-list. May not appear in the same sensor block with discard-
unless any-ipblocks.
discard-unless any-ipblocks cidr-block-list
Discards a flow for this sensor unless either the flow's source IP
or its destination IP address matches one of the CIDR blocks in
cidr-block-list. May not appear in the same sensor block with
discard-when any-ipblocks.
discard-when source-ipsets ipset-list
Discards a flow for this sensor when the flow's source IP address,
"sIP", is in one of IPset files in ipset-list. May not appear in
the same sensor block with discard-unless source-ipsets. Since
SiLK 3.10.0.
discard-unless source-ipsets ipset-list
Discards a flow for this sensor unless the flow's source IP
address, "sIP", is in one of IPset files in ipset-list. May not
appear in the same sensor block with discard-when source-ipsets.
Since SiLK 3.10.0.
discard-when destination-ipsets ipset-list
Discards a flow for this sensor when the flow's destination IP
address, "dIP", is in one of the IPset files in ipset-list. May
not appear in the same sensor block with discard-unless
destination-ipsets. Since SiLK 3.10.0.
discard-unless destination-ipsets ipset-list
Discards a flow for this sensor unless the flow's destination IP
address, "dIP", is in one of the IPset files in ipset-list. May
not appear in the same sensor block with discard-when destination-
ipsets. Since SiLK 3.10.0.
discard-when any-ipsets ipset-list
Discards a flow for this sensor when either the flow's source IP or
its destination IP address is in one of the IPset files in ipset-
list. May not appear in the same sensor block with discard-unless
any-ipsets. Since SiLK 3.10.0.
discard-unless any-ipsets ipset-list
Discards a flow for this sensor unless either the flow's source IP
or its destination IP address is in one of the IPset files in
ipset-list. May not appear in the same sensor block with discard-
when any-ipsets. Since SiLK 3.10.0.
EXAMPLES
All these examples assume you are using the packlogic-twoway(3) packing
logic plug-in to rwflowpack(8).
Group definitions
The following shows how to create groups that can be used in other
group blocks or in certain commands within a sensor block.
group G01
interfaces 1 2, 3
interfaces 4
end group
group G02
interfaces 5 @G01
end group
group G03
interfaces @G02
interfaces 6
end group
group G11
ipblocks 192.0.2.0/27 192.0.2.32/27, 192.0.2.64/26
ipblocks 192.0.2.128/25
end group
group G12
ipblocks 198.51.100.0/24 @G11
end group
group G13
ipblocks @G12
ipblocks 203.0.113.0/24
end group
group G21
ipsets /var/sets/ip1.set /var/sets/ip2.set, /var/sets/ip3.set
ipsets /var/sets/ip4.set
end group
group G22
ipsets /var/sets/ip5.set @G21
end group
group G23
ipsets @G22
ipsets /var/sets/ip6.set
end group
NetFlow v5 Categorized by SNMP Interface
The following two blocks define a probe that listens on 9900/udp for
NetFlow v5 from a router. The probe only accepts traffic originating
from 172.16.22.22 or 172.16.33.33. The associated sensor uses the SNMP
interfaces to categorize the flows, where traffic that enters the
router on interface 1 and leaves on interface 8 is in, traffic entering
on 8 and leaving on 1 is out, traffic from 1 to 0 is innull, traffic
from 8 to 8 is int2int, etc.
probe S1 netflow-v5
listen-on-port 9901
protocol udp
accept-from-host 172.16.22.22 172.16.33.33
end probe
sensor S1
netflow-v5-probes S1
external-interfaces 1
internal-interfaces 8
null-interfaces 0
end sensor
NetFlow v5 Categorized by IP Address
The probe in this example is the same as above, except the
administrator has chosen to log only messages about bad packets
(messages about missing packets will be ignored). The sensor is
categorizing flows by the source and destination IP address in the flow
record. The internal network is defined as 128.2.0.0/16, and all other
IPs are defined as external. For example, HTTP traffic whose source is
128.2.0.1 and destination is google.com will be categorized as outweb;
the reply (source of google.com and destination 128.2.0.1) will be
inweb.
probe S2 netflow-v5
listen-on-port 9902
protocol udp
accept-from-host 172.16.22.22 172.16.33.33
log-flags bad # ignore missing pkts
end probe
sensor S2
netflow-v5-probes S2
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
IPFIX Categorized by IP Address
This example uses an IPFIX probe to collect the flows on port 9903/tcp,
where the probe binds to address 192.168.1.92. The sensor
configuration is the same as in the previous example, but a group
definition is used to define the internal network.
probe S3 ipfix
listen-on-port 9903
protocol tcp
listen-as-host 192.168.1.92
end probe
group my-network
ipblocks 128.2.0.0/16
end group
sensor S3
ipfix-probes S3
internal-ipblocks @my-network
external-ipblocks remainder
end sensor
IPFIX Read from Files
This example uses the same sensor configuration as above. The probe
processes files that have been created by yaf(1) and stored in the
directory /tmp/var/yaf/.
probe S4 ipfix
poll-directory /tmp/var/yaf
end probe
sensor S4
ipfix-probes S4
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
NetFlow v9 Categorized by IP Address
This example uses a NetFlow v9 probe to collect the flows on port
9905/udp, where the probe binds to address 192.168.1.92. The sensor
configuration uses an IPset file to define the internal network.
probe S5 netflow-v9
listen-on-port 9905
protocol udp
listen-as-host 192.168.1.92
end probe
sensor S5
netflow-v9-probes S5
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
sFlow v5 Categorized by IP Address
This example uses an sFlow probe to collect the flows on port 9906/udp,
where the probe binds to the IPv6 address ::1. The sensor
configuration uses an IPset file to define the internal network.
probe S19 sflow
listen-on-port 9906
protocol udp
listen-as-host ::1
end probe
sensor S19
sflow-probes S19
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
NetFlow v9 from a Cisco ASA Router
When collecting NetFlow v9 data from a Cisco ASA (Adaptive Security
Appliance), specify the quriks statement as shown in this example to
enable special handling of the NetFlow data.
probe S20 netflow-v9
listen-on-port 9988
protocol udp
quirks firewall-event zero-packets
end probe
sensor S20
netflow-v9-probes S20
internal-ipsets /var/sets/my-network.set
external-ipsets remainder
end sensor
Multiple Sources Becoming One Sensor (One Port)
Consider a scenario where there are multiple input streams that need to
be treated as a single sensor. For example, you use multiple routers
for load-balancing but you want them treated as a single logical
sensor. In this configuration, you send all the input streams to a
single port, and you define a single probe listening on that port. As
long as the streams have a unique source IP, the streams will be
treated distinctly.
The following sensor and probe blocks accept any number of TCP-based
IPFIX connections to port 9907 and any number of NetFlow v5 connections
to 9908. This configuration works for all types of input as SiLK 3.4.0
when using libfixbuf-1.2.0. See the configuration in the following
example for a alternate approach.
probe S7 ipfix
listen-on-port 9907
protocol tcp
end probe
sensor S7
ipfix-probes S7
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
probe S8 netflow-v5
listen-on-port 9908
protocol udp
log-flags bad
end probe
sensor S8
netflow-v5-probes S8
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
Multiple Sources Becoming One Sensor (Multiple Ports)
Like the previous example, this example configuration causes multiple
input streams to be treated as a single sensor. In this solution, each
stream arrives on a separate port where it is collected by a separate
probe. The sensor block combines the probes into one sensor. This
type of approach works with all types of input for all releases of
SiLK.
probe S6-p1 netflow-v9
listen-on-port 9961
protocol udp
end probe
probe S6-p2 netflow-v9
listen-on-port 9962
protocol udp
end probe
probe S6-p3 netflow-v9
listen-on-port 9963
protocol udp
end probe
sensor S6
netflow-v9-probes S6-p1, S6-p2, S6-p3
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
Multiple Sources Becoming One Sensor (Specific Directions)
Consider the case of using yaf on a monitor at the border of a network
where all traffic entering the network arrives at the monitor on one
network interface card (NIC) and all traffic leaving the network
arrives at the monitor on a different NIC. Since yaf does not support
multiple interfaces yet, you must run two yaf processes, one for each
NIC. The sensor configuration for this monitor would list two probes,
each listening on a different port, and two sensor blocks both packing
to the same sensor. Each sensor block packs the traffic as incoming or
outgoing depending on which probe received the traffic.
probe S9-in ipfix
listen-on-port 9991
protocol tcp
end probe
probe S9-out ipfix
listen-on-port 9992
protocol tcp
end probe
sensor S9
ipfix-probes S9-in
source-network external
destination-network internal
end sensor
sensor S9
ipfix-probes S9-out
source-network internal
destination-network external
end sensor
Multiple Sources to Multiple Sensors (Same Port)
Suppose your network has multiple flow generators (NetFlow v5 routers
or yaf processes) that you wish to treat as separate sensors, but you
want to open a single port on your firewall to collect these flow
records. To support this configuration, configure the probes to
distinguish the traffic based on the source address. Specifically,
create a separate probe for each sensor where each probe uses the same
listen-on-port value but different accept-from-host values. The
following configuration uses a NetFlow v5 probe, which works for all
versions of SiLK. A similar configuration works for any type of input
as of SiLK 3.4.0 and libfixbuf-1.2.0.
probe S10 netflow-v5
listen-on-port 9910
accept-from-host 172.16.22.10
protocol udp
end probe
probe S11 netflow-v5
listen-on-port 9910
accept-from-host 172.16.22.11
protocol udp
end probe
group my-network2
ipblocks 128.2.0.0/16
end group
sensor S10
netflow-v5-probes S10
internal-ipblocks @my-network2
external-ipblocks remainder
end sensor
sensor S11
netflow-v5-probes S11
internal-ipblocks @my-network2
external-ipblocks remainder
end sensor
Single Source Becoming Multiple Sensors
Suppose you have instrumented a single router but you wish to split the
traffic into two sensors, where one part of the network (monitored by
sensor S12) is defined as 128.2.0.0/17, and the other (sensor S13) as
128.2.128.0/17. Traffic between 128.2.0.1 and google.com will be
assigned to sensor S12, but it will so appear as ext2ext traffic for
sensor S13 unless you explicitly discard that traffic using the
discard-unless command.
probe S12-S13 ipfix
listen-on-port 9912
protocol tcp
end probe
group S12-space
ipblocks 128.2.0.0/17
end group
group S13-space
ipblocks 128.2.128.0/17
end group
sensor S12
ipfix-probes S12-S13
discard-unless any-ipblock @S12-space
internal-ipblocks @S12-space
external-ipblocks remainder
end sensor
sensor S13
ipfix-probes S12-S13
discard-unless any-ipblock @S13-space
internal-ipblocks @S13-space
external-ipblocks remainder
end sensor
Discarding Flows Using VLAN Tags
You can configure rwflowpack to discard flows that do not have a
particular VLAN tag. First, specify the interface-values command to
instruct the probe to put the VLAN id into the fields that typically
store the SNMP interfaces. On the sensor, use the discard-unless
command to discard flows that do not have the desired VLAN tag (114 in
this example). Often you will not use the VLAN tags to determine a
flow's direction (category) since there is a single VLAN tag on each
flow; instead, you specify the IP space of the monitored network in the
sensor block. (However, see the next example.)
probe S14 ipfix
listen-on-port 9914
protocol tcp
interface-values vlan
end probe
sensor S14
ipfix-probes S14
discard-unless any-interface 114
internal-ipblocks 128.2.0.0/16
external-ipblocks remainder
end sensor
Categorizing Flows Using VLAN Tags
By repeating a sensor block and using different discard-unless commands
in each block, you may configure rwflowpack to categorize flow records
based on VLAN tags. Suppose yaf is monitoring a connection where
incoming flows are marked with VLAN tag 151 and outgoing flows are
marked with 152. You simply discard any traffic that does not have the
wanted VLAN tag, and use the source-network and destination-network
commands to assign the direction to the flow. In this example, any
flow record that does not have one of the expected VLAN tags has its
source-network set to "null", but since rwflowpack does not expect a
flow record to originate from the null network, it stores the record in
the other category for later analysis/debugging. (This example
requires SiLK 3.1 or later.)
probe S15 ipfix
listen-on-port 9915
protocol tcp
interface-values vlan
end probe
sensor S15
# vlan ID 151 is incoming
ipfix-probes S15
discard-unless source-interface 151
source-network external
destination-network internal
end sensor
sensor S15
# vlan ID 152 is outgoing
ipfix-probes S15
discard-unless source-interface 152
source-network internal
destination-network external
end sensor
sensor S15
# discard flows that have known IDs
# force unknown IDs into the "other" category
ipfix-probes S15
discard-when source-interface 151,152
source-network null
destination-network internal
end sensor
IPFIX Collected by a DAG Card
When yaf generates flow records from a multi-port Endace DAG card, it
is possible to use the port where the traffic was seen to categorize
the traffic in rwflowpack.
To do this, include the --dag-interface switch on the yaf command line.
This switch causes yaf to store the DAG port where the packet was
collected into the equivalent of the SNMP input field, and yaf sets the
SNMP output field to an offset of the port, specifically the port plus
256 (0x100|port).
Assume DAG port 0 is connected to the external side of the network (so
it sees incoming traffic), and assume DAG port 1 is on the internal
side. For incoming traffic, yaf sets the input and output values to 0
and 256, respectively. For outgoing traffic, the values are 1 and 257.
The sensor.conf configuration file for rwflowpack would be:
probe S16-dag ipfix
listen-on-port 9916
protocol tcp
end probe
sensor S16
ipfix-probes S16-dag
external-interface 0,257
internal-interface 1,256
end sensor
When rwflowpack processes the IPFIX flow records, it treats flow
records having an input of 0 and an output of 256 as traffic moving
from an external interface to an internal interface, and rwflowpack
packs those records as incoming. Similarly for the outgoing flow
records.
Repacking of SiLK Flows by IP Address
A SiLK probe must get its flows by polling a directory of SiLK Flow
files. The flows can be re-categorized based on the IP addresses or
based on the SNMP interfaces (beware: often the SNMP interface values
are 0 in SiLK Flow data). In this example, the files in the directory
/var/tmp/old-data/ are processed. The internal network is defined as
128.2.0.0/16, and all other IPs are defined as external.
probe S17 silk
poll-directory /var/tmp/old-data
end probe
sensor S17
silk-probes S17
internal-ipblock 128.2.0.0/16
external-ipblock remainder
end sensor
NetFlow From a File Categorized by SNMP Interfaces
Instead of listening on a UDP port for NetFlow traffic, you can
configure the probe to process a single file containing NetFlow v5
PDUs. This example assumes you are running rwflowpack with the
switches --input-mode=file --netflow-file=FILENAME. The --netflow-file
switch overrides the read-from-file command on the probe. rwflowpack
will exit once it processes that single file.
probe S18 netflow-v5
log-flags bad # ignore missing pkts
read-from-file /dev/null # use --netflow-file=<file>
end probe
sensor S18
netflow-v5-probes S18
external-interface 182
internal-interface 189
null-interface 0
end sensor
SEE ALSOrwflowpack(8), flowcap(8), packlogic-twoway(3), packlogic-generic(3),
rwsetbuild(1), silk.conf(5), silk(7), SiLK Installation Handbook,
pcap(3), yaf(1), gzip(1)NOTES
The accept-from-host command began to accept a list of arguments in
SiLK 3.10.1.
SiLK 3.10.0 added IPset file support to the group block and to some
commands in the sensor block.
Support for collecting sFlow records was added in SiLK 3.9.0.
The quirks command was introduced in SiLK 3.8.0.
SiLK 3.11.0.1 2016-02-19 sensor.conf(5)
