rwptoflow(1) SiLK Tool Suite rwptoflow(1)NAMErwptoflow - Generate SiLK Flow records from packet data
SYNOPSISrwptoflow [--plugin=PLUGIN [--plugin=PLUGIN ...]]
[--active-time=YYYY/MM/DD:hh:dd:mm:ss.uuuuuu-YYYY/MM/DD:hh:dd:mm:ss.uuuuuu]
[--flow-output=FLOW_PATH] [--packet-pass-output=PCKTS_PASS]
[--packet-reject-output=PCKTS_REJECT]
[--reject-all-fragments] [--reject-nonzero-fragments]
[--reject-incomplete] [--set-sensorid=SCALAR]
[--set-inputindex=SCALAR] [--set-outputindex=SCALAR]
[--set-nexthopip=IP_ADDRESS] [--print-statistics]
[--note-add=TEXT] [--note-file-add=FILE]
[--compression-method=COMP_METHOD] TCPDUMP_INPUT
rwptoflow [--plugin=PLUGIN ...] --help
rwptoflow--version
DESCRIPTIONrwptoflow attempts to generate a SiLK Flow record for every Ethernet IP
IPv4 packet in the pcap(3) (tcpdump(1)) capture file TCPDUMP_INPUT.
TCPDUMP_INPUT must contain data captured from an Ethernet datalink.
rwptoflow does not attempt to reassemble fragmented packets or to
combine multiple packets into a single flow record. rwptoflow is a
simple program that creates one SiLK Flow record for every IPv4 packet
in TCPDUMP_INPUT. (For an alternate approach, consider using the
rwp2yaf2silk(1) tool as described at the end of this section.)
rwptoflow will read from its standard input if TCPDUMP_INPUT is
specified as "stdin". The SiLK Flow records are written to the
specified flow-output file or to the standard output. The application
will fail when attempting to read or write binary data from or to a
terminal.
Packets outside of a user-specified active-time window can be ignored.
Additional filtering on the TCPDUMP_INPUT can be performed by using
tcpdump with an "expression" filter and piping tcpdump's output into
rwptoflow.
In addition to generating flow records, rwptoflow can write pcap files
containing the packets that it used to generate each flow, and/or the
packets that were rejected. Note that packets falling outside the
active-time window are ignored and are not written to the packet-
reject-output.
Statistics of the number of packets read, rejected, and written can be
printed.
rwptoflow will reject any packet that is not an IPv4 Ethernet packet
and any packet that is too short to contain the Ethernet and IP
headers. At the user's request, packets may be rejected when
· they are fragmented---either the initial (zero-offset) fragment or
a subsequent fragment
· they have a non-zero fragment offset
· they are not fragmented or they are the zero-fragment but the
capture file does not contain enough information about the packet
to set protocol-specific information---namely the ICMP type and
code, the UDP source and destination ports, or the TCP source and
destination ports and flags
Since the input packet formats do not contain some fields normally
found in NetFlow data, rwptoflow provides a way to set those flow
values in all packets. For example, it is possible to set the sensor-
id manually for a tcpdump source, so that flow data can be filtered or
sorted by that value later.
Alternative to rwptoflow
As mentioned above, rwptoflow is a simple program for processing
Ethernet IP IPv4 packets. rwptoflow does not:
· reassemble fragmented packets
· support IPv6 packets
· combine multiple packets into a single flow record
· support any decoding of packets (e.g., 802.1q)
For these features (and others), you should use the yaf(1) application
(<http://tools.netsa.cert.org/yaf/>) to read the pcap file and generate
an IPFIX stream, and pipe the IPFIX stream into rwipfix2silk(1) to
convert it to SiLK Flow records.
The rwp2yaf2silk(1) script makes this common usage more convenient by
wrapping the invocation of yaf and rwipfix2silk. You give rwp2yaf2silk
a pcap file and it writes SiLK Flow records.
By default, rwptoflow creates a flow record for every packet, fragments
and all. You can almost force yaf to create a flow record for every
packet: When you give yaf the --idle-timeout=0 switch, yaf creates a
flow record for every complete packet and for each packet that it is
able to completely reassemble from packet fragments. Any fragmented
packets that yaf cannot reassemble are dropped.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
--plugin=PLUGIN
Use the specified plug-in to ignore or reject packets or to modify
the flow record that is generated from the packet. The switch may
be repeated to load multiple plug-ins. See the "PLUG-IN SUPPORT"
section below for details.
--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]-YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
Ignore all packets whose time falls outside the specified range.
The times must be specified to at least day precision. The start
time is required; when the end-time is not present, it is treated
as infinite. The end-time will be rounded-up to instant before the
next time unit; i.e., an end-time of 2006/08/31:15 is treated as
2006/08/31:15:59:59.999999.
--flow-output=FLOW_PATH
Write the generated SiLK Flow records to the specified file at
FLOW_PATH. When this switch is not provided, the flows are written
to the standard output.
--packet-pass-output=PCKTS_PASS
For each generated SiLK Flow record, write the packet that
generated the flow to the pcap file specified by PCKTS_PASS. Use
"stdout" to write the packets to the standard output.
--packet-reject-output=PCKTS_REJECT
Write each packet that occurs within the active-time window but for
which a SiLK Flow record was not generated to the pcap file
specified by PCKTS_REJECT. Use "stdout" to write the packets to
the standard output.
The packets that get written to this file may include packets that
were shorter than that required to get the IP header, non-IPv4
packets, and packets that get treated as "reject" packets by the
following switches.
--reject-all-fragments
Do not generate a SiLK Flow record for the packet when the packet
is fragmented. This includes the initial (zero-offset) fragment
and all subsequent fragments. If --packet-reject-output is
specified, the packet will be written to that file.
--reject-nonzero-fragments
Do not generate a SiLK Flow record for the packet when the packet
is fragmented unless this is the initial fragment. That is, reject
all packets that have a non-zero fragmentation offset. Normally
flow records are generated for these packets, but the ports and TCP
flag information is set to zero. If --packet-reject-output is
specified, the packet will be written to that file.
--reject-incomplete
Do not generate a SiLK Flow record for the packet when the packet's
fragmentation-offset is zero yet the packet does not contain enough
information to completely specify an ICMP, UDP, or TCP record (that
is, the packet is too short to set the ICMP type and code, the UDP
or TCP source or destination port, or the TCP flags). Normally,
flow records are generated for these packets but the ports and TCP
flag information is set to zero. This switch has no effect on
packets where the protocol is not 1,6, or 17.
This switch does not imply --reject-nonzero-fragments; to indicate
that all generated flow records must have valid port and TCP flag
information, specify --reject-nonzero-fragments
--reject-incomplete.
--set-sensorid=SCALAR
Set the sensor ID for all flows to SCALAR. SCALAR should be an
integer value between 0 and 65534, inclusive. When not specified,
the sensor ID is set to 65535.
--set-inputindex=SCALAR
Set the input SNMP index value for all flows to SCALAR. SCALAR
should be an integer value between 0 and 65535, inclusive. When
not specified, the SNMP input is set to 0.
--set-outputindex=SCALAR
Set the output SNMP index value for all flows to SCALAR. SCALAR
should be an integer value between 0 and 65535, inclusive. When
not specified, the SNMP output is set to 0.
--set-nexthopip=IP_ADDRESS
Set the next-hop IP address for all flows to IP_ADDRESS; IP_ADDRESS
may be in its canonical form or an integer. When not specified,
the next-hop IP is set to 0.0.0.0.
--print-statistics
Print a summary of the packets that were processed. This summary
includes
· the total number of packets read
· the number that fell outside the time-window
· the number that were too short to get the IP header
· the number that were not IPv4
· the number that were discarded by a plug-in
· the total number of fragmented packets
· the number of fragments where the offset was zero
· the number of zero-offset packets that were incomplete
· the number of flows written to the output
--note-add=TEXT
Add the specified TEXT to the header of the output file as an
annotation. This switch may be repeated to add multiple
annotations to a file. To view the annotations, use the
rwfileinfo(1) tool.
--note-file-add=FILENAME
Open FILENAME and add the contents of that file to the header of
the output file as an annotation. This switch may be repeated to
add multiple annotations. Currently the application makes no
effort to ensure that FILENAME contains text; be careful that you
do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD
Specify how to compress the output. When this switch is not given,
output to the standard output or to named pipes is not compressed,
and output to files is compressed using the default chosen when
SiLK was compiled. The valid values for COMP_METHOD are determined
by which external libraries were found when SiLK was compiled. To
see the available compression methods and the default method, use
the --help or --version switch. SiLK can support the following
COMP_METHOD values when the required libraries are available.
none
Do not compress the output using an external library.
zlib
Use the zlib(3) library for compressing the output, and always
compress the output regardless of the destination. Using zlib
produces the smallest output files at the cost of speed.
lzo1x
Use the lzo1x algorithm from the LZO real time compression
library for compression, and always compress the output
regardless of the destination. This compression provides good
compression with less memory and CPU overhead.
best
Use lzo1x if available, otherwise use zlib. Only compress the
output when writing to a file.
--help
Print the available options and exit. Options that add fields can
be specified before --help so that the new options appear in the
output.
--version
Print the version number and information about how SiLK was
configured, then exit the application.
PLUG-IN SUPPORT
rwptoflow allows the user to provide additional logic to ignore or
reject packets, or to modify the flow record that is generated from the
packet. To do this, the user creates a plug-in that gets loaded at
run-time by giving rwptoflow the --plugin switch with the path to the
plug-in as the parameter to the switch.
A plug-in is a shared object file (a.k.a. dynamic library) that is
compiled from C source code. The plug-in should have four subroutines
defined:
setup()
is called when the object is first loaded. This is the place to
initialize global variables to their default values. If the plug-
in provides switches of its own, they must be registered in this
subroutine.
initialize()
gets called after all options have been processed but before any
packets are read from the input. If this subroutine does not
return 0, the application will quit.
ptoflow()
will be called for every packet that rwptoflow is able to convert
into a flow record just before the flow record is written. This
subroutine will not see packets that are short or that are not
IPv4; it will also not see fragmented packets if
--reject-all-fragments is specified.
The "ptoflow()" function is called with two parameters:
· a pointer to the "rwRec" object that rwptoflow created from the
packet. The subroutine may modify the record as it sees fit.
· a void pointer that the function may cast to a pointer to the C
structure:
typedef struct _sk_pktsrc_t {
/* the source of the packets */
pcap_t *pcap_src;
/* the pcap header as returned from pcap_next() */
const struct pcap_pkthdr *pcap_hdr;
/* the packet as returned from pcap_next() */
const u_char *pcap_data;
} sk_pktsrc_t;
This structure gives the user access to all the information
about the packet.
The following return values from "ptoflow()" determines whether
rwptoflow writes the flow and the packet:
0 Write the flow record to the flow-output and the packet to
the PCKTS_PASS unless another plug-in instructs otherwise.
1 Write the flow record to the flow-output and the packet to
the PCKTS_PASS immediately; do not call the "ptoflow()"
routine on any other plug-in.
2 Treat the packet as a reject: Do not write the flow record;
write the packet to the PCKTS_REJECT immediately; do not
call the "ptoflow()" routine on any other plug-in.
3 Ignore the packet immediately: Do not write the flow record
nor the packet; do not call the "ptoflow()" routine on any
other plug-in.
If "ptoflow()" returns any other value, the rwptoflow application
will terminate with an error.
teardown()
is called as the application exits. The user can use this routine
to print results and to free() any data structures that were used.
rwptoflow uses the following rules to find the plug-in: When PLUGIN
contains a slash ("/"), rwptoflow assumes the path to PLUGIN is
correct. Otherwise, rwptoflow will attempt to find the file in
$SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these
directories parallel to the application's directory: lib/silk,
share/lib, and lib. If rwptoflow does not find the file, it assumes
the plug-in is in the current directory. To force rwptoflow to look in
the current directory first, specify --plugin=./PLUGIN. When the
SILK_PLUGIN_DEBUG environment variable is non-empty, rwptoflow prints
status messages to the standard error as it tries to open each of its
plug-ins.
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Given the packet capture file data.pcap, convert it to a SiLK flow
file, data.rw, and copy the packets that rwptoflow understands to the
file good.pcap:
$ rwptoflow data.pcap --packet-pass=good.pcap --flow-out=data.rw
Use rwfilter to partition the SiLK Flows records, writing those records
whose source IPs are found in the IPset file sip.set to filtered.rw:
$ rwfilter --sipset=sip.set --pass=filtered.rw data.rw
Match the capture file, good.pcap, against the filtered SiLK file, in
affect generating a capture file which has been filtered by sip.set:
$ rwpmatch --flow-file=filtered.rw good.pcap > filtered.pcap
ENVIRONMENT
SILK_PLUGIN_DEBUG
When set to 1, rwptoflow print status messages to the standard
error as it tries to open each of its plug-ins.
SEE ALSOrwpmatch(1), rwpdedupe(1), rwfileinfo(1), silk(7), rwp2yaf2silk(1),
rwipfix2silk(1), yaf(1), tcpdump(1), pcap(3), mergecap(1), zlib(3)NOTES
SiLK supports millisecond timestamps. When reading packets whose
timestamps have finer precision, the times are truncated at the
millisecond position.
The mergecap(1) or rwpdedupe(1) programs can be used to join multiple
tcpdump capture files in order to convert into a single flow file.
SiLK 3.11.0.1 2016-02-19 rwptoflow(1)
