YAF(1) Yet Another Flowmeter YAF(1)NAMEyaf - Yet Another Flowmeter
SYNOPSISyaf [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
[--config CONFIG_FILE]
[--live LIVE_TYPE] [--ipfix TRANSPORT_PROTOCOL]
[--no-output]
[--decompress DECOMPRESS_DIR]
[--filter BPF_FILTER]
[--rotate ROTATE_DELAY] [--lock] [--caplist]
[--groupSPREAD_GROUP_NAME(s)]
[--groupby GROUPBY_TYPE]
[--stats INTERVAL][--no-stats] [--noerror]
[--export-interface]
[--gre-decode] [--no-frag]
[--max-frags FRAG_TABLE_MAX]
[--ip4-only] [--ip6-only]
[--idle-timeout IDLE_TIMEOUT]
[--active-timeout ACTIVE_TIMEOUT]
[--udp-temp-timeout TEMPLATE_TIMEOUT]
[--force-read-all]
[--flow-stats] [--delta]
[--ingress INGRESS_INT] [--egress EGRESS_INT]
[--max-payload PAYLOAD_OCTETS] [--udp-payload]
[--max-export PAYLOAD_OCTETS]
[--max-flows FLOW_TABLE_MAX]
[--export-payload] [--silk] [--udp-uniflow PORT]
[--uniflow] [--mac] [--force-ip6-export]
[--observation-domain DOMAIN_ID] [--entropy]
[--applabel] [--applabel-rules RULES_FILE]
[--ipfix-port PORT] [--tls] [--tls-ca CA_PEM_FILE]
[--tls-cert CERT_PEM_FILE] [--tls-key KEY_PEM_FILE]
[--become-user UNPRIVILEGED_USER]
[--become-group UNPRIVILEGED_GROUP]
[--log LOG_SPECIFIER] [--loglevel LOG_LEVEL]
[--verbose] [--version]
[--p0fprint] [--p0f-fingerprints FILENAME]
[--fpexport]
[--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]]
[--plugin-opts "OPTIONS[,OPTIONS...]"]
[--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]]
[--pcap PCAP_FILE_PREFIX] [--pcap-per-flow]
[--max-pcap MAX_FILE_MB] [--pcap-timer PCAP_ROTATE_DELAY]
[--pcap-meta-file META_FILE_PREFIX] [--index-pcap]
[--hash FLOW_KEY_HASH] [--stime FLOW_START_TIMEMS]
DESCRIPTIONyaf is Yet Another Flowmeter and yaf is a suite of tools to do flow
metering. yaf is used as a sensor to capture flow information on a
network and export that information in IPFIX format. It reads packet
data from pcap(3) dumpfiles as generated by tcpdump(1), from live
capture from an interface using pcap(3), pf_ring, an Endace DAG capture
device, a Napatech adapter, or Netronome NFE card aggregates these
packets into flows, and exports flow records via IPFIX over SCTP, TCP
or UDP, Spread, or into serialized IPFIX message streams (IPFIX files)
on the local file system.
Since yaf is designed to be deployed on white-box sensors attached to
local network segments or span ports at symmetric routing points, it
supports bidirectional flow assembly natively. Biflow export is done
via the export method specified in RFC 5103 Bidirectional Flow Export
using IPFIX. See the OUTPUT section below for information on this
format.
yaf also supports experimental partial payload capture, specifically
for banner-grabbing applications and protocol verification purposes.
The output of yaf is designed to be collected and manipulated by flow
processing toolchains supporting IPFIX. The yafscii(1) tool, which is
installed as part of yaf, can also be used to print yaf output in a
human-readable format somewhat reminiscient of tcpdump(1). yaf output
can also be analyzed using the SiLK suite, and the nafalize(1) tool,
both available from the CERT NetSA group.
OPTIONS
Configuration File
The YAF configuration file can be used instead of or in addition to
command line arguments. Lua must be installed for use of the yaf
configuration file.
--config CONFIGURATION_FILE
If present, use the variables set in the CONFIGURATION_FILE. The
CONFIGURATION_FILE is a Lua configuration file, a plain text file
that can also be a Lua program. A sample configuration file can be
found in etc/yaf.init. yaf will use the variables set in the
configuration file along with any command line arguments.
Input Options
These options control where yaf will take its input from. yaf can read
packets from a pcap dumpfile (as generated by tcpdump -w) or live from
an interface via libpcap, libdag, or libnapatech, or Netronome API. By
default, if no input options are given, yaf reads a pcap dumpfile on
standard input.
--in INPUT_SPECIFIER
INPUT_SPECIFIER is an input specifier. If --live is given, this is
the name of an interface (e.g. "eth0", "en0", "dag0", "nt3g",
"nt3g0:1", "0:0") to capture packets from. Otherwise, it is a
filename; the string - may be used to read from standard input (the
default). See --live for more information on formats for Napatech,
Dag, and Netronome Interface formats.
--caplist
If present, treat the filename in INPUT_SPECIFIER as an ordered
newline-delimited list of pathnames to pcap(3) dumpfiles. Blank
lines and lines beginning with the character '#' within this are
ignored. All pathnames are evaluated with respect to the working
directory yaf is run in. These dumpfiles are processed in order
using the same flow table, so they must be listed in ascending time
order. This option is intended to ease the use of yaf with rotated
or otherwise split tcpdump(1) output.
--noerror
Used with the --caplist option. When present, this prevents yaf
from exiting when processing a list of dumpfiles in the middle due
to an error in a file. yaf will continue to process all files
given in the INPUT_SPECIFIER despite errors within those files.
--live LIVE_TYPE
If present, capture packets from an interface named in the
INPUT_SPECIFIER. LIVE_TYPE is one of pcap for packet capture via
libpcap, pfring for packet capture via libpfring, or dag for packet
capture via an Endace DAG interface using libdag, or napatech for
packet capture via a Napatech Adapter, or netronome for packet
capture via a Netronome NFE card, or zc for packet capture via
PF_RING ZC. <pfring> is only available if yaf was built with
PF_RING support. See the yafzcbalance(1) man page for using yaf
with PF_RING ZC. dag is only available if yaf was built with Endace
DAG support. napatech is only available if yaf was built with
Napatech API support. If LIVE_TYPE is napatech, the INPUT_SPECIFIER
given to --in should be in the form nt3g[<streamID>:<ports>].
StreamID and Ports are optional. StreamID if given, is the ID that
the traffic stream will be assigned to on the incoming ports. Ports
may be a comma-separated list of ports to listen on. If [ports] is
not specified, the default is to listen on All ports. StreamID
defaults to 0. netronome is only available if yaf was built with
Netronome API support. If LIVE_TYPE is netronome, the
INPUT_SPECIFIER given to --in should be in the form <device>:<ring>
where device is the NFE card ID, typically 0. Ring is the capture
ring ID which is configured via a modprobe configuration file and
resides in /etc/modprobe.d/pcd.conf.
--export-interface
If present, the interface on which a packet was received will be
noted internally within yaf. When flow records are exported from
yaf, an "ingressinterface" and an "egressinterface" set of fields
will be added to the output. The "ingressinterface" field will be
the physical interface which captured the packet while the
"egressinterface" will be the physical interface | 0x100. This can
be used to separate traffic based on DAG physical ports. For use
with the DAG card, traffic received on separate ports will be
separated into different flows if yaf is configured with the
--enable-daginterface option. Otherwise the physical port will
simply be exported in the "ingressInterface" or "egressInterface"
fields in the IPFIX record (flows can exist over multiple
interfaces). This option requires building DAG, Netronome, or
Napatech support in yaf with the --with-dag, --with-napatech, or
--with-netronome switch. In previous versions of yaf this option
was enabled using the --dag-interface or --napatech-interface
switch. It is now enabled by default when yaf is built with DAG,
Netronome, or Napatech support. It can be disabled by configuring
yaf with <--enable-interface=no>. To separate traffic received on
separate ports into separate flows, you must use
--enable-daginterface when configuring yaf.
--filter BPF_FILTER
If present, enable Berkeley Packet Filtering (BPF) in yaf with
BPF_FILTER as the incoming traffic filter. Syntax of BPF_FILTER
follows the expression format described in the tcpdump(1) man page.
This option is not currently supported if --live is set to dag or
napatech or netronome as BPF filtering is implemented with libpcap.
However, you may be able to use a BPF filter by running yaf with
the DAG, Napatech, or Netronome implementations of libpcap.
--decompress DECOMPRESS_DIR
If present and the input file(s) are compressed (gzip'd),
decompress the file to a temporary file within DECOMPRESS_DIR. If
--caplist is also present, all files will be decompressed to
DECOMPRESS_DIR. If this option is not present, yaf will decompress
files to the variable specified by the TMPDIR environment variable
or /tmp if TMPDIR is not set. The zlib library must be installed
to use this feature.
Output Options
These options control where yaf will send its output. yaf can write
flows to an IPFIX file or export flows to an IPFIX collector over SCTP,
TCP, UDP, or Spread. By default, if no output options are given, yaf
writes an IPFIX file to standard output.
--out OUTPUT_SPECIFIER
OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the
OUTPUT_SPECIFIER specifies the hostname or IP address of the
collector to which the flows will be exported. Otherwise, if
--rotate is present, OUTPUT_SPECIFIER is the prefix name of each
output file to write to. If --ipfix is present and set to spread,
then OUTPUT_SPECIFIER should be set to the name of the Spread
daemon to connect to (See below examples of spread daemon names).
Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will
be written; the string - may be used to write to standard output
(the default).
Examples
Output to file
"--out flows.yaf"
Output to collector on port 18000 at IP address 1.2.3.4
"--out 1.2.3.4 --ipfix-port 18000 --ipfix tcp"
Connect to the Spread daemon named "4803" on the local machine
"--out 4803 or --out 4803@localhost"
Connect to the machine identified by the domain name
"host.domain.edu" on port 4803.
"--out 4803@host.domain.edu"
Connect to the machine identified by the IP address "x.y.123.45" on
port 18000.
"--out x.y.123.45 --ipfix-port 18000"
--ipfix TRANSPORT_PROTOCOL
If present, causes yaf to operate as an IPFIX exporter, sending
IPFIX Messages via the specified transport protocol to the
collector (e.g., SiLK's rwflowpack or flowcap facilities) named in
the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp,
udp, sctp, and spread; sctp is only available if yaf was built with
SCTP support; spread is only available if yaf was built with Spread
support. UDP is not recommended, as it is not a reliable transport
protocol, and cannot guarantee delivery of messages. As per the
recommendations in RFC 5101, yaf will retransmit templates three
times within the template timeout period (configurable using
--udp-temp-timeout or by default, 10 minutes). Use the
--ipfix-port, --tls, --tls-ca, --tls-cert, --tls-key, and --group
options to further configure the connection to the IPFIX collector.
--rotate ROTATE_DELAY
If present, causes yaf to write output to multiple files, opening a
new output file every ROTATE_DELAY seconds in the input data.
Rotated files are named using the prefix given in the
OUTPUT_SPECIFIER, followed by a suffix containing a timestamp in
"YYYYMMDDhhmmss" format, a decimal serial number, and the file
extension .yaf.
--lock
Use lockfiles for concurrent file access protection on output
files. This is recommended for interoperating with the Airframe
filedaemon facility.
--stats INTERVAL
If present, causes yaf to export process statistics every INTERVAL
seconds. The default value for INTERVAL is 300 seconds or every 5
minutes. yaf uses IPFIX Options Templates and Records to export
flow, fragment, and decoding statistics. If INTERVAL is set to
zero, stats will not be exported.
--no-stats
If present, yaf will not export process statistics. yaf uses IPFIX
Options Templates and Records to export flow, fragment, and
decoding statistics. --no-stats takes precedence over --stats.
--no-output
If present, yaf will not export IPFIX data. It will ignore any
argument provided to --out.
Decoder Options
These options are used to modify the yaf packet decoder's behavior.
None of these options are required; the default behavior for each
option when not present is noted.
--no-frag
If present, ignore all fragmented packets. By default, yaf will
reassemble fragments with a 30 second fragment timeout.
--max-frags FRAG_TABLE_MAX
If present, limit the number of outstanding, not-yet reassembled
fragments in the fragment table to FRAG_TABLE_MAX by prematurely
expiring fragments from the table. This option is provided to limit
yaf resource usage when operating on data from very large networks
or networks with abnormal fragmentation. The fragment table may
exceed this limit slightly due to limits on how often yaf prunes
the fragment table (every 5 seconds). By default, there is no
fragment table limit, and the fragment table can grow to resource
exhaustion.
--ip4-only
If present, ignore all IPv6 packets and export IPv4 flows only.
The default is to process both IPv4 and IPv6 packets.
--ip6-only
If present, ignore all IPv4 packets and export IPv6 flows only.
The default is to process both IPv4 and IPv6 packets.
--gre-decode
If present, attempt to decode GRE version 0 encapsulated packets.
Flows will be created from packets within the GRE tunnels.
Undecodeable GRE packets will be dropped. Without this option, GRE
traffic is exported as IP protocol 47 flows. This option is
presently experimental.
Flow Table Options
These options are used to modify the flow table behavior within yaf.
None of these options are required; the default behavior for each
option when not present is noted.
--idle-timeout IDLE_TIMEOUT
Set flow idle timeout in seconds. Flows are considered idle and
flushed from the flow table if no packets are received for
IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds
(5 minutes). Setting IDLE_TIMEOUT to 0 creates a flow for each
packet.
--active-timeout ACTIVE_TIMEOUT
Set flow active timeout in seconds. Any flow lasting longer than
ACTIVE_TIMEOUT seconds will be flushed from the flow table. The
default flow active timeout is 1800 seconds (30 minutes).
--udp-temp-timeout TEMPLATE_TIMEOUT
Set UDP template timeout in seconds if --ipfix is set to udp. As
per RFC 5101 recommendations, yaf will attempt to export templates
three times within TEMPLATE_TIMEOUT. The default template timeout
period is 600 seconds (10 minutes).
--max-payload PAYLOAD_OCTETS
If present, capture at most PAYLOAD_OCTETS octets from the start of
each direction of each flow. Non-TCP flows will only capture
payload from the first packet unless --udp-payload is set. If not
present, yaf will not attempt to capture payload. Payload capture
must be enabled for payload export (--export-payload), application
labeling (--applabel), and entropy evaluation (--entropy). Note
that payload capture is still an experimental feature.
--max-flows FLOW_TABLE_MAX
If present, limit the number of open flows in the flow table to
FLOW_TABLE_MAX by prematurely expiring the flows with the least
recently received packets; this is analogous to an adaptive idle
timeout. This option is provided to limit yaf resource usage when
operating on data from large networks. By default, there is no flow
table limit, and the flow table can grow to resource exhaustion.
--udp-payload
If present, capture at most PAYLOAD_OCTETS octets fom the start of
each direction of each UDP flow, where PAYLOAD_OCTETS is set using
the --max-payload flag.
--silk
If present, export flows in "SiLK mode". As of yaf 2.0, this will
export TCP information (flags, ISN) in the main flow record instead
of within the SubTemplateMultiList. This flag must be used when
exporting to SiLK for it to collect TCP flow information. This
also introduces the following incompatibilities with standard IPFIX
export:
· totalOctetCount and reverseTotalOctetCount are clamped to 32
bits. Any packet that would cause either of these counters to
overflow 32 bits will cause the flow to close with
flowEndReason 0x02 (active timeout), and will become the first
packet of a new flow. This is analogous to forcing an active
timeout when the octet counters overflow.
· The high-order bit of the flowEndReason IE is set on any flow
created on a counter overflow, as above.
· The high-order bit of the flowEndReason IE is set on any flow
created on an active timeout.
Since this changes the semantics of the exported flowEndReason IE,
it should only be used when generating flows and exporting to
rwflowpack, flowcap, or writing files for processing with
rwipfix2silk.
--force-read-all
If present, yaf will process out-of-sequence packets. However, it
will still reject out-of-sequence fragments.
Export Options
These options are used to modify the data exported by yaf.
--export-payload
If present, export at most PAYLOAD_OCTETS (the argument to
--max-payload) octets from the start of each direction of each
flow. Non-TCP flows will only export payload from the first packet.
By default, yaf will not export flow payload.
--max-export MAX_PAY_OCTETS
If present, export at most MAX_PAY_OCTETS from the start of each
direction of each flow. MAX_PAY_OCTETS can be less than or equal
to the value given to --max-payload. By default, MAX_PAY_OCTETS is
the value given to --max-payload if --export-payload is also
present.
--uniflow
If present, export biflows using the Record Adjacency method in
section 3 of RFC 5103. This is useful when exporting to IPFIX
Collecting Processes that are not biflow-aware.
--mac
If present, export MAC-layer information; presently, exports source
and destination MAC addresses.
--force-ip6-export
If present, force IPv4 flows to be exported with IPv6-mapped IPv4
addresses in ::FFFF/96. This will cause all flows to appear to be
IPv6 flows.
--observation-domain DOMAIN_ID
Set the observationDomainID on each exported IPFIX message to the
given integer value. If not present, the observationDomainId
defaults to 0. This value is also used as the exportingProcessId
in the yaf statistics Option Record as a Scope Field.
--udp-uniflow PORT
If present, export each UDP packet on the given port (or 1 for all
ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE
(0x1F). This will not close the flow. The flow will stay open
until it closes naturally by the idle and active timeouts. Most
useful with --export-payload in order to export every UDP payload
on a specific port.
--flow-stats
If present, export extra flow attributes and statistics in the
subTemplateMultiList field. This will maintain information such as
small packet count, large packet count, nonempty packet count,
average interarrival times, total data octets, and max packet size.
See the flow statistics template below for more information about
each of the fields yaf exports.
--delta
If present, export octet and packet total counts in the delta count
information elements. octetTotalCount will be exported in
octetDeltaCount (IE 1), reverseOctetTotalCount will be exported in
reverseOctetDeltaCount. packetTotalCount will be exported in
packetDeltaCount (IE 2), and reversePacketTotalCount will be
exported in reversePacketDeltaCount.
--ingress INGRESS_INT
If present, set the ingressInterface field in the flow template to
INGRESS_INT. This field will also be populated if yaf was
configured with --enable-daginterface or --enable-napatechinterface
or --with-bivio. If yaf is running on a dag, napatech, or bivio,
and the physical interface is available, this value will override
INGRESS_INT.
--egress EGRESS_INT
If present, set the egressInterface field in the flow template to
EGRESS_INT. This field will also be populated if yaf was
configured with --enable-daginterface or --enable-napatechinterface
or --with-bivio. If yaf is running on a dag, napatech, or bivio,
and the physical interface is available, this value will override
EGRESS_INT.
Application Labeler Options
If yaf is built with application labeler support enabled (using the
--enable-applabel option to ./configure when yaf is built), then yaf
can examine packet payloads and determine the application protocol in
use within a flow, and export a 16-bit application label with each
flow.
The exported application label uses the common port number for the
protocol. For example, HTTP traffic, independent of what port the
traffic is detected on, will be labeled with a value of 80, the default
HTTP port. Labels and rules are taken from a configuration file read
at yaf startup time.
Application labeling requires payload capture to be enabled with the
--max-payload option. A minimum payload capture length of 384 octets is
recommended for best results.
Application labeling is presently experimental. SiLK does support IPFIX
import and translation of the application label via rwflowpack,
flowcap, and rwipfix2silk.
--applabel
If present, export application label data. Requires --max-payload
to enable payload capture.
--applabel-rules RULES_FILE
Read application labeler rules from RULES_FILE. If not present,
rules are read by default from
/usr/local/etc/yafApplabelRules.conf.
Entropy Measurement
If yaf is built with entropy measurement enabled (using the
--enable-entropy option to ./configure when yaf is built,) then yaf can
examine the packet payloads and determine a Shannon Entropy value for
the payload. The entropy calculation does not include the network (IP)
or transport (UDP/TCP) headers. The entropy is calculated in terms of
bits per byte, (log base 2.) The calculation generates a real number
value between 0.0 and 8.0. That number is then converted into an 8-bit
integer value between 0 and 255. Roughly, numbers above 230 are
generally compressed (or encrypted) and numbers centered around
approximately 140 are English text. Lower numbers carry even less
information content. Another useful piece of information is that
SSL/TLS tends to zero pad its packets, which causes the entropy of
those flows to drop quite low.
--entropy
If present, export the entropy values for both the forward and
reverse payloads. Requires the --max-payload option to operate.
IPFIX Connection Options
These options are used to configure the connection to an IPFIX
collector.
--ipfix-port PORT
If --ipfix is present, export flows to TCP, UDP, or SCTP port PORT.
If not present, the default IPFIX port 4739 is used. If --tls is
also present, the default secure IPFIX port 4740 is used.
--tls
If --ipfix is present, use TLS to secure the connection to the
IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS
over UDP or SCTP is not yet supported. Requires the --tls-ca,
--tls-cert, and --tls-key options to specify the X.509 certificate
and TLS key information.
--tls-ca CA_PEM_FILE
Use the Certificate Authority or Authorities in CA_PEM_FILE to
verify the remote IPFIX Collecting Process' X.509 certificate. The
connection to the Collecting Process will fail if its certificate
was not signed by this CA (or by a certificate signed by this CA,
recursively); this prevents export to unauthorized Collecting
Processes. Required if --tls is present.
--tls-cert CERT_PEM_FILE
Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX
Exporting Process. This certificate should contain the public part
of the private key in KEY_PEM_FILE. Required if --tls is present.
--tls-key KEY_PEM_FILE
Use the private key in KEY_PEM_FILE for this IPFIX Exporting
Process. This key should contain the private part of the public key
in CERT_PEM_FILE. Required if --tls is present. If the key is
encrypted, the password must be present in the YAF_TLS_PASS
environment variable.
--group SPREAD_GROUP_NAME
If --ipfix is present and set to spread, use --group to specify the
spread group name(s) to publish output. It is possible to list
more than one group name in a comma-seperated list. To use Spread
as a manifold for different types of flows, use the format GROUP,
GROUP_NAME:VALUE, GROUP_NAME:VALUE as the argument to --group and
use the --groupby switch. This list should be contained in quotes
if it contains spaces (yaf will ignore spaces in quotes). It is
suggested to use one group as the catchall for all flows (no value
listed) so flows are not lost. The --groupby switch must be used
if --group uses GROUP:VALUE format. See the Spread Documentation,
www.spread.org, for more details on Spread.
--groupby GROUPBY_TYPE
If --group is used with group values, use --groupby to specify what
type of value should be used. Options are port, vlan, applabel,
protocol, version. --groupby accepts only one argument. The port
option is destination transport port of the flow. version is the
IP version of the flow.
Privilege Options
These options are used to cause yaf to drop privileges when running as
root for live capture purposes.
--become-user UNPRIVILEGED_USER
After opening the live capture device in --live mode, drop
privilege to the named user. Using --become-user requires yaf to be
run as root or setuid root. This option will cause all files
written by yaf to be owned by the user UNPRIVILEGED_USER and the
user's primary group; use --become-group as well to change the
group yaf runs as for output purposes.
If running as root for live capture purposes and --become-user is
not present, yaf will warn that privilege is not being dropped. We
highly recommend the use of this option, especially in production
environments, for security purposes.
--become-group UNPRIVILEGED_GROUP
--become-group can be used to change the group from the default of
the user given in --become-user. This option has no effect if
given without the --become-user option as well.
PCAP Options
These options are used to turn on and configure yaf's PCAP export
capability.
--pcap PCAP_FILE_PREFIX
This option turns on rolling PCAP export in yaf. It will capture
and write packets for all network traffic yaf has received and
processed to PCAP files with the given PCAP_FILE_PREFIX. yaf will
not create file directories. If yaf can't write to the file, yaf
will turn off PCAP export. Pcap files will have names in the form
of PCAP_FILE_PREFIX[datetime]_serialno.pcap". yaf will write to a
file until the file size has reached --max-pcap or every
--pcap-timer seconds (whichever happens first). By default, yaf
rotates files every 5 MB. Files will be "locked" (".lock" will be
appended to the filename) until yaf has closed the file. Be aware
that your Operating System will have a limit on the maximum number
of files in a directory and a maximum file size. If this limit is
reached, yaf will write warning messages and terminate PCAP export.
This may effect flow generation if yaf is also writing IPFIX files.
Optionally, you can also export meta information about the flows in
each rolling PCAP file with the --pcap-meta-file switch. If --pcap
is used in conjunction with --hash and --stime, the
PCAP_FILE_PREFIX should be the name of the PCAP file to write to
(it will not be used as a file prefix).
--pcap-per-flow
If present, yaf will write a pcap file for each flow in the output
directory given to --pcap. PCAP_FILE_PREFIX given to --pcap must
be a file directory. This option is experimental and should only
be used when reading pcap files of reasonable size. yaf only
writes up to --max-payload bytes of each packet to the pcap file.
Therefore, --max-payload must be set to an appropriate size to
prevent packets from being truncated in the pcap file. yaf will
use the last three digits of the flowStartMilliseconds as the
directory and the flow key hash, flowStartMilliseconds, and serial
number as the filename. See the included getFlowKeyHash program to
easily calculate the name of the file for a given flow. When the
pcap file has reached --max-pcap size, yaf will close the file,
increment the serial number, and open a new pcap file with the same
naming convention. Note that your operating system has a limit to
the number of open file handles yaf can maintain at any given time.
Therefore, the performance of yaf degrades when the number of open
flows is greater than the maximum number of file handles.
--max-pcap MAX_FILE_MB
If present, set the maximum file size of pcap files to MAX_FILE_MB
MB. The default is 5 MB.
--pcap-timer PCAP_ROTATE_DELAY
If present, yaf will rotate rolling pcap files every
PCAP_ROTATE_DELAY seconds or when the file reaches --max-pcap size,
whichever happens first. By default, yaf only rotates files based
on file size.
--pcap-meta-file META_FILENAME
If present and --pcap is also present, yaf will export metadata on
the flows contained in each rolling pcap file yaf is writing to the
filename specified by META_FILENAME. yaf will write a line in the
form:
flow_key_hash | flowStartMilliseconds | pcap_file_name
for each flow in the pcap. If a flow exists across 3 pcap files,
there will be 3 lines in META_FILENAME for that flow (each line
having a different filename). The pcap-meta-file will rotate
approximately every 4,500,000 lines (or approx 2G). A new file
will be created in the form META_FILENAME[datetime]_serialno.meta.
This file can be uploaded to a database for flow correlation and
flow-to-pcap analysis.
If --pcap-meta-file is present and --pcap is not present, yaf will
export information about the pcap file(s) it is presently reading,
as opposed to the pcap files yaf is writing. For each packet in
the pcap file, yaf will write a line in the form:
flow_key_hash | flowStartMilliseconds | file_num | offset | length
"file_num" is the sequential file number that yaf has processed.
If yaf was given a single pcap file, this number will always be 0.
"offset" is the offset into the pcap file of the beginning of the
packet, at the start of the pcap packet header. "length" is the
length of the packet including the pcap packet header. Using this
offset, a separate program, such as yafMeta2Pcap, will be able to
quickly extract packets for a flow. This file only rotates if
META_FILE reaches max size.
--index-pcap
If present and --pcap and --pcap-meta-file are also present, export
offset and length information about the packets yaf is writing to
the rolling pcap files. Adding this option will force yaf to write
one line per packet to the pcap-meta-file in the form:
flow_key_hash | flowStartMilliseconds | pcap_file_name | offset |
length
This option is ignored if --pcap is not present.
--hash FLOW_KEY_HASH
If present, only write PCAP data for the flow(s) with
FLOW_KEY_HASH. This option is only valid with the --pcap option.
--stime FLOW_START_TIMEMS
If present, only write PCAP data for the flow(s) with
FLOW_START_TIMEMS and FLOW_KEY_HASH given to --hash. This option
is only valid when used with the --hash and --pcap options.
Logging Options
These options are used to specify how log messages are routed. yaf can
log to standard error, regular files, or the UNIX syslog facility.
--log LOG_SPECIFIER
Specifies destination for log messages. LOG_SPECIFIER can be a
syslog(3) facility name, the special value stderr for standard
error, or the absolute path to a file for file logging. The default
log specifier is stderr if available, user otherwise.
--loglevel LOG_LEVEL
Specify minimum level for logged messages. In increasing levels of
verbosity, the supported log levels are quiet, error, critical,
warning, message, info, and debug. The default logging level is
warning.
--verbose
Equivalent to --loglevel debug.
--version
If present, print version and copyright information to standard
error and exit.
Plugin Options
These options are used to load, configure, and run a yaf plugin.
--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]
Specify the plugin to load. The loaded plugin must follow the yaf
plugin framework. LIBPLUGIN_NAME must be the full path to the
plugin library name. Two plugins are included with c<yaf>, a Deep
Packet Inspection plugin, and a DHCP Fingerprinting plugin. This
flag will only be recognized if yaf is configured with
--enable-plugins. There are also configure options to export only
DNS Authoritative and NXDomain responses. Read each plugin's
documentation for more information.
--plugin-opts "OPTIONS[,OPTIONS...]"
Specify the arguments to the plugin given to --plugin-name. This
flag will only be recognized if yaf is configured with
--enable-plugins and --plugin-name is set to a valid plugin. For
example, the DPI Plugin takes the well-known port of a protocol(s)
to enable DPI (default for DPI is all protocols).
--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]
Specify the path to a configuration file for the plugin given to
--plugin-name. This flag will only be recognized if yaf is
configured with --enable-plugins and --plugin-name is set to a
valid plugin. If this switch is not used, but the plugin requires
a configuration file, the default location /usr/local/etc will be
used.
Passive OS Fingerprinting (p0f)
These options are used to enable p0f in yaf. p0f is presently
experimental. There is no support in yafscii or SiLK for printing p0f
related data. Currently, yaf uses the p0f Version 2 SYN fingerprints
(see p0f.fp).
--p0fprint
If present, export p0f data. This data consists of three related
information elements; osName, osVersion, osFingerPrint. This flag
requires yaf to be configured with --enable-p0fprinter.
--p0f-fingerprints
Location of the p0f fingerprint file(s), p0f.fp. Default is
/usr/local/etc/p0f.fp. This version of yaf includes the updated
CERT p0f fingerprints. See
<https://tools.netsa.cert.org/confluence/display/tt/p0f+fingerprints>
for updates.
--fpexport
If present, enable export of handshake headers for external OS
fingerprinters. The related information elements are
firstPacketBanner and secondPacketBanner. This flag requires yaf
to be configured with --enable-fpexporter.
OUTPUT
Basic Flow Record
yaf's output consists of an IPFIX message stream. yaf uses a variety of
templates for IPFIX data records; the information elements that may
appear in these templates are enumerated below. For further information
about the IPFIX information model and IPFIX message stream, see RFC
5102, RFC 5101, and RFC 5103. As of yaf 2.0, yaf nests some templates
in an IPFIX subTemplateMultiList. In order to retain compatibility
with the SiLK Tools, use --silk to prevent yaf from nesting TCP
Information Elements. Below are descriptions of each of the templates
yaf will export. See the Internet-Draft Export of Structured Data in
IPFIX for more information on IPFIX lists.
yaf assigns information element numbers to reverse flow elements in
biflow capture based on the standard IPFIX PEN 29305. This applies
only for information elements defined in the standard IPFIX Information
Model (RFC 5102) that do not have a reverse information element already
defined. For information elements defined under the CERT PEN, a
standard method is used to calculate their reverse element identifier.
The method is that bit fourteen is set to one in the IE field, (e.g.
16384 + the forward IE number.)
flowStartMilliseconds IE 152, 8 octets, unsigned
Flow start time in milliseconds since 1970-01-01 00:00:00 UTC.
Always present.
flowEndMilliseconds IE 153, 8 octets, unsigned
Flow end time in milliseconds since 1970-01-01 00:00:00 UTC. Always
present.
octetTotalCount IE 85, 8 octets, unsigned
Number of octets in packets in forward direction of flow. Always
present (unless "--delta" is used.) May be encoded in 4 octets
using IPFIX reduced-length encoding.
reverseOctetTotalCount Reverse (PEN 29305) IE 85, 8 octets, unsigned
Number of octets in packets in reverse direction of flow. Present
if flow has a reverse direction. May be encoded in 4 octets using
IPFIX reduced-length encoding.
packetTotalCount IE 86, 8 octets, unsigned
Number of packets in forward direction of flow. Always present
(unless "--delta" is used.) May be encoded in 4 octets using IPFIX
reduced-length encoding.
reversePacketTotalCount Reverse (PEN 29305) IE 86, 8 octets, unsigned
Number of packets in reverse direction of flow. Present if flow has
a reverse direction. May be encoded in 4 octets using IPFIX
reduced-length encoding.
octetDeltaCount IE 1, 8 octets, unsigned
Number of octets in packets in forward direction of flow. Only
present if "--delta" is used. May be encoded in 4 octets using
IPFIX reduced-length encoding.
reverseOctetDeltaCount Reverse (PEN 29305) IE 1, 8 octets, unsigned
Number of octets in reverse direction of flow. Only present if
"--delta" is used and non-zero. May be encoded in 4 octets using
IPFIX reduced-length encoding.
packetDeltaCount IE 2, 8 octets, unsigned
Number of packets in forward direction of flow. Only present if
"--delta" is used. May be encoded in 4 octets using IPFIX reduced-
length encoding.
reversePacketDeltaCount Reverse (PEN 29305) IE 2, 8 octets, unsigned
Number of packets in reverse direction of flow. Only present if
"--delta" is used and non-zero. May be encoded in 4 octets using
IPFIX reduced-length encoding.
reverseFlowDeltaMilliseconds CERT (PEN 6871) IE 21, 4 octets, unsigned
Difference in time in milliseconds between first packet in forward
direction and first packet in reverse direction. Correlates with
(but does not necessarily represent) round-trip time. Present if
flow has a reverse direction.
sourceIPv4Address IE 8, 4 octets, unisigned
IPv4 address of flow source or biflow initiator. Present for IPv4
flows without IPv6-mapped addresses only.
destinationIPv4Address IE 12, 4 octets, unsigned
IPv4 address of flow source or biflow responder. Present for IPv4
flows without IPv6-mapped addresses only.
sourceIPv6Address IE 27, 16 octets, unsigned
IPv6 address of flow source or biflow initiator. Present for IPv6
flows or IPv6-mapped IPv4 flows only.
destinationIPv6Address IE 28, 16 octets, unsigned
IPv6 address of flow source or biflow responder. Present for IPv6
flows or IPv6-mapped IPv4 flows only.
sourceTransportPort IE 7, 2 octets, unsigned
TCP or UDP port on the flow source or biflow initiator endpoint.
Always present.
destinationTransportPort IE 11, 2 octets, usigned
TCP or UDP port on the flow destination or biflow responder
endpoint. Always present. For ICMP flows, contains ICMP type * 256
+ ICMP code. This is non-standard, and an open issue in yaf.
flowAttributes CERT (PEN 6871) IE 40, 2 octets, unsigned
Miscellaneous flow attributes for the forward direction of the
flow. Always present (yaf 2.1 or later). Current flag values:
Bit 1: All packets in the forward direction have fixed size
For TCP flows, only packets that have payload will be considered
(to avoid TCP handshakes and teardowns).
Bit 2: Packet(s) in the forward direction was received out-of-
sequence
Bit 3: Host may be MP_CAPABLE (MPTCP-capable)
For TCP flows, this bit will be set if a packet in the flow was
seen that had the MP_CAPABLE TCP option or attempted an MP_JOIN
operation.
reverseFlowAttributes CERT (PEN 6871) IE 16424, 2 octets, unsigned
Miscellaneous flow attributes for the reverse direction of the
flow. Always present (yaf 2.1 or later). Current flag values:
Bit 1: All packets in the reverse direction have fixed size
Bit 2: Packet(s) in the reverse direction was received out-of-
sequence
Bit 3: Host may be MP_CAPABLE (MPTCP-capable)
For TCP flows, this bit will be set if a packet in the flow was
seen that had the MP_CAPABLE TCP option or attempted an MP_JOIN
operation.
protocolIdentifier IE 4, 1 octet, unsigned
IP protocol of the flow. Always present.
flowEndReason IE 136, 1 octet, unsigned
Flow end reason code, as defined by the IPFIX Information Model.
Always present. In --silk mode, the high-order bit is set if the
flow was created by continuation.
silkAppLabel CERT (PEN 6871) IE 33, 2 octets, unsigned
Application label, defined as the primary well-known port
associated with a given application. Present if the application
labeler is enabled, and was able to determine the application
protocol used within the flow.
vlanId IE 58, 2 octets, unsigned
802.1q VLAN tag of the first packet in the forward direction of the
flow.
reverseVlanId Reverse (PEN 29305) IE 58, 2 octets, unsigned
802.1q VLAN tag of the first packet in the reverse direction of the
flow. Present if the flow has a reverse direction.
ingressInterface IE 10, 4 octets, unsigned
The index of the IP interface where packets of this flow are being
received. Use --ingress, --napatech-interface, --dag-interface or
configure yaf with bivio for this field to be present in the flow
template. Use --ingress to manually set this field.
egressInterface IE 14, 4 octets, unsigned
The index of the IP interface where packets of this flow are being
received. Use --egress, --napatech-interface, --dag-interface or
configure yaf with bivio for this field to be present in the flow
template. If using napatech, dag, or bivio, "egressinterface" will
be the physical interface | 0x100. Use --egress to manually set
this field.
ipClassOfService IE 5, 1 octet, unsigned
For IPv4 packets, this is the value of the TOS field in the IPv4
header. For IPv6 packets, this is the Traffic Class field in the
IPv6 header.
reverseIpClassOfService Reverse (PEN 29305) IE 5, 1 octet, unsigned
The TOS field in the IPv4 header for packets in the reverse
direction, and Traffic Class field in the IPv6 header for packets
in the reverse direction.
mplsTopLabelStackSection IE 70, 3 octets
The MPLS Label from the top of the MPLS label stack entry. yaf
does not include the Experimental bits and Bottom of the Stack bit
in the export field. yaf must have been enabled with MPLS support
for export of this field. Note that this field is defined as an
octet array in the default libfixbuf Information Model. yaf uses
the length override feature in libfixbuf to redefine it from
variable length to 3 bytes.
mplsLabelStackSection2, IE 71, 3 octets
The MPLS Label from the MPLS label stack entry immediately before
the top entry. yaf does not include the Experimental bits and
Bottom of the Stack bit in the export field. yaf must have been
enabled with MPLS support for export of this field. Note that this
field is defined as an octet array in the default libfixbuf
Information Model. yaf uses the length override feature in
libfixbuf to redefine it from variable length to 3 bytes.
mplsLabelStackSection3, IE 72, 3 octets
The MPLS Label from the third entry in the MPLS label stack. yaf
does not include the Experimental bits and Bottom of the Stack bit
in the export field. yaf must have been enabled with MPLS support
for export of this field. Note that this field is defined as an
octet array in the default libfixbuf Information Model. yaf uses
the length override feature in libfixbuf to redefine it from
variable length to 3 bytes.
subTemplateMultiList IE 293, variable length
Represents a list of zero or more instances of a structured data
type, where the data type of each list element can be different and
corresponds with different template definitions. The Information
Element Number will change upon updates to the IPFIX lists
specification and libfixbuf releases.
TCP Flow Template
The following six Information Elements will be exported as a template
within the subTemplateMultiList unless --silk is used.
tcpSequenceNumber IE 184, 4 octets, unsigned
Initial sequence number of the forward direction of the flow.
Present if the flow's protocolIdentifier is 6 (TCP). This element
is contained in the yaf TCP template within the
subTemplateMultiList unless --silk is used.
reverseTcpSequenceNumber Reverse (PEN 29305) IE 184, 4 octets, unsigned
Initial sequence number of the reverse direction of the flow.
Present if the flow's protocolIdentifier is 6 (TCP) and the flow
has a reverse direction. This element is contained in the yaf TCP
template within the subTemplateMultiList unless --silk is used.
initialTCPFlags CERT (PEN 6871) IE 14, 1 octet, unsigned
TCP flags of initial packet in the forward direction of the flow.
Present if the flow's protocolIdentifier is 6 (TCP). This element
is contained in the yaf TCP template within the
subTemplateMultiList unless --silk is used.
unionTCPFlags CERT (PEN 6871) IE 15, 1 octet, unsigned
Union of TCP flags of all packets other than the initial packet in
the forward direction of the flow. Present if the flow's
protocolIdentifier is 6 (TCP). This element is contained in the yaf
TCP template within the subTemplateMultiList unless --silk is used.
reverseInitialTCPFlags CERT (PEN 6871) IE 16398, 1 octet, unsigned
TCP flags of initial packet in the reverse direction of the flow.
Present if the flow's protocolIdentifier is 6 (TCP) and the flow
has a reverse direction. This element is contained in the yaf TCP
template within the subTemplateMultiList unless --silk is used.
reverseUnionTCPFlags CERT (PEN 6871) IE 16399, 1 octet, unsigned
Union of TCP flags of all packets other than the initial packet in
the reverse direction of the flow. Present if the flow's
protocolIdentifier is 6 (TCP) and the flow has a reverse direction.
This element is contained in the yaf TCP template within the
subTemplateMultiList unless --silk is used.
MPTCP Template
The following five Information Elements will be exported as a template
within the subTemplateMultiList if any MPTCP options are seen.
mptcpInitialDataSequenceNumber, CERT (PEN 6871) IE 289, 8 octets,
unsigned
The initial data sequence number found in the MPTCP Data Sequence
Signal (DSS) Option.
mptcpReceiverToken, CERT (PEN 6871) IE 290, 4 octets, unsigned
The token used to identify an MPTCP connection over multiple
subflows. This value is found in the MP_JOIN TCP Option for the
initial SYN of a subflow.
mptcpMaximumSegmentSize, CERT (PEN 6871) IE 291, 2 octets, unsigned
The maximum segment size reported in the Maximum Segment Size TCP
Option. This should be consistent over all subflows.
mptcpAddressID, CERT (PEN 6871), IE 292, 1 octet, unsigned
The address ID of the subflow found in the SYN/ACK of an MP_JOIN
operation.
mptcpFlags, CERT (PEN 6871), IE 293, 1 octet, unsigned
Various MPTCP Values:
Bit 1: Priority was changed during the life of the subflow (MP_PRIO
was seen)
Bit 2: Subflow has priority at setup (backup flag was not set at
initialization).
Bit 3: Subflow failed. (MP_FAIL option was seen).
Bit 4: Subflow experienced fast close. (MP_FASTCLOSE options was
seen).
MAC Flow Template
The following two Information Elements will be exported as a template
within the subTemplateMultiList.
sourceMacAddress, IE 56, 6 octets, unsigned
Source MAC Address of the first packet in the forward direction of
the flow. This element is contained in the yaf MAC template within
the subTemplateMultiList.
destinationMacAddress, IE 80, 6 octets, unsigned
Destination MAC Address of the first packet in the reverse
direction of the flow. This element is contained in the yaf MAC
template within the subTemplateMultiList.
Payload Flow Template
The following two Information Elements will be exported as a template
within the subTemplateMultiList.
payload CERT (PEN 6871) IE 18, variable-length
Initial n bytes of forward direction of flow payload. Present if
payload collection is enabled and payload is present in the forward
direction of the flow. This element is contained in the yaf Payload
template within the subTemplateMultiList.
reversePayload CERT (PEN 6871) IE 16402, variable-length
Initial n bytes of reverse direction of flow payload. Present if
payload collection is enabled and payload is present in the reverse
direction of the flow. This element is contained in the yaf Payload
template within the subTemplateMultiList.
Entropy Flow Template
The following two Information Elements will be exported as a template
within the subTemplateMultiList.
payloadEntropy CERT (PEN 6871) IE 35, 1 octet, unsigned
Shannon Entropy calculation of the forward payload data. This
element is contained in the yaf Entropy template within the
subTemplateMultiList.
reversePayloadEntropy CERT (PEN 6871) IE 16419, 1 octet, unsigned
Shannon Entropy calculation of the reverse payload data. This
element is contained in the yaf Entropy template within the
subTemplateMultiList.
p0f Flow Template
The following six Information Elements will be exported as a template
within the subTemplateMultiList if present and only if p0f is enabled.
osName CERT (PEN 6871) IE 36, variable-length
p0f OS Name for the forward flow based on the SYN packet and p0f
SYN Fingerprints. Present only if p0f is enabled. This element is
contained in the yaf p0f template within the subTemplateMultiList.
reverseOsName CERT (PEN 6871) IE 16420, variable-length
p0f OS Name for the reverse flow based on the SYN packet and p0f
SYN Fingerprints. Present only if p0f is enabled. This element is
contained in the yaf p0f template within the subTemplateMultiList.
osVersion CERT (PEN 6871) IE 37, variable-length
p0f OS Version for the forward flow based on the SYN packet and p0f
SYN Fingerprints. Present only if p0f is enabled. This element is
contained in the yaf p0f template within the subTemplateMultiList.
reverseOsVersion CERT (PEN 6871) IE 16421, variable-length
p0f OS Version for the reverse flow based on the SYN packet and p0f
SYN fingerprints. Present only if p0f is enabled. This element is
contained in the yaf p0f template within the subTemplateMultiList.
osFingerPrint CERT (PEN 6871) IE 107, variable-length
p0f OS Fingerprint for the forward flow based on the SYN packet and
p0f SYN fingerprints. Present only if p0f is enabled. This element
is contained in the yaf p0f template within the
subTemplateMultiList.
reverseOsFingerPrint CERT (PEN 6871) IE 16491, variable-length
p0f OS Fingerprint for the reverse flow based on the SYN packet and
p0f SYN Fingerprints. Present only if p0f is enabled. This element
is contained in the yaf p0f template within the
subTemplateMultiList.
Fingerprint Exporting Template
The following four Information Elements will be exported as a template
within the subTemplateMultiList if present and only if fpexport is
enabled.
firstPacketBanner CERT (PEN 6871) IE 38, variable-length
IP and transport headers for first packet in forward direction to
be used for external OS Fingerprinters. Present only if fpexport
is enabled. This element is contained in the yaf FPExport template
within the subTemplateMultiList.
reverseFirstPacketBanner CERT (PEN 6871) IE 16422, variable-length
IP and transport headers for first packet in reverse direction to
be used for external OS Fingerprinters. Present only if fpexport
is enabled. This element is contained in the yaf FPExport template
within the subTemplateMultiList.
secondPacketBanner CERT (PEN 6871) IE 39, variable-length
IP and transport headers for second packet in forward direction
(third packet in sequence) to be used for external OS
Fingerprinters. Present only if fpexport is enabled. This element
is contained in the yaf FPExport template within the
subTemplateMultiList.
reverseSecondPacketBanner CERT (PEN 6871) IE 16423, variable-length
IP and transport headers for second packet in reverse direction
(currently not used). Present only if fpexport is enabled. This
element is contained in the yaf FPExport template within the
subTemplateMultiList.
Hooks Templates
yaf can export other templates within the subTemplateMultiList if
plugins are enabled in yaf. See yafdpi(1) for descriptions of the yaf
Deep Packet Inspection Information Elements. See yafdhcp(1) for
descriptions of the DHCP Fingerprint Information Elements.
Flow Statistics Template
yaf can maintain and export more information about each flow than what
is exported in the Basic Flow Template. If yaf is run with
--flow-statsyaf will export the following attributes with every flow
as long as one of the following characteristics is nonzero. The
following flow attributes have been known to help in traffic
classification.
dataByteCount CERT (PEN 6871) IE 502, 8 octets, unsigned
Total bytes transferred as payload.
averageInterarrivalTime CERT (PEN 6871) IE 503, 8 octets, unsigned
Average number of milliseconds between packets.
standardDeviationInterarrivalTime CERT (PEN 6871) IE 504, 8 octets,
unsigned
Standard deviation of the interarrival time for up to the first ten
packets.
tcpUrgTotalCount IE 223, 4 octets, unsigned
The number of TCP packets that have the URGENT Flag set.
smallPacketCount CERT (PEN 6871) IE 500, 4 octets, unsigned
The number of packets that contain less than 60 bytes of payload.
nonEmptyPacketCount CERT (PEN 6871) IE 501, 4 octets, unsigned
The number of packets that contain at least 1 byte of payload.
largePacketCount CERT (PEN 6871) IE 510, 4 octets, unsigned
The number of packets that contain at least 220 bytes of payload.
firstNonEmptyPacketSize CERT (PEN 6871) IE 505, 2 octets, unsigned
Payload length of the first non-empty packet.
maxPacketSize CERT (PEN 6871) IE 506, 2 octets, unsigned
The largest payload length transferred in the flow.
standardDeviationPayloadLength CERT (PEN 6871) IE 508, 2 octets,
unsigned
The standard deviation of the payload length for up to the first 10
non empty packets.
firstEightNonEmptyPacketDirections CERT (PEN 6871) IE 507, 1 octet,
unsigned
Represents directionality for the first 8 non-empty packets. 0 for
forward direction, 1 for reverse direction.
reverseDataByteCount CERT (PEN 6871) IE 16886, 8 octets, unsigned
Total bytes transferred as payload in the reverse direction.
reverseAverageInterarrivalTime CERT (PEN 6871) IE 16887, 8 octets,
unsigned
Average number of milliseconds between packets in reverse
direction.
reverseStandardDeviationInterarrivalTime CERT (PEN 6871) IE 16888, 8
octets, unsigned
Standard deviation of the interarrival time for up to the first ten
packets in the reverse direction.
reverseTcpUrgTotalCount Reverse (PEN 29305), IE 223, 4 octets, unsigned
The number of TCP packets that have the URGENT Flag set in the
reverse direction.
reverseSmallPacketCount CERT (PEN 6871) IE 16884, 4 octets, unsigned
The number of packets that contain less than 60 bytes of payload in
reverse direciton.
reverseNonEmptyPacketCount CERT (PEN 6871) IE 16885, 4 octets, unsigned
The number of packets that contain at least 1 byte of payload in
reverse direction.
reverseLargePacketCount CERT (PEN 6871) IE 16894, 4 octets, unsigned
The number of packets that contain at least 220 bytes of payload in
the reverse direction.
reverseFirstNonEmptyPacketSize CERT (PEN 6871) IE 16889, 2 octets,
unsigned
Payload length of the first non-empty packet in the reverse
direction.
reverseMaxPacketSize CERT (PEN 6871) IE 16890, 2 octets, unsigned
The largest payload length transferred in the flow in the reverse
direction.
reverseStandardDeviationPayloadLength CERT (PEN 6871) IE 16892, 2
octets, unsigned
The standard deviation of the payload length for up to the first 10
non empty packets in the reverse direction.
Statistics Option Template
yaf will export information about its process periodically using IPFIX
Options Template Record. This record gives information about the
status of the flow and fragment table, as well as decoding information.
This can be turned off using the --no-stats option. The following
Information Elements will be exported:
systemInitTimeMilliseconds IE 161, 8 octets, unsigned
The time in milliseconds of the last (re-)initialization of yaf.
exportedFlowRecordTotalCount IE 42, 8 octets, unsigned
Total amount of exported flows from yaf start time.
packetTotalCount IE 86, 8 octets, unsigned
Total amount of packets processed by yaf from yaf start time.
droppedPacketTotalCount IE 135, 8 octets, unsigned
Total amount of dropped packets according to statistics given by
libpcap, libdag, or the Napatech or Netronome APIs.
ignoredPacketTotalCount IE 164, 8 octets, unsigned
Total amount of packets ignored by the yaf packet decoder, such as
unsupported packet types and incomplete headers, from yaf start
time.
notSentPacketTotalCount IE 167, 8 octets, unsigned
Total amount of packets rejected by yaf because they were received
out of sequence.
expiredFragmentCount CERT (PEN 6871) IE 100, 4 octets, unsigned
Total amount of fragments that have been expired since yaf start
time.
assembledFragmentCount CERT (PEN 6871) IE 101, 4 octets, unsigned
Total number of packets that been assembled from a series of
fragments since yaf start time.
flowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets, unsigned
Total number of times the yaf flow table has been flushed since yaf
start time.
flowTablePeakCount CERT (PEN 6871) IE 105, 4 octets, unsigned
The maximum number of flows in the yaf flow table at any one time
since yaf start time.
exporterIPv4Address IE 130, 4 octets, unsigned
The IPv4 Address of the yaf flow sensor.
exportingProcessId IE 144, 4 octets, unsigned
Set the ID of the yaf flow sensor by giving a value to
--observation-domain. The default is 0.
meanFlowRate CERT (PEN 6871) IE 102, 4 octets, unsigned
The mean flow rate of the yaf flow sensor since yaf start time,
rounded to the nearest integer.
meanPacketRate CERT (PEN 6871) IE 103, 4 octets, unsigned
The mean packet rate of the yaf flow sensor since yaf start time,
rounded to the nearest integer.
SIGNALSyaf responds to SIGINT or SIGTERM by terminating input processing,
flushing any pending flows to the current output, and exiting. If
--verbose is given, yaf responds to SIGUSR1 by printing present flow
and fragment table statistics to its log. All other signals are
handled by the C runtimes in the default manner on the platform on
which yaf is currently operating.
EXAMPLES
To generate flows from an pcap file into an IPFIX file:
"yaf --in packets.pcap --out flows.yaf"
To capture flows from a pcap interface and export them to files in the
current directory rotated hourly:
"yaf --live pcap --in en1 --out en1_capture --rotate 3600"
To capture flows from an Endace DAG card and export them via IPFIX over
TCP:
"yaf --live dag --in dag0 --ipfix tcp --out my-collector.example.com"
To capture flows from a Napatech Adapter card using stream ID 20 and
export them via IPFIX over UDP:
"yaf --live napatech --in nt3g20 --ipfix udp --out localhost
--ipfix-port 18000"
To capture flows from a Netronome NFE card and export to a file:
"yaf --live netronome --in 0:0 --out /data/yaf/myipfix.yaf"
To convert a pcap formatted packet capture into IPFIX:
"yaf <packets.pcap >flows.yaf"
To publish to spread group TST_SPRD for a spread daemon running locally
on port 4803:
"yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group
TST_SPRD"
To publish to spread groups based on application label for spread
daemon running locally on port 4803:
"yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group
"SPRD_CATCHALL, SPRD_DNS:53, SPRD_HTTP:80, SPRD_SMTP:25" --groupby
applabel --applabel --max-payload=400"
To run yaf with application labeling enabled and export via IPFIX over
TCP:
"yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix tcp
--ipfix-port=18001 --applabel
--applabel-rules=/usr/local/etc/yafApplabelRules.conf
--max-payload=300"
To run yaf with BPF on UDP Port 53
"yaf --live pcap --in en1 --out /path/to/dst/ --rotate 120
--filter="udp port 53""
To run yaf with Deep Packet Inspection enabled for HTTP, IMAP, and DNS:
"yaf --in packets.pcap --out flows.yaf --applabel --max-payload=400
--plugin-name=/usr/local/lib/dpacketplugin.la --plugin-opts="80 143
53""
To run yaf with Deep Packet Inspection and DHCP Fingerprinting:
"yaf --in packets.pcap --out flows.yaf --applabel --max-payload=1000
--plugin-name=/usr/local/lib/dpacketplugin.la,/usr/local/lib/dhcp_fp_plugin.la"
To run yaf with pcap generation:
"yaf --in eth0 --live pcap --out localhost --ipfix tcp
--ipfix-port=18001 --pcap /data/pcap --pcap-meta-file=/data/pcap_info"
To generate a pcap file for one particular flow in a pcap:
"yaf --in packets.pcap --no-output --max-payload=2000 --pcap
/data/oneflow.pcap --hash 2181525080 --stime 1407607897547"
KNOWN ISSUES
YAF BPF Filtering is ignored when using --live dag, napatech, or
netronome because libpcap is not used.
YAF PCAP Export options are ignored when using --live dag, napatech, or
netronome.
YAF requires libfixbuf 1.7.0 or later.
YAF 2.0 must be used with an IPFIX Collecting Process that can handle
IPFIX lists elements, especially the subTemplateMultiList Information
Element in order to retrieve certain flow information. Older versions
of YAF can read YAF 2.0 flow files, but will ignore anything contained
in the subTemplateMultiList.
The plugin infrastructure has been modified in YAF 2.0 to export
templates in YAF's subTemplateMultiList element.
YAF 2.0 will export statistics in an Options Template and Options Data
Records unless the --no-stats switch is given. The IPFIX Collecting
Process should be able to differentiate between options records and
flow records in order to prevent incorrect transcoding of statistics
records into flow records.
YAF will not rotate output files if it is not seeing any flow data.
However, it will continue to write process statistics messages at the
configured interval time to the most recent output file.
When using PF_RING ZC with yaf, a load balancing program is required.
See yafzcbalance(1) for more information.
When running yaf with --live=pfring or --live=zc, the call to receive
packets is blocking so yaf will not export statistics messages or
respond to SIGUSR1 signals unless it is receiving data.
Presently, the destinationTransportPort information element contains
ICMP type and code information for ICMP or ICMP6 flows; this is
nonstandard and may not be interoperable with other IPFIX
implementations.
Bug reports and feature requests may be sent via email to
<netsa-help@cert.org>.
AUTHORS
Brian Trammell, Chris Inacio, Michael Duggan, Emily Sarneso, Dan Ruef,
and the CERT Network Situational Awareness Group Engineering Team,
<http://www.cert.org/netsa>.
SEE ALSOyafscii(1), tcpdump(1), pcap(3), nafalize(1), yafzcbalance(1), Spread
Documentation at www.spread.org, libp0f at
<https://tools.netsa.cert.org/confluence/display/tt/libp0f>, and the
following IETF Internet RFCs: Specification of the IPFIX Protocol for
the Exchange of IP Traffic Flow Information RFC 5101, Information Model
for IP Flow Information Export RFC 5102, Bidirectional Flow Export
using IPFIX RFC 5103, Export of Structured Data in IPFIX RFC 6313
2.8.0 19-Feb-2016 YAF(1)
