rwscanquery(1) SiLK Tool Suite rwscanquery(1)NAMErwscanquery - Query the network scan database
SYNOPSISrwscanquery [options]
Report Options:
--report=REPORT_TYPE Select query and output options. Values
for REPORT_TYPE are standard, volume,
scanset, scanflows, respflows, and export
--start-date=YYYY/MM/DD:HH Report on scans active after this date.
--end-date=YYYY/MM/DD:HH Defaults to start-date.
--saddress=ADDR_SPEC Show scans originating from matching hosts.
--sipset=IPSET_FILE Show scans originating from hosts in set.
--daddress=IP_WILDCARD Show only scans targeting matching hosts.
--dipset=IPSET_FILE Show only scans targeting hosts in set.
--show-header Display column titles at start of output.
--columnar Display more human-readable columnar view.
--output-path=PATH Write results to the specified file.
Configuration Options:
--database=DBNAME Query an alternate scan database
Help Options:
--help Display this brief help message.
--man Display the full documentation.
--version Display the version information.
DESCRIPTIONrwscanquery queries the network scan database---that is, the database
that contains scans found by rwscan(1). The type of output rwscanquery
creates is controlled by the --report switch as described in the
"Report Options" section below. rwscanquery writes its output to the
location specified by the --output-path switch or to the standard
output when that switch is not provided.
rwscanquery runs a query of the scan database and then, depending on
the report type, either displays the result set as text or creates a
binary SiLK from the result set. The database rows that are part of
the result set may be limited by using the --start-date, --end-date,
--saddress, and --sipset switches. The result set is always limited to
a time window, and the current day is used when no --start-date is
given.
The following three report types produce textual output. The default
output displays the values separated by a vertical bar ("|") with no
spacing. The --columnar switch causes the output to appear in columns
with a space-delimiter between the columns. The output includes no
title line unless the --show-header switch is specified.
· The "standard" report contains most of the columns in the database
for the rows in the result set. (The columns containing the scan
model and scan probability are not included.)
· The "volume" report groups the rows in the result set by day and
shows sums the flows, packets, and bytes columns for each day.
· The "export" report contains all the columns in the database for
the rows in the result set, and the rows are displayed in a format
compatible with rwscan.
The following three report types create a binary SiLK file as their
result. These report types invoke other SiLK tools (namely
rwfilter(1), rwset(1), rwsetbuild(1), and rwsetcat(1)) and the report
types assume rwfilter has access to a SiLK data repository.
The first step in all three of these report types is for rwscanquery to
get the distinct IP addresses for the rows in the result set and pass
them into rwsetbuild to create a temporary IPset file containing the
scanning IPs.
· A "scanflows" report produces a file of SiLK Flow records whose
source IP is a scanning IPs. rwscanquery uses the temporary IPset
as an argument to rwfilter to find flow records in your data
repository that originated from the scanning IPs within the time
window. You may choose to limit the report to particular IPs
targeted by the scanning IPs by specifying the --daddress or
--dipset switches. The output from rwfilter is the output of the
report. The rwfilter invocation uses the configuration values
"rw_in_class" and "rw_in_type" if they are specified in the
configuration file (c.f. "CONFIGURATION").
· A "respflows" report produces a file of SiLK Flow records whose
destination IP is a scanning IP. These flow records may represent
responses to a scan. To create this report, rwscanquery performs
steps similar to those for the "scanflows" report except the
direction of the rwfilter command is reversed to find flow records
going to the scanning IPs. You may choose to limit the report to
particular IPs that responded to the scan by specifying the
--daddress or --dipset switches. The output from rwfilter is the
output of the report. The rwfilter invocation uses the
configuration values "rw_out_class" and "rw_out_type" if they are
specified in the configuration file (c.f. "CONFIGURATION").
· A "scanset" report produces a binary IPset file.
· If neither the --daddress nor --dipset switches are specified,
the output of the this report is the temporary IPset file
containing the scanning IPs; that is, all the scanning IPs in
the time window.
· Otherwise, rwscanquery performs the same steps it does as when
creating "scanflows" report. Next, instead of returning the
output from rwfilter, rwscanquery passes the flow records into
rwset to create an IPset file containing the scanning IPs that
targeted particular IP addresses.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option. A parameter to an option may be specified
as --arg=param or --arg param, though the first form is required for
options that take optional parameters.
Report Options
--report=TYPE
Specify the query and the type of output to create. When this
switch is not specified, the default is a "standard" report. The
supported values for TYPE are:
standard
Write one textual line of output for each scan record in the
scan database. By default, the output has no titles and it is
not in columnar form. Specify the --show-header and/or
--columnar switches to make the output more human readable.
volume
Write a daily scan activity volume summary report for each day
within the time period. By default, the output has no titles
and it is not in columnar form. Specify the --show-header
and/or --columnar switches to make the output more human
readable.
scanset
Write an IPset file containing the IP addresses which were the
sources of scan activity during the selected time period. The
output of this report type is binary, so you must redirect or
pipe the output to a location or specify the --output-path
switch.
scanflows
Write a SiLK Flow file containing all flows originating from
scanning IP addresses within the specified time period. This
flow data includes flows originating from any host that would
be listed as a scan source by your query, from any time within
the time period specified by --start-date and --end-date. Note
that this may include flows that were not identified by the
scan analysis as being part of a scan. The output of this
report type is binary, so you must redirect or pipe the output
to a location or specify the --output-path switch.
respflows
Write a SiLK Flow file containing all flows sent to scanning IP
addresses within the specified time period---that is, possible
responses to the scanners. The output of this report type is
binary, so you must redirect or pipe the output to a location
or specify the --output-path switch.
export
Write textual output consistent with the output format of the
rwscan(1) tool. Specify the --show-header switch to include a
title line.
--start-date=YYYY/MM/DD:HH
Display scans which were active after this hour. When this
argument contains a date with no hour and no --end-date switch is
specified, scans for that entire day are returned. If this switch
is not specified at all, scans for the current day (based on the
local time on the host machine) are returned.
--end-date=YYYY/MM/DD:HH
Display scans which were active before the end of this hour. If no
end-date is given, defaults to the same as start-date. It is an
error to provide an end-date without a start-date.
--saddress=ADDR_SPEC
Display scans originating from hosts described in ADDR_SPEC, where
ADDR_SPEC is a list of addresses, address ranges, and CIDR blocks.
Only scans originating from hosts in the list are displayed.
--sipset=IPSET_FILE
Display scans originating from hosts in IPSET_FILE, where
IPSET_FILE is a standard SiLK IPset file as created by rwset(1) or
rwsetbuild(1). Note that a very complex IPset may take a long time
to process, or even fail to return any results.
--daddress=IP_WILDCARD
Display scans targeting hosts described in IP_WILDCARD, where
IP_WILDCARD is a single IP address, a single CIDR block, or an IP
Wildcard expression accepted by rwfilter(1). To match on multiple
IPs or networks, use the --dipset switch. This switch is ignored
for --report types other than "scanset", "scanflows", and
"respflows".
--dipset=IPSET_FILE
Display scans targeting hosts in IPSET_FILE, where IPSET_FILE is a
standard SiLK IPset file. This switch is ignored for --report
types other than "scanset", "scanflows", and "respflows".
--show-header
Display a header line giving a short name (or title) for each field
when printing textual output with the "standard", "volume", or
"export" report types. By default, no header is displayed.
--columnar
Display output in more human-readable columnar format when printing
textual output with the "standard" or "volume" report types. When
this switch is not given, the output is presented as data fields
delimited by the | character.
--output-path=PATH
Write results to PATH instead of to the standard output.
Configuration Options
--database=DBNAME
Select a database instance other than the default. The default is
specified by the "db_instance" value in the configuration file as
described in "CONFIGURATION" below.
Other Options
--help
Display a brief usage message and exit.
--man
Display full documentation for rwscanquery and exit.
--version
Print the version number and exit the application.
CONFIGURATIONrwscanquery reads configuration information from a file named
.rwscanrc. If the RWSCANRC environment variable is set, it is used as
the location of the .rwscanrc file. When RWSCANRC is not set,
rwscanquery attempts to find a file name .rwscanrc in the directories
specified in the "FILES" section below.
The format of the .rwscanrc file is name=value pairs, one per line.
The configuration parameters currently read from .rwscanrc are:
db_driver
The type of database to connect to. rwscanquery supports "oracle",
"postgresql", "mysql", and "sqlite".
db_userid
The userid to use when connecting to the scan database.
db_password
The password to use when connecting to the scan database.
db_instance
The name of the database instance to connect to if none is provided
with the --database command line switch. If neither this
configuration option nor the --database command line switch are
specified, the hard-coded default database instance "SCAN" is used.
rw_in_class
The class for incoming flow data. The "rw_in_class" and "rw_in_type"
values are used to query scan flows when the "scanflows" report type
is requested or when the --daddress or --dipset switches are used for
the "scanset" report type. If not specified, rwfilter's default is
used.
rw_in_type
The type(s) for incoming flow data. See "rw_in_class" for details.
rw_out_class
The class for outgoing flow data. The "rw_out_class" and
"rw_out_type" values are used to query scan flows when the
"respflows" report type is requested. If not specified, rwfilter's
default is used.
rw_out_type
The type(s) for outgoing flow data. See "rw_out_class" for details.
(Note that rwfilter often defaults to querying incoming flows, so
this parameter ought to be specified.)
EXAMPLES
In the following examples, the dollar sign ("$") represents the shell
prompt. The text after the dollar sign represents the command line.
Lines have been wrapped for improved readability, and the back slash
("\") is used to indicate a wrapped line.
Display information on all scans occurring during the 12:00 hour
(12:00:00 to 12:59:59) of 2009/02/12.
$ rwscanquery--show-header --start-date=2009/02/12:12
scan-id|stime|etime|proto|srcaddr|flows|packets|bytes
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
365|2009-02-12 12:08:40|2009-02-12 12:14:54|6|10.146.88.117|256|256|10240
57|2009-02-12 12:28:51|2009-02-12 12:34:55|6|10.29.23.160|256|256|10240
627|2009-02-12 11:52:07|2009-02-12 12:41:16|17|10.253.24.230|1023|1023|30175
366|2009-02-12 12:41:50|2009-02-12 12:48:14|6|10.146.89.46|256|256|10240
182|2009-02-12 12:54:39|2009-02-12 13:01:20|6|10.79.26.176|256|256|10240
4|2009-02-12 12:41:19|2009-02-12 13:33:57|17|10.2.47.87|1023|1023|30205
Create the IPset file scan.set containing the scanners discovered
during that hour.
$ rwscanquery --report=scanset --start-date=2009/02/12:12 \
--output-path=scan.set
$ rwsetcat scan.set
10.2.47.87
10.29.23.160
10.79.26.176
10.146.88.117
10.146.89.46
10.199.151.231
10.253.24.230
Repeat the first query but limit the output to scanners coming from the
CIDR block 10.199.0.0/16.
$ rwscanquery--show-header --start-date=2009/02/12:12 \
--saddr=10.199.0.0/16
scan-id|stime|etime|proto|srcaddr|flows|packets|bytes
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
Expand the query for that CIDR block to include the preceding and
following hours (11:00:00 to 13:59:59).
$ rwscanquery --start-date=2009/02/12:11 --end-date=2009/02/12:13 \
--saddr=10.199.0.0/16
499|2009-02-12 12:01:56|2009-02-12 12:08:39|6|10.199.151.231|256|256|10240
497|2009-02-12 13:33:57|2009-02-12 14:24:35|17|10.199.98.5|1023|1023|30079
Create the IPset file scanning-cidr.set that contains the CIDR block
10.199.0.0/16, and then search for scans coming from that IP on Feb 13,
2009.
$ cat scanning-cidr.txt
10.199.0.0/16
$ rwsetbuild scanning-cidr.txt scanning-cidr.set
$
$ rwscanquery --start-date=2009/02/13 --sipset=scanning-cidr.set
500|2009-02-13 22:42:25|2009-02-13 22:48:45|6|10.199.207.32|256|256|10240
Print the volume of data attributed to scans over a three day period.
$ rwscanquery --report=volume --show-header \
--start-date=2009/02/12 --end-date=2009/02/14
date|flows|packets|bytes
2009/02/12|137452|137499|17149008
2009/02/13|74727|76167|2798040
2009/02/14|76160|76160|2750531
The following limits the volume report to the IPs in the file
scanning-cidr.set and displays the results in columns.
$ rwscanquery --report=volume --show-header --columnar \
--start-date=2009/02/12 --end-date=2009/02/14 \
--sipset=scanning-cidr.set
date flows packets bytes
2009/02/12 1279 1279 40319
2009/02/13 256 256 10240
2009/02/14 256 256 10240
Get the SiLK Flow records coming from the scanners during the 12:00
hour on 2009/02/12 and store in the file scanning-flows.rw.
$ rwscanquery --report=scanflows --start-date=2009/02/12:12 \
--output=scanning-flows.rw
Use rwuniq(1) to summarize the file scanning-flows.rw.
$ rwuniq --fields=sip --values=flows,packets,bytes \
--sort-output scanning-flows.rw
sIP| Records| Packets| Bytes|
10.2.47.87| 373| 373| 11032|
10.29.23.160| 256| 256| 10240|
10.79.26.176| 203| 203| 8120|
10.146.88.117| 256| 256| 10240|
10.146.89.46| 256| 256| 10240|
10.199.151.231| 256| 256| 10240|
10.253.24.230| 846| 846| 24921|
Run a respflows report to verify that there were no responses to the
scan.
$ rwscanquery --report=respflows --start-date=2009/02/12:12 \
--output=scanning-response.rw
$
$ rwuniq --fields=sip --values=flows,packets,bytes \
--sort-output scanning-response.rw
sIP| Records| Packets| Bytes|
Create the IPset subnet-scan.set for scanners that targeted the
192.168.186.0/24 CIDR block during the 12:00 hour on 2009/02/12.
$ rwscanquery --report=scanset --start-date=2009/02/12:12 \
--daddress=192.168.186.0/24 --output-path=subnet-scan.set
Store the corresponding flow records for those scans in the file
subset-scan.rw.
$ rwscanquery --report=scanflows --start-date=2009/02/12:12 \
--daddress=192.168.186.0/24 --output-path=subnet-scan.rw
Determine how many IPs in that subnet were targeted.
$ rwuniq --fields=sip --values=flows,distinct:dip subnet-scan.rw
sIP| Records|dIP-Distin|
10.146.89.46| 256| 256|
Display the title line for an export report.
$ rwscanquery --report=export --start-date=2009/02/12:12 \
--show-header | head -1
id|sip|proto|stime|etime|flows|packets|bytes|scan_model|scan_prob
ENVIRONMENT
RWSCANRC
This environment variable allows the user to specify the location
of the .rwscanrc configuration file. The value may be a complete
path or a file relative to the user's current directory. See the
"FILES" section for standard locations of this file.
SILK_CLOBBER
The SiLK tools normally refuse to overwrite existing files.
Setting SILK_CLOBBER to a non-empty value removes this restriction
for the report types of "scanset", "scanflows", and "respflows".
SILK_CONFIG_FILE
This environment variable is used as the location for the site
configuration file, silk.conf, for report types that use rwfilter.
When this environment variable is not set, rwfilter searches for
the site configuration file in the locations specified in the
"FILES" section.
SILK_DATA_ROOTDIR
This environment variable specifies the root directory of data
repository for report types that use rwfilter. This value
overrides the compiled-in value. In addition, rwfilter may use
this value when searching for the SiLK site configuration files.
See the "FILES" section for details.
SILK_RWFILTER_THREADS
The number of threads rwfilter uses when reading files from the
data store.
SILK_PATH
This environment variable gives the root of the install tree. When
searching for the site configuration file, rwfilter may use this
environment variable. See the "FILES" section for details.
PATH
This is the standard UNIX path (c.f., environ(7)). Depending on
the report type, rwscanquery may invoke rwfilter(1), rwset(1),
rwsetbuild(1), or rwsetcat(1) as part of its processing.
RWFILTER
Complete path to rwfilter. If not set, rwscanquery attempts to
find rwfilter on your PATH.
RWSET
Complete path to rwset. If not set, rwscanquery attempts to find
rwset on your PATH.
RWSETBUILD
Complete path to rwsetbuild. If not set, rwscanquery attempts to
find rwsetbuild on your PATH.
RWSETCAT
Complete path to rwsetcat. If not set, rwscanquery attempts to
find rwsetcat on your PATH.
FILES
${RWSCANRC}
${HOME}/.rwscanrc
/usr/local/share/silk/.rwscanrc
Possible locations for the rwscanquery configuration file,
.rwscanrc. In addition, rwscanquery checks the parent directory of
the directory containing the rwscanquery script.
${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/local/share/silk/silk.conf
/usr/local/share/silk.conf
Possible locations for the SiLK site configuration file---for
report types that use rwfilter.
SEE ALSOrwscan(1), rwfilter(1), rwset(1), rwsetbuild(1), rwsetcat(1),
rwuniq(1), silk.conf(5), silk(7), environ(7)SiLK 3.11.0.1 2016-02-19 rwscanquery(1)
