NEUBOT(1) Neubot manual NEUBOT(1)NAMEneubot - The network neutrality bot
SYNOPSISneubot--help
neubot-V
neubot subcommand ...
DESCRIPTION
Neubot is a free-software Internet bot that gathers network performance
data useful to investigate network neutrality. Neubot runs in the back‐
ground and periodically performs active network tests with test servers
provided by Measurement Lab (a distributed server platform that hosts
open network measurement tools). Implemented tests are described in the
IMPLEMENTED TESTS section.
Neubot does not perform any network test until you grant it the permis‐
sion to collect and publish your IP address for research purposes.
Neubot behaves like this because it is developed in the European Union;
therefore, it must comply with European privacy laws (which consider IP
addresses personal data). See the PRIVACY section for more info.
Neubot is a background process. You can control it by using its subcom‐
mands, its web interface or its web API. Neubot listens for web
requests at http://127.0.0.1:9774/. To access the web interface, use
either your favorite browser or the viewer subcommand. To change the
address and/or port where Neubot listens for web requests, edit
/etc/neubot/api (see the FILES section).
Neubot runs with the privileges of the _neubot user, or with the ones
of the user indicated in the /etc/neubot/users configuration file (see
the FILES section).
The command line interface allows you to get the usage string (neubot
--help), get the version number (neubot -V), and run a Neubot's subcom‐
mand (neubot subcommand...).
IMPLEMENTED TESTS
All Neubot tests receive and send random data. Neubot does not monitor
the user's traffic.
Neubot implements three active network tests: bittorrent, raw and
speedtest. For each test, there is a Neubot subcommand that allows to
run the test immediately. Moreover, Neubot schedules one of the three
tests at random every 23 - 27 minutes.
The bittorrent test emulates BitTorrent peer-wire protocol and esti‐
mates the round-trip time, the download and the upload goodput (i.e.
the application-level speed measured at the receiver). It uses the
time that connect() takes to complete as an estimator of the round-trip
time. It estimates the goodput by dividing the amount of transferred
bytes by the elapsed time. To avoid consuming too much user resources,
the bittorrent test adapts the number of bytes to transfer such that
the test runs for about five seconds.
The raw test performs a raw 10-second TCP download to estimate the
download goodput. Also, it estimates the goodput from the round-trip
time, and it collects statistics about the TCP sender by using Web100
(see http://www.web100.org). In addition, it estimates the round-trip
time in two ways: (1) by measuring the time that connect() takes to
complete (like bittorrent) and (2) by measuring the average time
elapsed between sending a small request and receiving a small response.
The speedtest test emulates HTTP and estimates the round-trip time, the
download and the upload goodput. It estimates the round-trip time in
two ways: (1) by measuring the time that connect() takes to complete
(like bittorrent) and (2) by measuring the average time elapsed between
sending a small request and receiving a small response (like raw). It
estimates the goodput by dividing the amount of transferred bytes by
the elapsed time. To avoid consuming too much user resources, the bit‐
torrent test adapts the number of bytes to transfer such that the test
runs for about five seconds.
SUBCOMMANDS
This section documents Neubot's subcommands.
neubot bittorrent [-6fv] [-A address] [-p port]
Asks Neubot to run a bittorrent test using the web API and fails
if Neubot is not running in the background.
Accepts the following options:
-6 Prefer IPv6 to IPv4.
-A address
Address of the remote test server.
-f Force the test. Run the test in the local process context
(instead of using the web API) and override privacy set‐
tings if needed. Useful for developers and for debugging.
-p port
Port of the remote test server.
-v Makes the command more verbose.
neubot database [-f database] [action]
Performs the specified action or prints the database's path if
no action is specified. We do not recommended to use this com‐
mand to modify the database while Neubot is running, since
Neubot does not expect the database to change while it is run‐
ning, so it won't pick the changes up. This command requires
root privileges to modify the database: if you are not root, the
database is opened in readonly mode.
Accepts the following options:
-f database
Force file. Forces the command to use database instead of
the default database path.
Implements the following actions:
delete_all
Deletes all the results in the database.
dump Dumps to the standard ouput the content of the database
in JSON format.
prune Removes results older than one year.
regen_uuid
Generates a new random unique identifier (or UUID) for
Neubot. (See the PRIVACY section for more on the UUID).
show Pretty prints to the standard ouput the content of the
database in JSON format.
neubot privacy [-Pt] [-D setting=value] [-f database]
Manage privacy settings. When invoked without options, this com‐
mand prints the current privacy settings values.
Accepts the following options:
-D setting=value
Turn on (nonzero) and off (zero) the specified privacy
setting.
This command just modifies the database: you have to
restart Neubot to make changes effective. To modify pri‐
vacy settings when Neubot is running, we recommend to use
the web interface.
Privacy settings:
informed
The user has read the privacy policy.
can_collect
The user provides the permission to collect
his/her IP address for research purposes.
can_publish
The user provides the permission to publish
his/her IP address allowing anyone to reuse it for
research purposes.
-f database
Force file. Forces the command to use database instead of
the default database path.
-P Prints privacy policy on the standard output.
-t Test. Exits with success (exit value 0) if all privacy
settings all nonzero. Exits with failure (exit value
nonzero) if at least one setting is zero.
neubot raw [-6fv] [-A address] [-p port]
Asks Neubot to run a raw test using the web API and fails if
Neubot is not running in the background.
Accepts the following options:
-6 Prefer IPv6 to IPv4.
-A address
Address of the remote test server.
-f Force the test. Run the test in the local process context
(instead of using the web API) and override privacy set‐
tings if needed. Useful for developers and for debugging.
-p port
Port of the remote test server.
-v Makes the command more verbose.
neubot speedtest [-6fv] [-A address] [-p port]
Asks Neubot to run a speedtest test using the web API and fails
if Neubot is not running in the background.
Accepts the following options:
-6 Prefer IPv6 to IPv4.
-A address
Address of the remote test server.
-f Force the test. Run the test in the local process context
(instead of using the web API) and override privacy set‐
tings if needed. Useful for developers and for debugging.
-p port
Port of the remote test server.
-v Makes the command more verbose.
neubot start
On MacOS this command runs launchctl(1), which in turn starts
Neubot. You must be root to run this command. On MacOS,
Neubot's installer configures the system to launch Neubot at
startup; i.e. you do not typically need to run this command.
(On MacOS, Neubot is implemented by two daemons: the usual
unprivileged daemon and a privileged daemon. The latter controls
the former and periodically forks an unprivileged child to check
for updates.)
On MacOS, the start command accepts the following options:
-a Auto-updates. When this flag is specified, the privileged
daemon periodically forks an unprivileged child to check
for updates.
-d Debug. When this flag is specified, Neubot runs in the
foreground.
-v Verbose. When this flag is specified, the start command
is verbose (i.e. it prints on the standard error the com‐
mands it is about to invoke).
When both -v and -d are specified, Neubot runs in verbose
mode in the foreground.
At boot time, launchctl(1) starts Neubot with the -a and -d com‐
mand line options.
On other UNIX systems, the start command forks the Neubot dae‐
mon, which drops root privileges and runs in the background. On
such systems, this command does not accept any command line
option.
neubot status
This command asks the status of Neubot using the web API. It
returns 0 if connect() succeeds and the response is OK, nonzero
otherwise.
On MacOS this command accepts the -v option, which makes it more
verbose. On other UNIX systems, it does not accept any command
line option.
neubot stop
On MacOS, this command runs launchctl(1), which in turn stops
Neubot. You must be root to run this command. On MacOS, this
command accepts the -v option, which makes it more verbose.
On other UNIX systems, this command uses the web API to request
Neubot to exit.
neubot viewer
This command shows the web interface by embedding a web render‐
ing engine into a window manager's window. Currently, the only
implemented viewer is based on python-webkit and pygtk.
FILES
Assuming that Neubot is installed at /usr/local, this is the list of
the files installed.
/etc/neubot/api
Configuration file that indicates the endpoint where Neubot
should listen for web API requests. Example (which also shows
the syntax and indicates the default values):
#
# /etc/neubot/api - controls address, port where Neubot listens
# for incoming web API requests.
#
address 127.0.0.1 # Address where the listen
port 9774 # Port where to listen
/etc/neubot/users
Configuration file that indicates the unprivileged user names
that Neubot should use. Example (which also shows the syntax and
indicates the default values):
#
# /etc/neubot/users - controls the unprivileged user names used
# by Neubot to perform various tasks.
#
update_user _neubot_update # For auto-updates (MacOS-only)
unpriv_user _neubot # For network tests
/usr/local/bin/neubot
The Neubot executable script.
/usr/local/share/neubot/
Location where Neubot Python modules are installed.
/usr/local/share/neubot/www/
Location where the web interface files are installed. The web
interface is described in the WEB INTERFACE FILES section.
/var/lib/neubot/database.sqlite3
System-wide results database for Linux systems, created when
Neubot starts for the first time.
/var/neubot/database.sqlite3
System-wide results database for non-Linux systems, created when
Neubot starts for the first time.
EXAMPLES
In this section, we represent the unprivileged user prompt with $ and
the root user prompt with #.
Run on-demand bittorrent test:
$ neubot bittorrent
Run on-demand raw test:
$ neubot raw
Run on-demand speedtest test:
$ neubot speedtest
Start Neubot:
# neubot start
Stop Neubot:
# neubot stop # MacOS
$ neubot stop # other UNIX
Run Neubot in the foreground with verbose logging:
# neubot start -dv # MacOS
$ neubot agent -v -D agent.daemonize=no # other UNIX
Export Neubot results to JSON:
# neubot database dump > output.json
Read Neubot's privacy policy:
$ neubot privacy -P
Run Neubot command from the sources directory:
$ ./UNIX/bin/neubot command
WEB INTERFACE FILES
Here we provide a brief description of the core files of the web inter‐
face:
css/ Directory that contains CSS files.
favicon.ico
Neubot's favicon.
footer.html
Common footer for all web pages (Neubot uses server-side
includes).
header.html
Common header for all web pages (Neubot uses server-side
includes).
img/ Directory that contains images.
js/ Directory that contains javascript files. In addition to jQuery
and jqPlot, it contains the following scripts:
js/contrib.js
Helper functions from many authors.
js/i18n.js
Implementation of web user interface internationalization
(aka i18n).
js/index.js
Contains functions to retrieve and process the state of
Neubot.
js/log.js
Contains code to retrieve and process Neubot logs.
js/privacy.js
Contains code to query and modify privacy settings.
js/results.js
Contains code to process Neubot results, as well as code
to display them as plots and tables.
js/settings.js
Contains code to retrieve and modify Neubot settings.
js/state.js
Helper code for retrieving and processing Neubot state.
js/update.js
Minimal script included by updater.html. It just sets the
active tab in the web interface.
js/utils.js
Miscellaneous helper functions.
lang/ Directory that contains one javascript file for each language in
which the web interface is translated. Each of these javascripts
contains a dictionary, named LANG, that maps a string (or a key
representing a string) to its translation.
In javascript, you mark strings for translation by wrapping them
with i18n.get() calls. For example, to indicate that the string
"Disable automatic tests" should be translated, you should
write:
...
i18n.get("Disable automatic tests");
In HTML code, you mark the content of an HTML tag for transla‐
tion by adding the tag to the i18n class. Differently from
javascript, we don't map the content of an HTML tag to its
translation; instead, we map a key that represents the HTML tag
content to its translation. The key is another HTML class, which
must start with i18n_, as in the following example:
...
<p class="i18n i18n_foobar">Neubot web interface</p>
To translate the two examples above in, for example, Italian you
edit the www/lang/it.css file and add:
var LANG = {
...
"Disable automatic tests": "Disabilita test automatici",
"i18n_foobar": "Interfaccia web di Neubot",
...
};
log.html
Shows Neubot logs.
not_running.html
Page displayed when Neubot is not running.
privacy.html
Shows, and allows to modify, privacy settings.
results.html
The results page, dynamically filled by javascript using Neubot
web API. It allows you to see the results of recent experiments,
both in form of plots and tables.
settings.html
Shows (and allows to modify) Neubot settings.
test/ Directory that contains a foo.html and a foo.json file for each
test foo. The list of available tests in results.html is auto‐
matically generated from the files in this directory.
test/foo.html
Description of the foo test. It is included into the
results.html page when the test is selected.
test/foo.json
Description of the plots and tables included into
results.html when test foo is selected. The format of the
JSON is documented into the WEB API section of this man‐
ual page.
test/foo.json.local
When foo.json.local exists, Neubot will use it (instead
of foo.json) to prepare plots and tables in results.html.
Allows the user to heavily customize the results page for
test foo.
update.html
Page displayed on Windows when Neubot needs to be manually
updated. Now that automatic updates are implemented, it should
never pop up.
WEB API
To access Neubot API, you send HTTP requests to the address and port
where Neubot is listening (which is 127.0.0.1:9774 by default, and
which can be changed by editing /etc/neubot/api).
Here is a detailed description of each API.
/api This API is an alias for /api/.
/api/ This API allows you to get (GET) the list of available APIs,
encoded as a JSON.
Returned JSON example:
[
"/api",
"/api/",
"/api/config",
"/api/data",
"/api/debug",
"/api/exit",
"/api/index",
"/api/log",
"/api/results",
"/api/runner",
"/api/state",
"/api/version"
]
/api/config[?options]
This API allows to you get (GET) and set (POST) the variables
that modify the behavior of Neubot.
GET returns a dictionary, encoded using JSON, that maps each
variable to its value. POST sends a url-encoded string, which
contains one or more variable=new_value atoms separated by &.
The API accepts the following query-string options:
debug=integer [default: 0]
When nonzero, the API returns a pretty-printed JSON. Oth‐
erwise, the JSON is serialized on a single line.
labels=integer [default: 0]
When nonzero, returns the description of the variables
instead of their values.
Returned JSON example:
{
"enabled": 1,
"negotiate.max_thresh": 64,
"negotiate.min_thresh": 32,
"negotiate.parallelism": 7,
"privacy.can_collect": 1,
"privacy.can_publish": 1,
"privacy.can_informed": 1,
...
"uuid": "0964312e-f451-4579-9984-3954dcfdeb42",
"version": "4.2",
"www.lang": "default"
}
We have not standardized variable names yet. Therefore, we don't
provide here a list of variable names, types, and default val‐
ues.
/api/data?test=string[&options]
This API allows you to retrieve (GET) the data collected during
Neubot tests. As we have a single API for all tests, you must
provide the test name using the query string.
This API returns a JSON that serializes a list of dictionaries,
in which each dictionary is the data collected during a test. We
dedicate a section of the manual page to the structure returned
by each test.
This API accepts the following query-string parameters:
debug=integer [default: 0]
When nonzero, the API returns a pretty-printed JSON. Oth‐
erwise, the JSON is serialized on a single line.
since=integer [default: 0]
Returns only the data collected after the specified time
(indicated as the number of seconds elapsed since mid‐
night of January, 1st 1970).
test=string
This parameter is mandatory and specifies the test whose
data you want to retrieve.
until=integer [default: 0]
Returns only the data collected before the specified time
(indicated as the number of seconds elapsed since mid‐
night of January, 1st 1970).
/api/debug
This API allows you to get (GET) text/plain information about
Neubot internals, which is typically useful for debugging pur‐
poses. As such, the consistency of the output format is not
guaranteed.
Returned text example:
{'WWW': '/usr/share/neubot/www',
'notifier': {'_subscribers': {},
'_timestamps': {'statechange': 1336727245277393,
'testdone': 1336727245277246}},
'queue_history': [],
'typestats': {'ABCMeta': 26,
'BackendNeubot': 1,
'BackendProxy': 1,
...
}}
/api/exit
When this API is invoked, Neubot exits immediately (i.e. without
sending any response).
Don't use this API to shut down Neubot on MacOS, use the neubot
stop command instead. This API, in fact, has effect on the
unprivileged Neubot process only, and the privileged process
will respawn the unprivileged process once it notices it died.
/api/index
This API uses 302 Found and Location to redirect the caller to
either index.html (if privacy settings are OK) or on pri‐
vacy.html (if privacy settings are not OK).
/api/log[?options]
This API allows you to get (GET) Neubot logs, as a list of dic‐
tionaries. Each dictionary represents a log record and contains
the following fields:
timestamp (integer)
Time when this log was generated, expressed as number of
seconds elapsed since midnight of January, 1st 1970.
severity (string)
The log message severity; one of: DEBUG, INFO, WARNING,
and ERROR.
message (string)
The log message string.
This API accepts the following query-string options:
debug (int) [default: 0]
If nonzero, the API formats logs like they are printed on
the system logger (i.e. as a text/plain sequence of
lines). Otherwise, the API returns the JSON list of dic‐
tionaries described above.
reversed (int) [default: 0]
If nonzero logs are reversed (i.e. the most recent log
record is the first element of the list). Otherwise logs
are returned in natural order (the most recent log record
is the last element of the list).
verbosity (int) [default: 1]
When the verbosity is less than 1, only ERROR and WARNING
messages are returned. When the verbosity is 1, the API
returns also INFO messages. When the verbosity is greater
than 1, the API returns also INFO and DEBUG messages.
Returned JSON example:
[
{
"message": "raw_negotiate: not reached final state",
"severity": "WARNING",
"timestamp": 1366195042
},
{
"message": "raw_negotiate: bad response",
"severity": "ERROR",
"timestamp": 1366236483
},
{
"message": "raw_negotiate: not reached final state",
"severity": "WARNING",
"timestamp": 1366236484
}
]
/api/results?test=string[&options]
This API allows the web interface to get (GET) information on
how to format results. It returns a dictionary, encoded as JSON,
that indicates the plots and the tables to be generated in the
results.html page for the selected test (which is either the
test specified via query string or the default test, speedtest,
if none was specified).
The dictionary for test foo is generated using www/test/foo.json
(or www/test/foo.json.local) as template and contains the fol‐
lowing fields:
available_tests (list of strings)
List that contains the name of all the available tests.
description (string)
String that contains a long description of the selected
test. This is the content of www/test/foo.html.
plots (list of dictionaries)
List of dictionaries. Each dictionary contains the
instructions to generate a plot:
datasets (list of dictionaries)
List of dictionaries. Each dictionary contains the
instructions to plot one serie of data:
label (string)
Label to use in the legend.
marker (string)
Indicates the marker to use, either circle
or square.
recipe (list)
LISP-like code that describes how to gener‐
ate one point on the Y axis from one row of
the selected test's data. We describe this
lisp-like language in the DATA PROCESSING
LANGUAGE section of this manual page.
title (string)
Title of the plot.
xlabel (string)
Label for the X axis.
ylabel (string)
Label for the Y axis.
selected_test
The selected test name.
table (list of dictionaries)
List of dictionaries. Each dictionary is one column of
the table to be added to results.html:
label (string)
Label of the column header.
recipe (list)
LISP-like code that describes how to generate the
value of the current column in the table from one
row of the selected test's data. We describe this
lisp-like language in the DATA PROCESSING LANGUAGE
section of this manual page.
title (string)
The title of the test (e.g. 'BitTorrent test').
www_no_description (integer)
Whether to include a description of the test in the
results page (zero) or not (nonzero).
www_no_legend (integer)
Whether to include a legend in the plots (zero) or not
(nonzero).
www_no_plot (integer)
Whether to generate plots (zero) or not (nonzero).
www_no_split_by_ip (integer)
Whether to split the selected test's data by IP and plot
a different line for each IP (zero) or not (nonzero).
www_no_table (integer)
Whether to generate a table that contains the selected
test's data (zero) or not (nonzero).
www_no_title (integer)
Whether to include the title of the test in the results
page (zero) or not (nonzero).
The API accepts the following query-string options:
debug=integer [default: 0]
When nonzero, the API returns a pretty-printed JSON. Oth‐
erwise, the JSON is serialized on a single line.
test=string
This parameter is mandatory and specifies the selected
test.
Returned JSON example:
{
"available_tests": [
"raw",
"speedtest",
"bittorrent"
],
"description": "...",
"www_no_split_by_ip": 0,
"title": "Your recent Speedtest results",
"www_no_legend": 0,
"selected_test": "speedtest",
"www_no_plot": 0,
"www_no_table": 0,
"table": [
{
"recipe": ["to-datetime",
["select", "timestamp", "result"]],
"label": "Timestamp"
},
{
"recipe": ["select", "internal_address", "result"],
"label": "Internal address"
},
{
"recipe": ["select", "real_address", "result"],
"label": "Real address"
},
{
"recipe": ["select", "remote_address", "result"],
"label": "Remote address"
},
{
"recipe": ["to-millisecond-string",
["select", "connect_time", "result"]],
"label": "Connect time"
},
{
"recipe": ["to-millisecond-string",
["select", "latency", "result"]],
"label": "Appl. latency"
},
{
"recipe": ["to-speed-string",
["select", "download_speed", "result"]],
"label": "Download speed"
},
{
"recipe": ["to-speed-string",
["select", "upload_speed", "result"]],
"label": "Upload speed"
}
],
"www_no_description": 0,
"plots": [
{
"datasets": [
{
"marker": "circle",
"recipe": ["to-speed",
["select", "download_speed",
"result"]],
"label": "Dload"
},
{
"marker": "square",
"recipe": ["to-speed",
["select", "upload_speed",
"result"]],
"label": "Upload"
}
],
"ylabel": "Goodput (Mbit/s)",
"xlabel": "Date",
"title": "Download and upload speed"
},
{
"datasets": [
{
"marker": "circle",
"recipe": ["to-millisecond",
["select", "latency", "result"]],
"label": "Appl. latency"
},
{
"marker": "square",
"recipe": ["to-millisecond",
["select", "connect_time",
"result"]],
"label": "Connect time"
}
],
"ylabel": "Delay (ms)",
"xlabel": "Date",
"title": "Connect time and latency"
}
],
"www_no_title": 0
}
/api/runner?test=string[&options]
This API allows the caller to schedule a test for immediate exe‐
cution. If a test is already running the API returns an error
500, otherwise it returns 200.
The API accepts the following query-string options:
test=string
This option is mandatory and indicates the name of the
test that Neubot should schedule for execution.
streaming=integer [default: 0]
When nonzero, Neubot streams logs generated during the
test in the response body and closes the connection when
the test is complete. Otherwise, the response body is an
empty dictionary.
When you invoke tests from the command line (e.g. neubot
bittorrent), streaming is the feature that allows to
print logs generated by the test on the console.
Returned JSON example:
{}
Returned text example:
1366299354 [INFO] runner_core: Need to auto-discover first...
1366299355 [INFO] runner_mlabns: server discovery...
1366299356 [INFO] runner_mlabns: server discovery... done
1366299356 [INFO] raw_clnt: connection established with ...
1366299356 [INFO] raw_clnt: connect_time: 13.6 ms
1366299357 [INFO] raw_clnt: sending auth to server...
1366299357 [INFO] raw_clnt: sending auth to server... done
1366299357 [INFO] raw_clnt: receiving auth from server...
1366299357 [INFO] raw_clnt: receiving auth from server... done
1366299357 [INFO] raw_clnt: estimating ALRTT...
1366299357 [INFO] raw_clnt: alrtt_avg: 14.3 ms
1366299357 [INFO] raw_clnt: estimating ALRTT... done
1366299357 [INFO] raw_clnt: raw goodput test...
1366299367 [INFO] raw_clnt: raw goodput test... done
1366299367 [INFO] raw_clnt: goodput: 65.5 Mbit/s
/api/state[?options]
This API allows you to get (GET) and track (via comet) the state
of Neubot. The API returns a dictionary with the following
fields:
current=string
The name of the current state; one of: idle, rendezvous,
negotiate, test, and collect.
events=dictionary
A dictionary that maps the name of an event (a string) to
the most recent value related to such event (a string, an
integer, a list, or a dictionary).
While running, Neubot generates a limited set of events,
which drive the web interface. For example, the
test_download event value is used to update the download
speed in the right sidebar of the web interface.
The list of generated events is not standardized yet, so
we don't provide it here.
t=integer
The identifier of the current event.
The API accepts the following query-string options:
debug=integer [default: 0]
When nonzero, the API returns a pretty-printed JSON. Oth‐
erwise, the JSON is serialized on a single line.
t=integer
When this option is present, Neubot does not return a
response until the next event after the one identified by
integer is fired (or until a timeout expires). This
behavior allows to implement the comet pattern and to
timely update the web interface with low overhead.
/api/version
This API allows you to get (GET) the version number of Neubot,
in text/plain format.
Returned text example:
0.4.15.7
BITTORRENT DATA FORMAT
We represent the data collected by the bittorrent test with a dictio‐
nary that contains the following fields:
connect_time (float)
RTT estimated by measuring the time that connect() takes to com‐
plete, measured in seconds.
download_speed (float)
Download speed measured by dividing the number of received bytes
by the elapsed download time, measured in bytes per second.
internal_address (string)
Neubot's IP address, as seen by Neubot. It is typically either
an IPv4 or an IPv6 address.
neubot_version (string)
Neubot version number, encoded as a floating point number and
printed into a string. Given a version number in the format
<major>.<minor>.<patch>.<revision>, the encoding is as follows:
<major> + 1e-03 * <minor> + 1e-06 * <patch>
+ 1e-09 * <revision>
For example, the 0.4.15.3 version number is encoded as
0.004015003.
platform (string)
The operating system platform, e.g. linux2, win32.
privacy_can_collect (integer)
The value of the can_collect privacy setting.
privacy_can_publish (integer)
The value of the can_publish privacy setting.
privacy_informed (integer)
The value of the informed privacy setting.
real_address (string)
Neubot's IP address, as seen by the server. It is typically
either an IPv4 or an IPv6 address.
remote_address (string)
The server's IP address. It is typically either an IPv4 or an
IPv6 address.
timestamp (integer)
Time when the test was performed, expressed as number of seconds
elapsed since midnight of January, 1st 1970.
upload_speed (float)
Upload speed measured by dividing the number of sent bytes by
the elapsed upload time, measured in bytes per second.
uuid (string)
Random unique identifier of the Neubot instance, useful to per‐
form time series analysis.
Example:
[
{
"connect_time": 0.003387928009033203,
"download_speed": 4242563.145733707,
"internal_address": "130.192.91.231",
"neubot_version": "0.004015007",
"platform": "linux2",
"privacy_can_collect": 1,
"privacy_can_publish": 1,
"privacy_informed": 1,
"real_address": "130.192.91.231",
"remote_address": "194.116.85.224",
"test_version": 1,
"timestamp": 1366045628,
"upload_speed": 4231443.875881268,
"uuid": "7528d674-25f0-4ac4-aff6-46f446034d81"
},
...
RAW TEST DATA FORMAT
We represent the data collected by the raw test with a dictionary that
contains the following fields:
connect_time (float)
RTT estimated by measuring the time that connect() takes to com‐
plete, measured in seconds.
download_speed (float)
Download speed measured by dividing the number of received bytes
by the elapsed download time, measured in bytes per second.
json_data (string)
This string contains the serialization of a JSON object, which
contains all the data collected during the test, both on the
server and on the client side. The dictionary that we are
describing, in fact, contains just a subset of the collected
results. We can not store the full JSON object directly until
Neubot's database module and web interface are ready to process
it.
internal_address (string)
Neubot's IP address, as seen by Neubot. It is typically either
an IPv4 or an IPv6 address.
latency (float)
RTT estimated by measuring the average time elapsed between
sending a small request and receiving a small response, measured
in seconds.
neubot_version (float)
Neubot version number, encoded as a floating point number and
printed into a string. Given a version number in the format
<major>.<minor>.<patch>.<revision>, the encoding is as follows:
<major> + 1e-03 * <minor> + 1e-06 * <patch>
+ 1e-09 * <revision>
For example, the 0.4.15.3 version number is encoded as
0.004015003.
platform (string)
The operating system platform, e.g. linux2, win32.
real_address (string)
Neubot's IP address, as seen by the server. It is typically
either an IPv4 or an IPv6 address.
remote_address (string)
The server's IP address. It is typically either an IPv4 or an
IPv6 address.
timestamp (integer)
Time when the test was performed, expressed as number of seconds
elapsed since midnight of January, 1st 1970.
uuid (string)
Random unique identifier of the Neubot instance, useful to per‐
form time series analysis.
Example:
[
{
"connect_time": 0.2981860637664795,
"download_speed": 3607.120929707688,
"internal_address": "130.192.91.231",
"json_data": "...",
"latency": 0.29875500202178956,
"neubot_version": "0.004015007",
"platform": "linux2",
"real_address": "130.192.91.231",
"remote_address": "203.178.130.237",
"timestamp": 1365071100,
"uuid": "7528d674-25f0-4ac4-aff6-46f446034d81"
},
...
Once unserialized, the JSON object saved into the json_data field of
the raw dictionary (henceforth, 'outer dictionary') is a dictionary
that contains the following fields:
client (dictionary)
A dictionary that contains data collected on the client side.
server (dictionary)
A dictionary that contains data collected on the server side.
The client dictionary contains the following fields:
al_capacity (float)
Median bottleneck capacity computed at application level (exper‐
imental).
al_mss (float)
MSS according to the application level (information gathered
using setsockopt(2)).
al_rexmits (list)
Likely retransmission events computed at application level
(experimental).
alrtt_avg (float)
Same as latency in the outer dictionary.
alrtt_list (list of tuples)
List of RTT samples estimated by measuring the average time
elapsed between sending a small request and receiving a small
response, measured in seconds.
connect_time (float)
Same as connect_time in the outer dictionary.
goodput (dictionary)
The dictionary contains the following fields:
bytesdiff
Total number of received bytes.
ticks (float)
Timestamp when this piece of data was collected,
expressed as number of seconds elapsed since midnight of
January, 1st 1970.
timediff (float)
Total download time.
goodput_snap (list of dictionaries)
List that contains a dictionary, which is updated roughly every
second during the download, and which contains the following
fields:
ticks (float)
Time when the current dictionary was saved, expressed as
number of seconds since midnight of January, 1st 1970.
bytesdiff (integer)
Number of bytes received since stats were previously
saved.
timediff (float)
Number of seconds elapsed since stats were previously
saved.
utimediff (float)
Difference between current tms_utime field of the tms
struct modified by times(3) and the previous value of the
same field.
stimediff (float)
Difference between current tms_stime field of the tms
struct modified by times(3) and the previous value of the
same field.
myname (string)
Neubot's address (according to the server). This is same as
real_address in the outer dictionary.
peername (string)
Servers's address. This is same as server_address in the outer
dictionary.
platform (string)
Same as platform in the outer dictionary.
uuid (string)
Same as uuid in the outer dictionary.
version (string)
Same as neubot_version in the outer dictionary.
The server dictionary contains the following fields:
goodput (dictionary)
The dictionary contains the following fields:
bytesdiff
Total number of sent bytes.
ticks (float)
Timestamp when this piece of data was collected,
expressed as number of seconds elapsed since midnight of
January, 1st 1970.
timediff (float)
Total upload time.
goodput_snap (list of dictionaries)
List that contains a dictionary, which is updated roughly every
second during the upload, and which contains the following
fields:
ticks (float)
Time when the current dictionary was saved, expressed as
number of seconds since midnight of January, 1st 1970.
bytesdiff (integer)
Number of bytes sent since stats were previously saved.
timediff (float)
Number of seconds elapsed since stats were previously
saved.
utimediff (float)
Difference between current tms_utime field of the tms
struct modified by times(3) and the previous value of the
same field.
stimediff (float)
Difference between current tms_stime field of the tms
struct modified by times(3) and the previous value of the
same field.
myname (string)
Servers's address. This is same as server_address in the outer
dictionary.
peername (string)
Neubot's address (according to the server). This is same as
real_address in the outer dictionary.
platform (string)
Same as platform in the outer dictionary.
timestamp (integer)
Time when the server dictionary was created, expressed as number
of seconds elapsed since midnight of January, 1st 1970.
version (string)
Same as neubot_version in the outer dictionary.
web100_snap (list)
A list that contains dictionaries. Each dictionary is a snapshot
of the Web100 TCP state. We take one Web100 snapshot every sec‐
ond during the upload.
On the client side, this field is empty. We are working to iden‐
tify the most interesting fields that is interesting to save.
Example:
[
{
"client": {
"al_mss": 1448,
"uuid": "7528d674-25f0-4ac4-aff6-46f446034d81",
"goodput": {
"bytesdiff": 128200,
"timediff": 35.540810108184814,
"ticks": 1365071098.203412
},
"al_rexmits": [],
"connect_time": 0.2981860637664795,
"alrtt_list": [
0.31011295318603516,
0.30966901779174805,
0.29677391052246094,
0.2957899570465088,
0.29570794105529785,
0.2956199645996094,
0.29558706283569336,
0.2956211566925049,
0.2958400249481201,
0.296828031539917
],
"myname": "130.192.91.231",
"peername": "203.178.130.237",
"platform": "linux2",
"version": "0.004015007",
"al_capacity": 10982553.692585895,
"alrtt_avg": 0.29875500202178956,
"goodput_snap": [
{
"bytesdiff": 24616,
"timediff": 1.0001380443572998,
"ticks": 1365071063.66274,
"stimediff": 0.0,
"utimediff": 0.0
},
...
]
},
"server": {
"timestamp": 1365070933,
"myname": "203.178.130.237",
"peername": "130.192.91.231",
"platform": "linux2",
"version": "0.004015007",
"goodput": {
"bytesdiff": 131092,
"timediff": 34.94503116607666,
"ticks": 1365070933.95337
},
"goodput_snap": [
{
"bytesdiff": 31856,
"timediff": 1.0005459785461426,
"ticks": 1365070900.008885,
"stimediff": 0.0,
"utimediff": 0.0
},
...
],
"web100_snap": []
}
}
SPEEDTEST DATA FORMAT
We represent the data collected by the speedtest test with a dictionary
that contains the following fields:
connect_time (float)
RTT estimated by measuring the time that connect() takes to com‐
plete, measured in seconds.
download_speed (float)
Download speed measured by dividing the number of received bytes
by the elapsed download time, measured in bytes per second.
internal_address (string)
Neubot's IP address, as seen by Neubot. It is typically either
an IPv4 or an IPv6 address.
latency (float)
RTT estimated by measuring the average time elapsed between
sending a small request and receiving a small response, measured
in seconds.
neubot_version (string)
Neubot version number, encoded as a floating point number and
printed into a string. Given a version number in the format
<major>.<minor>.<patch>.<revision>, the encoding is as follows:
<major> + 1e-03 * <minor> + 1e-06 * <patch>
+ 1e-09 * <revision>
For example, the 0.4.15.3 version number is encoded as
0.004015003.
platform (string)
The operating system platform, e.g. linux2, win32.
privacy_can_collect (integer)
The value of the can_collect privacy setting.
privacy_can_publish (integer)
The value of the can_publish privacy setting.
privacy_informed (integer)
The value of the informed privacy setting.
real_address (string)
Neubot's IP address, as seen by the server. It is typically
either an IPv4 or an IPv6 address.
remote_address (string)
The server's IP address. It is typically either an IPv4 or an
IPv6 address.
timestamp (integer)
Time when the test was performed, expressed as number of seconds
elapsed since midnight of January, 1st 1970.
upload_speed (float)
Upload speed measured by dividing the number of sent bytes by
the elapsed upload time, measured in bytes per second.
uuid (string)
Random unique identifier of the Neubot instance, useful to per‐
form time series analysis.
Example:
[
{
"connect_time": 0.0017991065979003906,
"download_speed": 11626941.501993284,
"internal_address": "130.192.91.231",
"latency": 0.003973397341641513,
"neubot_version": "0.004015007",
"platform": "linux2",
"privacy_can_collect": 1,
"privacy_can_publish": 1,
"privacy_informed": 1,
"real_address": "130.192.91.231",
"remote_address": "194.116.85.237",
"test_version": 1,
"timestamp": 1365074302,
"upload_speed": 10974865.674026133,
"uuid": "7528d674-25f0-4ac4-aff6-46f446034d81"
},
...
DATA PROCESSING LANGUAGE
The data processing language is a simple LISP-like language. As such,
it describes processes whose goal is to transform pieces of collected
data by using lists.
Differently from traditional LISP syntax, however, the data processing
language is encoded using JSON.
The language implements the following operations:
["divide", atom-or-list, atom-or-list]
Divides the left atom (or list) by the right atom (or list) and
returns the result.
["map-select", atom, list]
Cycles over the list and, for each element, it selects the field
indicated by the atom.
["parse-json", atom-or-list]
Parses the value of the atom (or list) into an object.
["reduce-avg", list]
Computes the average value of the list.
["select", atom, object]
Selects the element of object indicated by atom.
["to-datetime", atom-or-list]
Converts atom (or list) to datetime string.
["to-millisecond", atom-or-list]
Converts atom (or list) to millisecond.
["to-millisecond-string", atom-or-list]
Converts atom (or list) to millisecond string.
["to-speed", atom-or-list]
Converts atom (or list) to speed (in bits per second).
["to-speed-string", atom-or-list]
Converts atom (or list) to speed string (in bits per second).
"result"
The current piece of data we are processing.
Example (select the json_data field of the result, convert it to json,
take the client field, take and compute the average of the alrtt_list
field, convert the result to millisecond):
["to-millisecond",
["reduce-avg",
["select", "alrtt_list",
["select", "client",
["parse-json",
["select", "json_data", "result"]]]]]]
PRIVACY
Neubot collects your IP address, which is personal data according to
European privacy laws. For this reason, Neubot needs to obtain your
permission to collect your IP address for research purposes, as well as
to publish it on the web for the same purpose. In addition, it also
needs that you assert that you have read the privacy policy.
Without the assertion that you have read the privacy policy and the
permission to collect and publish your IP address, Neubot can not per‐
form automatic (or manual) tests.
You can read Neubot's privacy policy by running the neubot privacy -P
command. The privacy policy is also available at:
http://127.0.0.1:9774/privacy.html
Of course, if you modified the address and/or port where Neubot lis‐
tens, you need to update the URI accordingly.
In addition to the above, each Neubot is identified by a random unique
identifier (UUID) that is used to perform time series analysis. We
believe that this identifier does not brach your privacy: in the worst
case, we would be able to say that a given Neubot has changed Internet
address (anche, hence, ISP and/or location). To regenerate your unique
identifier, you can run the neubot database regen_uuid command.
AUTHOR
Neubot authors are:
Simone Basso <bassosimone@gmail.com>
The following people have contributed patches to the project:
Alessio Palmero Aprosio <alessio@apnetwork.it>
Antonio Servetti <antonio.servetti@polito.it>
Roberto D'Auria <everlastingfire@autistici.org>
Marco Scopesi <marco.scopesi@gmail.com>
The following people have helped with internationalization:
Claudio Artusio <claudioartusio@gmail.com>
COPYRIGHT
Neubot as a collection is:
Copyright (c) 2010-2013 Nexa Center for Internet & Society,
Politecnico di Torino (DAUIN)
Neubot is free software: you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation, either version
3 of the License, or (at your option) any later version.
SEE ALSO
· http://www.neubot.org/
· http://github.com/neubot/neubot
· http://twitter.com/neubot
Neubot 0.4.16.9 2013-04-23 NEUBOT(1)
