fetchnews(8)fetchnews(8)NAMEfetchnews - fetch news via NNTP
SYNOPSISfetchnews [-q] [-v] ... [-x number] [-l] [-n] [-f] [-P] [-w]
DESCRIPTION
Leafnode is a USENET package intended for small sites, where there are
few users and little disk space, but where a large number of groups is
desired.
The design of leafnode is intended to self-repair after problems, and
to require no manual maintenance.
Fetchnews (formerly called "fetch") is the program which submits and
retrieves new articles to or from the upstream NNTP server.
OPTIONS-q Be quiet. Suppresses some warnings. This flag cancels any previ‐
ously encountered -v flags.
-v Be verbose. This flag cancels any previously encountered -q
flags. A maximum of four -v can be used (the more v, the more
verbose):
-v Warn about non-critical errors.
-vv Be verbose whenever a group command occurs.
-vvv Be verbose whenever an article command occurs.
-vvvv For debugging purposes. Don't expect the additional output to
make sense.
-w Wait mode. This causes the XOVER update (which runs after the
article fetch phase) to run in the foreground. By default, the
XOVER update runs in the background, detached from the shell, on
systems that offer a working fork(2) call. Helpful for debugging
and to avoid terminal clobbering when fetchnews is run manually.
-x number
Go number articles back and try to re-fetch what's missing. This
can be useful to pull data from broken servers that have either
overview and group data out of synch or create articles out of
article number order.
-l Don't use supplementary servers (-l is supposed to mean
"local"). This does speed up the operation quite a bit.
-n Do not expire newsgroup subscriptions. Technically: do not
unlink files in the interesting.groups directory.
-f Erase the active.read file, thus force redownloading the whole
active file from all upstream servers. This is usually occurring
only very infrequently because it takes a long time. See time‐
out_active in leafnode(8). DO NOT use this option in cron jobs
or otherwise regularly.
-P Only post articles that have accumulated in the out.going direc‐
tory but do not fetch articles from upstream servers.
EXIT STATUSfetchnews will exit with code 0 if it could connect to all servers and
encountered no other errors. It exits with code 2 when it was unable to
connect to at least one server, and it exits with code 1 on other
errors.
FILES AND CONFIGURATION
/var/spool/leafnode/leaf.node/active.read
is a file to track when the active files have been successfully
downloaded from the upstream servers the last time. If it is
missing, this has the same effect as giving the -f option. It is
only created after the active file of each upstream servers has
been downloaded successfully.
See leafnode(8) for the list of other files affected, and for informa‐
tion on configuring fetchnews.
ENVIRONMENT
LN_SKIP_GROUPS=pattern[,pattern[...]] (since v1.9.53)
If set, all groups that match one of the wildmat(3) patterns
will be skipped during the fetch. This can be used as a quick
way of skipping binary groups during daytime, for instance,
example:
env 'LN_SKIP_GROUPS=*binary*,alt.*' /usr/local/sbin/fetchnews
-nq
LN_LOCK_TIMEOUT
This variable is parsed as an unsigned integer value and deter‐
mines how many seconds fetchnews will wait when trying to obtain
the lock file from another leafnode program. 0 means to wait
indefinitely. This variable takes precedence over the configura‐
tion file.
LN_SUPPRESS_DATE
(since v1.11.7) If this variable is defined (regardless of its
value, including empty), leafnode will skip checking the
upstream server's time by means of the NNTP DATE command.
ALGORITHM
Here is a brief description of what fetchnews does.
First, after connecting to the upstream NNTP server, all articles that
have the u+r permission (0400) flag set in /var/spool/leafn‐
ode/out.going are posted using POST. If a posting fails, that article
is moved to /var/spool/leafnode/failed.postings which an administrator
has to clean up once in a while. Moving a file back to
/var/spool/leafnode/out.going is sufficient to retry the post it. The
u+r flag is checked to prevent posting an (incomplete) article that
leafnode(8) is still receiving at the same time. If a posting suc‐
ceeds, the article is removed from out.going in order to not post it
twice into a moderated newsgroup on a different server (which would
send multiple copies of the article to the moderator). Note: older
fetchnews versions tried to post on all servers first, but failed to do
so when one of the servers could not be reached. If you need to work
around a dodgy upstream server that is fast but doesn't forward your
posts reliably, prohibit posting to this server using the nopost server
option in the configuration file, see leafnode(8).
Second, the upstream server's list of newsgroups (obtained using LIST
and LIST NEWSGROUPS) is merged into leafnode's (if timeout_active has
expired, the active.read file is not present or the active file is
empty) or updated (with NEWGROUPS).
Third, fetchnews iterates over the list of newsgroups, performing a
GROUP, an XOVER (if supported) or XHDR Message-ID and a number of HEAD
and BODY or ARTICLE commands for each group which has been read
recently. It assumes that for the particular group it is looking at,
"maxage" was set to the groupexpire or global expire. As a consequence,
it makes only sense to set maxage lower than or equal to expire.
NOTES
Fetchnews does not use NEWNEWS. NEWNEWS is not used because it is
often very expensive and therefore also disabled at many sites.
After some experiments, I decided that it was better to fetch all the
articles in a group than to optimize away some; users would complain
too loudly when the optmizer guessed wrongly.
The only way to control what newsgroups are accessible to local users
is to control what the upstream server lets fetchnews read, and that
will not keep out all crossposts (endemic in e.g. the alt.sex groups).
If your site is big enough that you need to control newsgroup access,
consider using a news server which is designed for a big site.
Fetchnews sets its real and effective uid to "news" when started as
root and refuses to run if it cannot get this user ID.
BUGS
Fetchnews ignores the "Supersedes:" and "Control:" headers.
HISTORY
Older versions of fetchnews posted any articles found in out.going.
Since v1.9.23, fetchnews will only post article files that have their
user executable bit set. Leafnode sets this bit after it has success‐
fully received a post. This behaviour avoids posting incomplete arti‐
cles when leafnode receives a post at the same time fetchnews is post‐
ing.
AUTHOR
Written by Arnt Gulbrandsen <agulbra@troll.no> and copyright
1995\-–\-96 Troll Tech AS, Postboks 6133 Etterstad, 0602 Oslo, Norway,
fax +47 22646949.
Modified by Cornelius Krasel <krasel@wpxx02.toxi.uni-wuerzburg.de>,
Markus Enzenberger <enz@cip.physik.uni-muenchen.de> and Randolf Skerka
<Randolf.Skerka@gmx.de>. Copyright of the modifications 1997\-–\-1999.
Modified by Matthias Andree <matthias.andree@gmx.de>, copyright of the
modifications 2001\-–\-2004.
SEE ALSOleafnode(8), texpire(8), newsq(1), /usr/local/etc/leafnode/leafn‐
ode/filters.example.
leafnode 1.11.11 fetchnews(8)
