Leafnode

NAME¶

fetchnews - fetch news via NNTP

SYNOPSIS¶

fetchnews -V
fetchnews [GLOBAL OPTIONS] [-BfHnPR] [-M message-id] [...] [-x number] [-S server[:port]] [-N {newsgroup|group.pattern}] [...] [-t #]

DESCRIPTION¶

Leafnode is a USENET package intended for small sites, where there are few users and little disk space, but where a large number of groups is desired.

The design of leafnode is intended to self-repair after problems, and to require no manual maintenance.

Fetchnews(8) is the program which submits and retrieves new articles to or from the upstream NNTP server.

OPTIONS¶

For a description of the GLOBAL OPTIONS, see leafnode(8).

-B: Get article bodies from groups that are in "delaybody" mode. Done by default unless one of -H -M -R -P is used. See the configuration section of leafnode(8) for "delaybody".
-f: Force rereading of whole active file from all upstream servers. This is usually occurring only very infrequently because it takes a long time.
Can not be used with -S.
-H: Get article headers for groups that are in "delaybody" mode. Done by default unless one of -B -M -R -P is used. See the configuration section of leafnode(8) for "delaybody".
-M message-id: Get article with this message-id. NOTE: The message-id MUST be surrounded by angle brackets. Use single quotes to protect the message-ID angle brackets, dollar signs and other meta characters from shell interpretation. The -M option can be used multiple times to get more than one article. Example:
fetchnews -M '<1234$5678@example.org>'
-n: Don't unlink files in the interesting.groups directory. This avoids automatic unsubscription of newsgroups, see EXPIRY OF NEWSGROUPS below.
-N newsgroup: Get only articles posted to newsgroup. The -N option can be used multiple times to get articles from more than one group. -N group.pattern Get articles only from interesting groups matching group.pattern.
-P: Post articles that have accumulated in the out.going directory. Done by default unless one of -B -H -M -R is used.
-R: Get articles from groups that are not in "delaybody" mode. Done by default unless one of -B -H -M -P is used. See the configuration section of leafnode(8) for "delaybody".
-S server[:port]: Only get articles on "server". You can specify this option multiple times with different servers if you want to query several servers at once.
If a server is running its NNTP service not on the standard port, you can specify an alternative port by adding a colon and the port number immediately behind the server name, without any whitespace inbetween.
Can not be used with -f.
-t number: Wait this many seconds before fetching next article. This is useful if you need bandwidth for something else, but it makes fetching slow.
-x number: Go number articles back and try to re-fetch what's missing. Useful to fill the gaps after filter rules that used to be too aggressive have been relaxed.

ALGORITHM¶

Here is a brief description of what fetchnews does.

First, it checks which newsgroups are no longer requested by clients, according to the procedure shown below in the section EXPIRY OF NEWSGROUPS.

Then it will connect to the upstream NNTP servers one after another, and, unless changed by options, perform these steps:

First after connecting to the upstream NNTP server, all articles in /var/spool/news/out.going are posted using POST. If a posting fails, that article is moved to /var/spool/news/failed.postings which an administrator has to clean up once in a while. Moving a file back to /var/spool/news/out.going is sufficient to retry the submission.

Second, the upstream server's list of newsgroups (obtained using LIST ACTIVE and LIST NEWSGROUPS) is merged into leafnode's (this step is most often optimized away by using NEWGROUPS).

Third, fetchnews iterates over the list of newsgroups, performing a GROUP, an XOVER and a number of ARTICLE commands for each group which has been read recently.

EXPIRY OF NEWSGROUPS¶

Fetchnews(8) will drop newsgroup subscriptions after a certain amount of time a particular "interesting" group has not been read, unless run with the -n option.

Leafnode(8) will track for each group (a) when the last posting has been downloaded from the upstream (.last.posting), (b) when an article in the group or the group overview data has been requested by a client. Newsgroups are removed from the interesting.groups directory and hence no longer fetched when the newsgroup has not been accessed by any reader for timeout_long days after the freshest posting has arrived, or timeout_short days when the group has never been accessed since the placeholder article has been read.

This avoids the unsubscription of low-traffic newsgroups, for instance, announcements groups. As a consequence, newsgroups will only be removed from interesting.groups when fetchnews(8) has downloaded postings AFTER you have unsubscribed the groups in your newsreader.

Technically, the arrival time of the freshest postings former is marked in the ctime of the .last.posting file in the group directory by fetchnews(8), and the time of the last access by newsreaders is marked in the ctime of the group.name file below /var/spool/news/interesting.groups by leafnode(8).

For a newly subscribed group that has not been read more than once, the interesting.group/group.name's ctime will match its mtime. In this case, timeout_short applies instead of timeout_long.

You can use "ls -lc" or "ls -ltc" or the lsmac.pl(1) utility to view the ctime(s) of a particular directory or file or set of files.

Dormant groups do not expire.

Note that this time mechanism does not recover when your system clock is changed, so make sure your system clock is right. A few minutes off don't matter practically; daylight savings time is invisible to leafnode and hence does not matter either.

It is a good idea to run some NTP software, for instance, "ntpd" or "chrony", against your ISP's NTP server, when you've got a permanent internet connection, or to run "ntpd -g -q -x" or "ntpdate" right after your dialup connection goes online. Chrony and Ntpd are third-party packages that do not ship with leafnode and the documentation of which are beyond the scope of this manual.

FILES AND CONFIGURATION¶

leafnode(8) for the list of files affected, and for information on configuring fetchnews. If you want to use filtering, see also filterfile(5) for the format of the filterfile and pcre(3) on the format of perl-compatible regular expressions.

NOTES¶

Fetchnews does not use NEWNEWS because it is often very expensive on the remote server.

The only way to control what newsgroups are accessible to local users is to control what the upstream server lets fetchnews read, and that will not keep out all crossposts (endemic in e.g. the alt.sex groups). If your site is big enough that you need to control newsgroup access, consider using a news server which is designed for a big site.

Fetchnews sets its real and effective uid to "news", because almost all of the problems I have had with leafnode are due to files which aren't writable for user news.

BUGS¶

Fetchnews ignores "Also-Cancel" headers.

Fetchnews cannot delete groups from the list of active newsgroups without getting full active files from all upstream servers. Strictly spoken, this is not a fetchnews bug but a weakness of the underlying NNTP protocol.

AUTHOR¶

Modified and Copyright 1997 - 2003 by Cornelius Krasel <krasel@wpxx02.toxi.uni-wuerzburg.de>, Markus Enzenberger <enz@cip.physik.uni-muenchen.de>, Randolf Skerka <Randolf.Skerka@gmx.de> and Jörg Lüders.
Written and Copyright 2002 by Jörg Dietrich <joerg@dietrich.net>.
Written and Copyright 2002 - 2004 by Ralf Wildenhues <ralf.wildenhues@gmx.de>.
Written and Copyright 2000 - 2005 by Matthias Andree <matthias.andree@gmx.de>.

Source file:	fetchnews.8.en.gz (from leafnode 2.0.0+git.1677927696.44d2783-3.6)
Source last updated:	2023-03-04T12:29:58Z
Converted to HTML:	2024-05-04T01:26:47Z