<howto@fdd.com>
This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO.
Copyright (c) 1999, Rick Dean.
Copyright (c) 1996, Robert Hart.
The authors retain their copyright of this document. You are hereby granted permission to redistribute this document in whole or in part as long as it includes this copyright notice. Commercial redistribution is allowed and encouraged. All translations or derivative works of this document must be covered under this copyright notice, and without additional restrictions on distribution. This arrangement is also known as a copyleft.
This copyright notice, itself, is hereby placed in the public domain. You may copy it without atribution.
The latest version of this document is available at http://fdd.com/howto/.
Many Linux distributions include mini-HOWTOs in the /usr/doc/HOWTO directory.
This mini-HOWTO like most Linux mini-HOWTO's was written initially is SGML. This allows the text to be automatically translated to many formats including text, HTML, PostScript, etc. Those other formats are available somewhere.
More information about Linux documentation can be found at the Linux Documentation Project, and so many other places.
<hartr@redhat.com> in 1996<howto@fdd.com> in 1999
This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO.
A mailing list (also known as a remailer), is an address where e-mail will be resent to a list of other addresses. This is useful for colaboration of geographically disperse groups. Many standards bodies like the working groups of the IETF use mailing lists.
Unfortunately, if one is subscribed to several mailing lists, one's inbox may be routinely flooded. Furthermore, some companies (such as 3Com) specify which e-mail client (such as Lotus Notes) their employees must suffer with. Redirecting these e-mails to a news server frees people to choose a news reader and utilize refined features specifically designed for the task (of deriving signal from noise).
News servers started on the Internet long ago, many years before the WWW. They (and the news reader clients) have features such as...
A big focus of news servers is sharing news between servers. The largest of these groups became known as USENET. This mini-HOWTO does not address that. You could share the newsgroups created with this mini-HOWTO on your own, but you will live just fine without it. Like a web server, ubiquitous Internet connectivity has made centralized news servers acceptable. Furthermore, recent benchmarks have shown single-processor Linux boxes can handle 1300+ HTTP hits per second, so scalability is a minor issue.
Athough you do not need to own the mailing list to use mail2news, it is a good idea to own the news server.
This document assumes you are using Linux, but other Unices are nice.
Currently only sendmail (for a mail delivery agent) is described, but as
qmail grows in popularity (in part because it is easier to configure).
Hopefully someone with submit configuration notes for it too. For a
news server, this document describes innd. It is pretty dominant
as news servers go, but any NNTP compliant one should work.
A bit of glue called mail2news.pl is a perl script, thus you
need the Perl interpreter, but it is
very common and probably already installed. Finally, I assume
you are running all this (except the mailing list remailer) on
one machine. Dividing it up is left as an exercise for the
reader. :-)
At the time of this writing this mini-HOWTO was only tested against a RedHat-6.0 distribution. As a good computer scientist, you should not believe anything works until it has been specifically tested. Any feedback or notes relating to other distributions would be welcomed by the author.
For most of this mini-HOWTO you will need root access unless otherwise specified.
This mini-HOWTO is presented backwards, as this is the easiest way to build and debug it. Backwards means we start with the newsreader and work upstream to the mailing list remailer, the opposite direction of normal data flow. This systems uses several hairy pieces (like sendmail and innd) which are sizeable mini-HOWTO's in thier own right.
This mini-HOWTO does not cover...
Please do not e-mail me about these subjects (or SPAM).
Netscape comes with an integrated news reader. The easy way to subscribe to a group is type in (or click on) a link like
news://fuji.sfour.com/ietf.confctrl
Once you have subscribed, you only need to go to the message center. This can be done by clicking on the small talk balloons icon in the bottom right of a browser window.
Free Agent is a wonderful news reader by Forte for Windoze. See http://www.forteinc.com/agent/freagent.htm
The old command line newsgreaders like trn and tin are a good
standby. Be sure to set the environment variable NNTPSERVER first.
For example...
export NNTPSERVER=fuji.sfour.com
trn
You need to install a news server. I used inn
but others are available (somewhere).
Preferably just check the "News Server" box during your initial install, but alternatively if your distribution uses RPM (RedHat package manager), then use something like...
rpm -i inn-2.2.9.i386.rpm
To manually start or stop the news server, use a command like
/etc/rc.d/init.d/innd start
/etc/rc.d/init.d/innd stop
/etc/rc.d/init.d/innd restart
To have the news server start at boot you could add a command
like this to end of /etc/rc.d/rc.local, but that is not the best.
Many distributions have a graphical tool for choosing which daemons
run. You can also try the command line program
setup
chkconfig --add innd
innd is pretty picky about permissions and ownership. For
much of the news config you will need to be the user news.
To become this user from root...
su - news
Using ctlinnd, create the newsgroup on your news server. Remember, the
newsgroup will be local, so start it with a distinctive name
so you can filter it out from your news distributions
if you do that stuff. I shamelessly named my newsgroup
ietf.confctrl. The words from left to right go from less to more specific.
You also need to tell innd that the group is moderated (by using
ctlinnd). Indicating a moderated group is done by specifying
m to the newgroup command. For example...
ctlinnd newgroup ietf.confctrl m confctrl@isi.edu
The newsgroup is set up as a moderated group, as this allows us to take advantage of the email capabilities of innd. Any messages posted to a moderated group are not immediately submitted to the group. Instead, messages are emailed to the moderator of the group. In our example confctrl@isi.edu is the address which is resent by the remailer.
If you are sharing news with other servers, remember to edit your newsfeeds so that this group is not not distributed (unless you specificaly wish this to occur).
By default the news server, doesn't let any clients read news, so I needed to disable the user authentication of innd. This was done in the /etc/news/nnrp.access. Check out the nnrp.access man page to learn the syntax of this file. I changed the first non-comment line to ...
*:Read Post:::*
If you want a username/password, fill in the 3rd and 4th (colin separated) fields. For more information on the syntax, check the man page...
man nnrp.access
I had to change the permissions of /usr/bin/rnews. It was
not world read/executable, but sendmail runs scripts as nobody.
chmod a+rx /usr/bin/rnews
If you copy the following article to a file named rick.article...
Path: rick From: rick@fdd.com Message-ID: <199907120548.AAA05475@fdd.com> Subject: test Date: Mon, 12 Jul 1999 00:48:49 -0500 (CDT) Newsgroups: ietf.confctrl Approved: ietf-confctrl@kepler.hedland.edu.au NNTP-Posting-Host: localhost Organisation: (mail2news gateway) test
Then you should be able to post a file with...
/usr/bin/rnews -r localhost <rick.article
You don't need to wait for
the article to show up as unread, just look at the /var/spool/news/articles/
subdirectories for files being created.
You may want to increase the expiration time for articles of your new newsgroup. In my case I wanted them never to expire, so I added the following line....
ietf*:A:never:never:never
/etc/news/expire.ctl file.
To learn more about the syntax of this file type...
man expire.ctl
Perl stands for Practical Extension and Report Language. It is very popular for small scripts which manipulate text which is exactly what we need.
Perl is installed by default on almost every Unix system.
If you perl intepreter is in an unusual place (not /usr/bin/) then
you will have to modify the first line of the script.
If this line is wrong, on my 2.2 kernel system I get
"bash: /usr/local/bin/mail2news.pl: No such file or directory"
Can we please change this to "bash: /usr/local/bin/mail2news.pl:
Interpreter not found. Check first line of script." ?
#!/usr/bin/perl
($program = $0) =~ s%.*/%%;
#( $version ) = $] =~ /(\d+\.\d+).*\nPatch level/;
#die "$program: requires at least version 3 of perl\n"
# if $version < 3;
# $news_poster_program = "/usr/bin/inews";
# $news_poster_options = "-h -o \"mail2news gateway\"";
$news_poster_program = "/usr/bin/rnews";
$news_poster_options = "-r localhost";
$postinghost = "localhost";
if ($#ARGV < 0) {
# $newsgroup = "test";
# we'll expect the newsgroup line in the body
} elsif ($#ARGV == 0) {
$newsgroup = $ARGV[0];
} else {
die "usage: $program [newsgroup]\n";
}
# in case inews dumps core or something crazy
$SIG{'PIPE'} = "plumber";
sub plumber { die "$program: \"$news_poster_program\" died prematurely!\n"; }
open (INEWS, "| $news_poster_program $news_poster_options") ||
die "$program: can't run $news_poster_program\n";
# header munging loop
while (<STDIN>) {
last if /^$/;
# transform real from: line back to icky style
s/^From:\s+(.*) <(.*)>/From: $2 ($1)/;
s/Message-Id/Message-ID/;
# transform from_ line to path header; also works locally
s/^From\s+(\S+)@(\S+).*/Path: $2!$1/
|| s/^From\s+(\S+)[^@]*$/Path: $1\n/;
print INEWS
# if /^(Date|From|Subject|Path|Newsgroups|Organization|Message-ID):/i;
if /^(Date|From|Subject|Path|Newsgroups|Message-ID):/i;
$saw_subject |= ( $+ eq 'Subject' );
$saw_msgid |= ( $+ eq 'Message-ID' );
# $saw_newsgroup |= ( $+ eq 'Newsgroups' );
}
warn "$program: didn't expect newsgroup in both headers and ARGV\n"
if $newsgroup && $saw_newsgroup;
die "$program: didn't get newsgroup from either headers or ARGV\n"
unless $newsgroup || $saw_newsgroup;
$approved = $newsgroup;
$approved =~ s/\./'-'/eg;
($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
$madeupid = "\<$year$mon$mday.$hour$min$sec.$$\@kepler.hedland.edu.au\>";
printf INEWS "Newsgroups: %s\n", $newsgroup if $newsgroup;
printf INEWS "Approved: %s\@kepler.hedland.edu.au\n", $approved;
print INEWS "Subject: Untitled\n" unless $saw_subject;
printf INEWS "Message-ID: %s\n", $madeupid unless $saw_msgid;
printf INEWS "NNTP-Posting-Host: %s\n", $postinghost;
print INEWS "Organisation: (mail2news gateway)\n";
print INEWS "\n";
print INEWS while <STDIN>; # gobble rest of message
close INEWS;
exit $?;
I saved the script in /usr/local/bin (and will use this path throughout the HOWTO).
Be sure to make the script executable by all, but not writable by group or other. Sendmail is picky.
chmod a+x /usr/local/bin/mail2news.pl
chmod go-w /usr/local/bin/mail2news.pl
chmod 555 /usr/local/bin/mail2news.pl
I tested this script by changed my news poster from /usr/bin/rnews to /bin/cat. I then saved an e-mail send to myself in a file. Finally I ran the mail2news.pl on the saved mail and captured the output to a file.
/usr/local/bin/mail2news.pl ietf.confctrl </tmp/savedMailFile >/tmp/article
Yes, the output of your posting script should contain the e-mail address of an austrailian. My guess is that the Austrailian's address is trusted address in your news configuration (althogh I could not find it in mine) for approving moderated postings.
If the mail2news.pl script is not in my path, I get
the error bash: mail2news.pl: command not found. You will need
to either add this directory to your path
PATH=$PATH:/usr/local/bin
/usr/local/bin/mail2news.pl
echo $PATH
This document only describes modifying sendmail. Many people prefer qmail.
Out of the box, sendmail does not come with the "pipe to program" feature enabled. Rightly so, this feature is a security risk. It enambles users to have their mail fed to a program instead of appended to a file. (Users can configure this in thier .forward file.)
Please don't believe me. Test your own system first. Much of this chapter might not be needed by you.
IMHO, sendmail installs with woefully insufficient documentation.
http://www.sendmail.org/ ca/email/doc/op-sh-5.html describes the syntax of sendmail.cf. I hope you don't have to use it.
This is a description of the features you are trashing in sendmail.mc http://www.sendmail.org/m4/features.html
Sendmail has a man page...
man sendmail
To get the "pipe to program" stuff in the aliases file to work
you need to modify the default sendmail.cf (by modifying the
sendmail.mc) so that the restricted shell is not used. I
suppose the proper solution would be to add the one program
to the restricted shell list, but their was no man page
on smrsh. Strangely, uncommenting the smrsh feature didn't
work, I needed to change the shell from /usr/sbin/smrsh
to /bin/bash. Yeah, this is slightly risky, but
it was not an issue on my machine. Without
this change I kept getting a "Service unavailable" error
message in the /var/log/maillog file.
The header of /etc/senmail.mc of RedHat-6 has a bug. The proper
command line is...
m4 /etc/sendmail.mc >/etc/sendmail.cf
sendmail.mc. Hopefully,
RedHat will extend the super cool Makefile idea in /etc/mail
You will need to install sendmail-cf.something.rpm first.
e.g. ...
rpm -i sendmail-cf-8.9.3-10.i386.rpm
Whenever you modify the sendmail.cf file, you should restart sendmail...
/etc/rc.d/init.d/sendmail restart
Instead of creating a new user account, we will only create an alias.
When modifying the /etc/aliases file, the double quotes
are required. There cannot be a space between the
first double quotes and the | (pipe) character, or
sendmail will complain "User unknown"
Add a line like ...
confctrl: "| /usr/local/bin/mail2news.pl ietf.confctrl "
Whenever you modify the /etc/aliases file you need to
notify sendmail.
sendmail -bi
Check the /var/log/maillog to see if it worked, or for error messages. I found it useful to open up another terminal (ssh) window to watch the log with
tail -f /var/log/maillog
If you are having trouble, and create an e-mail alias with a
different (simpler) target program to test it, remember that sendmail
runs the program as an unprivledged user, who probably doesn't
have privledges to write anywhere except globally writable directories
such as /tmp.
The method of subscribing to each e-mail list is different, although most involve sending some kind of e-mail message. Please be very careful not to send to the replicating address of the mailgroup. This is a sure way to annoy the members of the list. Most remailers (but not all) have a separate address for subscribing. In confctrl, the administration address is confctrl-request@isi.edu
Some mailing lists do not let you subscribe an address you are not mailing from. In this case you will need to forge the return address of an e-mail.
When you successfully subscribe you should (hopefully) see a welcome message in the newsgroup.
Forgeing mail is easiest to do with an old Netscape client. It would accept anything for a return address. Fortunately, modern browsers require you to retreive mail successfully for an address before they will let you send. Thus you are required to fall back the tried and true method....telnet.
When you speficy a web page, your browser does a DNS lookup to convert the domain name into an IP address. This is a lookup of an "A" record. (Also "CNAME" records are considered.) When sending mail a "MX" record is looked up. If this is missing a "CNAME" or "A" one is used. Thus, to forge mail you need to do an MX lookup.
dig mx isi.edu
; <<>> DiG 8.2 <<>> mx isi.edu
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4
;; QUERY SECTION:
;; isi.edu, type = MX, class = IN
;; ANSWER SECTION:
isi.edu. 1D IN MX 0 tnt.isi.edu.
isi.edu. 1D IN MX 10 venera.isi.edu.
;; AUTHORITY SECTION:
isi.edu. 1D IN NS venera.isi.edu.
isi.edu. 1D IN NS ns.isi.edu.
isi.edu. 1D IN NS east.isi.edu.
;; ADDITIONAL SECTION:
tnt.isi.edu. 1D IN A 128.9.128.128
venera.isi.edu. 1D IN A 128.9.176.32
ns.isi.edu. 1D IN A 128.9.128.127
east.isi.edu. 1D IN A 38.245.76.2
;; Total query time: 448 msec
;; FROM: fdd.com to SERVER: default -- 127.0.0.1
;; WHEN: Sun Jul 25 15:49:32 1999
;; MSG SIZE sent: 25 rcvd: 182
Mail is delivered using the Simple Mail Transport Protocol (SMTP). Like most good Internet protocols, it is ASCII based to make troubleshooting and development easier. I will not explain everything, but simply give an example. Hopefully, this is enough.
The protocol is line oriented. Each email as specified in RFC822, is composed of headers and body which are separated by the first blank line (no characters, not even spaces). SMTP specifies the end of an e-mail with a line containing only a period.
halyard$ telnet tnt.isi.edu 25
Trying 128.9.128.128...
Connected to tnt.isi.edu.
Escape character is '^]'.
220 tnt.isi.edu ESMTP Sendmail 8.8.7/8.8.6; Sun, 25 Jul 1999 14:01:25 -0700 (PDT)
helo isi.edu
250 tnt.isi.edu Hello rick@node-d8e9822 [216.233.8.34] (may be forged), pleased to meet you
mail from:<confctrl@fuji.sfour.com>
250 <confctrl@fuji.sfour.com>... Sender ok
rcpt to:<confctrl-request@isi.edu>
250 <confctrl-request@isi.edu>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
From:<confctrl@fuji.sfour.com>
Subject: help
help