Chapter 24. Mail and news

Table of Contents

24.1. sendmail
24.1.1. Configuration with genericstable
24.1.2. Testing the configuration
24.1.3. Using an alternative MTA
24.2. fetchmail
24.3. Reading and writing mail with mutt
24.4. Strategy for receiving mail
24.5. Strategy for sending mail
24.6. Advanced mail tools
24.7. News with tin

This chapter explains how to set up NetBSD to use mail and news. Only a simple but very common setup is described: the configuration of a host connected to the Internet with a modem through a provider. You can think of this chapter as the continuation of Chapter 21, Setting up TCP/IP on NetBSD in practice, assuming a similar network configuration. Even this “simple” setup proves to be difficult if you don't know where to start or if you've only read introductory or technical documentation; in fact you will notice that some details are really challenging (for example, mapping your internal network names to “real” names requires a good knowledge of sendmail). A general description of mail and news configuration is beyond the scope of this guide; please read a good Unix Administration book (some very good ones are listed on the NetBSD site). The problem is in fact very complex because of the myriad of possible configurations and connections and because it's not enough to configure a single program - you need to correctly match the configuration of several cooperating components.

This chapter also briefly describes the configuration (but not the usage) of two popular applications, mutt for mail and tin for news. The usage is not described because they are easy to use and well documented. Obviously, both mutt and tin are not mandatory choices: many other similar applications exist but I think that they are a good starting point because they are widely used, simple, work well and don't use too much disk space and memory. Both are console mode programs; if you prefer graphics applications there are also many choices for X.

In short, the programs required for the configuration described in this chapter are:

Of these, only sendmail and m4 are installed with the base system; you can install the other programs from the NetBSD package collection, pkgsrc - see Chapter 30, The package collection.

Before continuing, remember that none of the programs presented in this chapter is mandatory: there are other applications performing similar tasks and many users prefer them. You'll find different opinions reading the mailing lists. You can also use different strategies for sending and receiving mail: the one explained here is only a starting point; once you understand how it works you'll probably want to modify it to suit your needs or to adopt a different method altogether.

At the opposite extreme of the example presented here, there is the usage of an application like Mozilla, which does everything and frees you from the need of configuring many components: with Mozilla you can browse the Internet, send and receive mail and read news. Besides, the setup is very simple. There is a price to pay, though: Mozilla is a “closed” program that will not cooperate easily with other standard Unix utilities.

Another possibility is to use emacs to read mail and news. Emacs needs no introduction to Unix users but, in case you don't know, it is an extensible editor (although calling emacs an editor is somewhat reductive) which becomes a complete work environment, and can be used to read mail, news and to perform many operations. For many people emacs is the only environment that they need and they use it for all their work. The configuration of emacs for mail and news is described in the emacs manual.

In the rest of this chapter we will deal with a host connected to the Internet through a PPP connection via serial modem to a provider.

First some basic terminology:

MUA (mail user agent)

a program to read and write mail. For example: mutt, elm and pine but also the simple mail application installed with the base system.

MTA (mail transfer agent)

a program that transfers mail between two host but also locally (on the same host). The MTA decides the path that the mail will follow to get to the destination. On BSD systems (but not only) the standard MTA is sendmail, other examples are postfix, qmail, exim and Microsoft Exchange.

MDA (mail delivery agent)

a program, usually used by the MTA, that delivers the mail; for example, it physically puts the messages in the recipient's mailbox. For example, sendmail uses one or more MDA to deliver mail, and procmail is another well-known MDA.

Figure 24.1, “Structure of the mail system” depicts the mail system that we want to set up. Between the local network (or the single host) and the provider there is a modem PPP connection. The “bubbles” with the thick border (sendmail, fetchmail, mutt) are the programs launched manually by the user; the remaining bubbles are the programs that are launched automatically. The circled numbers refere to the logical steps of the mail cycle:

  1. In step (1) mail is downloaded from the provider's POP server using fetchmail, which hands messages off to sendmail to put the messages in the user's mailbox.

  2. In step (2) the user launches mutt (or another MUA) to read mail, reply and write new messages.

  3. In step (3) the user “sends” the mail from within mutt. Messages are accumulated in the spool area.

  4. In step (4) the user calls sendmail to transfer the messages to the provider's SMTP server, that will deliver them to the final destination (possibly through other mail servers). The provider's SMTP server acts as a relay for our mail.

The connection with the provider must be up only during steps (1) and (4); for the remaining steps it is not needed.

Figure 24.1. Structure of the mail system

Structure of the mail system

24.1. sendmail

When an MTA must deliver a local message, it is delivered directly. If the message is intended for a different domain, the MTA must find out the address of the mail server for that domain. Sendmail uses the DNS service (described in Chapter 23, The Domain Name System) to find a mail exchanger handling mail for the given domain, and delivers the message to that mail server then.

The most used MTA in the world is probably sendmail. Sendmail is controlled by a set of configuration files and databases, of which /etc/mail/sendmail.cf is the most important. In general, if you are not an expert it is better not to modify the /etc/mail/sendmail.cf file directly; instead, use a set of predefined macros and the m4 preprocessor, which greatly (well, almost) simplify the configuration.

Note

Prior to version 1.5 of NetBSD, the mail configuration files were in /etc instead of /etc/mail.

Even using the macros, the configuration of sendmail is not for the faint of heart, and the next sections only describe an example which can be modified to suit different needs and different configurations. If you connect to the Internet with a modem, the example configuration file will probably fit all your needs: just replace the fictitious data with yours.

The first problem to be solved is that the local network we are dealing with is an internal network, i.e. not directly accessible from the Internet. This means that the names used internally have no meaning on the Internet; in short, “ape.insetti.net” cannot be reached by an external host: no one will be able to reply to a mail sent with this return address (many mail systems will even reject the message as spam prevention as it comes from an unknown host). The true address, the one visible from everybody, is assigned by the provider and, therefore, it is necessary to convert the local address “carlo@ape.insetti.net” to the real address “alan@bignet.it”. Sendmail, if correctly configured, will take care of this when it transfers the messages.

You'll probably also want to configure sendmail in order to send the e-mails to the provider's mail server, using it as a relay. In the configuration described in this chapter, sendmail does not directly contact the recipient's mail server (as previously described) but relays all its mail to the provider's mail server.

Note

The provider's mail server acts as a relay, which means that it delivers mail which is not destined to its own domain, to another mail server. It acts as an intermediary between two servers.

Since the connection with the provider is not always active, it is not necessary to start sendmail as a daemon in /etc/rc.conf: you can disable it with the line “sendmail=NO”. As a consequence it will be necessary to launch sendmail manually when you want to transfer mail to the provider. Local mail is delivered correctly even if sendmail is not active as a daemon.

Let's start configuring sendmail.

24.1.1. Configuration with genericstable

This type of configuration uses the file /etc/mail/genericstable which contains the mapping used by sendmail to rewrite the internal hostnames.

The first step is therefore to write the genericstable file. For example:

carlo:         alan@bignet.it
root:          alan@bignet.it
news:          alan@bignet.it

These entries will map the mail sent from the users given on the left side into the globally valid email addresses given on the right, making it appear as if the mail was really sent from that address.

For the sake of efficiency, genericstable must be transformed into a binary file with the following command:

# /usr/sbin/sendmail -bi -oA/etc/mail/genericstable

Now it's time to create the prototype configuration file which we'll use to create the sendmail configuration file.

# cd /usr/share/sendmail/cf

The new sendmail configuration file, which we'll call mycf.mc, contains:

divert(-1)dnl
include(`../m4/cf.m4')dnl
VERSIONID(`mycf.mc created by carlo@ape.insetti.net May 18 2001')dnl
OSTYPE(bsd4.4)dnl

dnl # Settings for masquerading.  Addresses of the following types
dnl # are rewritten
dnl #     carlo@ape.insetti.net
dnl #     carlo@ape
GENERICS_DOMAIN(ape.insetti.net ape)dnl
FEATURE(genericstable)dnl
FEATURE(masquerade_envelope)dnl

define(`SMART_HOST',`mail.bignet.it')dnl

FEATURE(redirect)dnl
FEATURE(nocanonify)dnl

dnl # The following feature is useful if sendmail is called by
dnl # fetchmail (which is usually the case).  If sendmail cannot
dnl # resolve the name of the sender, the mail will not be delivered.
dnl # For example:
dnl #	MAIL FROM:<www-owner@NetBSD.org> SIZE=2718
dnl #	501 <www-owner@NetBSD.org>... Sender domain must exist
FEATURE(`accept_unresolvable_domains')dnl

dnl # accept_unqualified_senders is useful with some MUA, which send
dnl # mail as, for example:
dnl #	MAIL FROM:<carlo>
FEATURE(`accept_unqualified_senders')dnl

dnl # Mail for `smtp' mailer is marked `expensive' (`e' flag):
dnl # instead of connecting with the relay, sendmail puts it in
dnl # a queue for delayed processing.
dnl # Sendmail starts complaining about undelivered messages after
dnl # 16 hours.
define(`SMTP_MAILER_FLAGS',`e')dnl
define(`confCON_EXPENSIVE',`True')dnl
define(`confTO_QUEUEWARN', `16h')dnl

dnl # For european users
define(`confDEF_CHAR_SET',`ISO-8859-1')dnl

dnl # Enable the following three lines to use procmail as a local
dnl # delivery agent.  The third line is optional, only the first
dnl # two are required.
dnl # define(`PROCMAIL_MAILER_PATH', /usr/pkg/bin/procmail)dnl
dnl # FEATURE(local_procmail)dnl
dnl # MAILER(procmail)dnl

dnl # The following two mailers must always be defined
MAILER(local)dnl
MAILER(smtp)dnl

Note

In the previous example, everything after a “dnl” is considered a comment and will be discarded by the m4 preprocessor.

This configuration tells sendmail to rewrite the addresses of type “ape.insetti.net” using the real names found in the /etc/mail/genericstable file. It also says that mail should be sent to the “mail.bignet.it” server. The meaning of the options is described in detail in the file /usr/share/sendmail/README.

In order to create your own version of the example configuration file, you must change only two lines, substituting your real data:

GENERICS_DOMAIN(ape.insetti.net ape)dnl
define(`SMART_HOST',`mail.bignet.it')dnl

Finally, the new configuration file must be generated, after having saved the previous version:

# cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.bak
# m4 mycf.mc > /etc/mail/sendmail.cf

Note that there is afile netbsd-proto.mc in the /usr/share/sendmail/cf directory which is used to create the default /etc/mail/sendmail.cf shipped with NetBSD. With the make command it can be rebuilt, if needed.

Now everything is ready to start sending mail.

24.1.2. Testing the configuration

Sendmail is finally configured and ready to work, but before sending real mail it is better to do some simple tests. First let's try sending a local e-mail with the following command:

$ sendmail -v carlo
Subject: test

Hello world
.
carlo... Connecting to local...
carlo... Sent

Please follow exactly the example above: leave a blank line after Subject: and end the message with a line containing only one dot. Now you should be able to read the message with your mail client and verify that the From: field has been correctly rewritten.

From: alan@bignet.it

Next you can verify the address rewriting rules directly, using sendmail in address test mode with the -bt option. This mode shows the parsing performed by sendmail for an address and how it gets rewritten according to the rules in the configuration file. It is also possible to perform other tests and view some information.

$ sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

You can display the help with the “?” command.

First, let's verify that the generictable map file works correctly:

/map generics carlo
map_lookup: generics (carlo) returns alan@bignet.it

Everything's ok here; sendmail found the local name and its real counterpart in the map.

Now we can test the rewriting of the envelope's sender address with the following commands:

/tryflags ES
/try smtp carlo@ape.insetti.net

The result should be similar to the following:

Trying envelope sender address carlo@ape.insetti.net for mailer smtp
rewrite: ruleset   3   input: carlo @ ape . insetti . net
rewrite: ruleset  96   input: carlo < @ ape . insetti . net >
rewrite: ruleset  96 returns: carlo < @ ape . insetti . net . >
rewrite: ruleset   3 returns: carlo < @ ape . insetti . net . >
rewrite: ruleset   1   input: carlo < @ ape . insetti . net . >
rewrite: ruleset   1 returns: carlo < @ ape . insetti . net . >
rewrite: ruleset  11   input: carlo < @ ape . insetti . net . >
rewrite: ruleset  51   input: carlo < @ ape . insetti . net . >
rewrite: ruleset  51 returns: carlo < @ ape . insetti . net . >
rewrite: ruleset  61   input: carlo < @ ape . insetti . net . >
rewrite: ruleset  61 returns: carlo < @ ape . insetti . net . >
rewrite: ruleset  94   input: carlo < @ ape . insetti . net . >
rewrite: ruleset  93   input: carlo < @ ape . insetti . net . >
rewrite: ruleset   3   input: alan @ bignet . it
rewrite: ruleset  96   input: alan < @ bignet . it >
rewrite: ruleset  96 returns: alan < @ bignet . it >
rewrite: ruleset   3 returns: alan < @ bignet . it >
rewrite: ruleset  93 returns: alan < @ bignet . it >
rewrite: ruleset  94 returns: alan < @ bignet . it >
rewrite: ruleset  11 returns: alan < @ bignet . it >
rewrite: ruleset   4   input: alan < @ bignet . it >
rewrite: ruleset   4 returns: alan @ bignet . it
Rcode = 0, addr = alan@bignet.it
>

As you can see, the local address has been rewritten to the real address, which will appear in your e-mails when they leave your system.

You can achieve a similar result with the following command:

/try smtp carlo

We can also verify the rewriting of the header's sender with the following commands:

/tryflags HS
/try smtp carlo@ape.insetti.net

24.1.3. Using an alternative MTA

Starting from version 1.4 of NetBSD sendmail is not called directly:

$ ls -l /usr/sbin/sendmail
lrwxr-xr-x  1 root  wheel  21 Nov  1 01:14 /usr/sbin/sendmail@ -> /usr/sbin/mailwrapper

The purpose of mailwrapper is to allow the usage of an alternative MTA instead of sendmail (for example, postfix). If you plan to use a different mailer I suggest that you read the mailwrapper(8) and the mailer.conf(5) manpages, which are very clear.

24.2. fetchmail

If someone sends me mail, it is received and stored by the provider, and not automatically transfered to the local hosts; therefore it is necessary to download it. Fetchmail is a very popular program that downloads mail from a remote mail server (using e.g. the Post Office Protocol, POP) and forwards it to the local system for delivery (usually using sendmail). It is powerful yet easy to use and configure: after installation, the file ~/.fetchmailrc must be created and the program is ready to run (~/.fetchmailrc contains a password so appropriate permissions on the file are required).

This is an example .fetchmailrc:

poll mail.bignet.it
protocol POP3
username alan there with password pZY9o is carlo here
flush
mda "/usr/sbin/sendmail -oem %T"

The last line (“mda ...”) is used only if sendmail is not active as daemon on the system. Please note that the POP-mail server indicated in this file (mail.bignet.it) is only used to retrieve mails, and that it is not necessary the same as the mail relay used by sendmail to send out mails.

After setting up the .fetchmailrc file, the following command can be used to download and deliver mail to the local system:

$ fetchmail

The messages can now be read with mutt.

24.3. Reading and writing mail with mutt

Mutt is one of the most popular mail programs: it is “lightweight”, easy to use and has lots of features. The man page mutt is very bare bones; the real documentation is in /usr/pkg/share/doc/mutt/, in particular manual.txt.

Mutt's configuration is defined by the ~/.muttrc file. The easiest way to create it is to copy mutt's example muttrc file (usually /usr/pkg/share/examples/mutt/sample.muttrc) to the home directory and modify it. The following example shows how to achieve some results:

  • Save a copy of sent mail.

  • Define a directory and two files for incoming and outgoing mail saved by mutt (in this example the directory is ~/Mail and the files are incoming and outgoing).

  • Define some colors.

  • Define an alias.

set copy=yes
set edit_headers
set folder="~/Mail"
unset force_name
set mbox="~/Mail/incoming"
set record="~/Mail/outgoing"
unset save_name

bind pager <up> previous-page
bind pager <down> next-page

color normal white black
color hdrdefault blue black
color indicator white blue
color markers red black
color quoted cyan black
color status white blue
color error red white
color underline yellow black

mono quoted standout
mono hdrdefault underline
mono indicator underline
mono status bold

alias pippo Pippo Verdi <pippo.verdi@pluto.net>

To start mutt:

$ mutt

Please note that mutt supports color, but this depends on the terminal settings. Under X you can use "xterm-color", for example:

$ env TERM=xterm-color mutt

24.4. Strategy for receiving mail

This section describes a simple method for receiving and reading mail. The connection to the provider is activated only for the time required to dowload the messages; mail is then read offline.

  1. Activate the connection to the provider.

  2. Run fetchmail.

  3. Deactivate the connection.

  4. Read mail with mutt.

24.5. Strategy for sending mail

When mail has been written and “sent” with mutt, the messages must be transferred to the provider with sendmail. Mail is sent from mutt with the y command, but this does not really send it; the messages are enqueued in the spool area; if sendmail is not active as a daemon it is necessary to start it manually or the messages will remain on the hard disk. The necessary steps are:

  1. Write mail with mutt, send it and exit mutt. You can check if and what messages are in the sendmail mail queue using the mailq(1) program.

  2. Activate the connection with the provider.

  3. If your provider requires you to do "SMTP-after-POP", i.e. it first wants to make sure to know who you are before you are allowed to send mail (and no spam), you need to run fetchmail again first.

  4. Write the command /usr/sbin/sendmail -q -v to transfer the queued messages to the provider.

  5. Deactivate the connection.

24.6. Advanced mail tools

When you start using mail, you won't probably have very sophisticated requirements and the already described standard configuration will satisfy all your needs. But for many users the number of daily messages will increase with time and a more rational organization of the mail storage will become necessary, for example subdividing mail in different mail boxes organized by topic. If, for example, you subscribe to a mailing list, you will likely receive many messages every day and you will want to keep them separate from the rest of your mail. You will soon find that you are spending too much time every day repeating the same manual operations to organize your mail boxes.

Why repeat the same operations manually when you can have a program perform them automatically for you? There are numerous tools that you can add to your mail system to increase its flexibility and automatically process your messages. Amongst the most known and used there are:

  • procmail, an advanced mail delivery agent and general purpose mail filter for local mail, which automatically processes incoming mail using user defined rulesets. It integrates smoothly with sendmail.

  • spamassassin or spamprobe, to help fight spam.

  • metamail, a tool to process attachments.

  • formail, a mail formatter.

In the remaining part of this section a sample configuration for procmail will be presented for a very common case: delivering automatically to a user defined mailbox all the messages coming from a mailing list. The configuration of sendmail will be modified in order to call procmail directly (procmail will be the local mailer used by sendmail). and a custom configuration file for procmail will be created.

First, procmail must be installed using the package system (mail/procmail) or pkg_add.

Next, the configuration of sendmail must be changed, in order to use procmail as local mailer. Uncomment the following three lines from the mycf.mc sendmail M4 prototype file and recreate the sendmail configuration file as described above:

define(`PROCMAIL_MAILER_PATH', /usr/pkg/bin/procmail)dnl
FEATURE(local_procmail)dnl
MAILER(procmail)dnl

The first line defines the path of the procmail program (you can see where procmail is installed with the command which procmail). The second line tells sendmail to use procmail for local mail delivery and the third adds procmail to the list of sendmail's mailers. The third line adds procmail to the list of sendmail's mailers (this line is optional).

The last step is the creation of the procmail configuration file, containing the recipes for mail delivery.

Let's say that, for example, you subscribed to a mailing list on roses whose address is “roses@flowers.org” and that every message from the list contains the following line in the header:

Delivered-To: roses@flowers.org

Assuming you want to automatically sort all mails going over that list into the local mail folder "roses_list", the procmail configuration file (.procmailrc) looks like this:

PATH=/bin:/usr/bin:/usr/pkg/bin
MAILDIR=$HOME/Mail
LOGFILE=$MAILDIR/from

:0
* ^Delivered-To: roses@flowers.org
roses_list

The previous file contains only one rule, beginning with the line containing “:0”. The following line identifies all messages containing the string “Delivered-To: roses@flowers.org” and the last line says that the selected messages must go to the roses_list mailbox (which you should have created in $MAILDIR). The remaining messages will be delivered to the default mailbox. Note that $MAILDIR is the same directory that you have configured with mutt:

set folder="~/Mail"

Of course the mailing list is only an example; procmail is a very versatile tool which can be used to filter mail based on many criteria. As usual, refer to the man pages for more details: procmail(1), procmailrc(5), and procmailex(5) (this last one contains many examples of configuration files).

You can check that procmail is used as local mailer by sendmail if you run the latter in test mode:

$ /usr/sbin/sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

The following command displays the list of mailers known to sendmail:

> =M

You should find a line like the following one:

mailer 3 (local): P=/usr/pkg/bin/procmail S=EnvFromL/HdrFromL ...

24.7. News with tin

The word news indicates the set of messages posted to the USENET newsgroups, a service available on the Internet. Each newsgroup contains articles related to a specific topic. Reading a newsgroup is different than reading a mailing list: when you subscribe to a mailing list you receive the articles by mail and you read them with a standard mail program like mutt, which you use also to send replies. News, instead, are read directly from a news server with a dedicated program called newsreader like, for example, tin. With tin you can subscribe to the newsgroups that you're interested in and follow the threads. A thread is a sequence of articles which all derive from an article that we could call “original”. In short, a message is sent to the group, someone answers, other people answer to those who answered in the first place and so on, creating a tree like structure of messages and replies: without a newsreader it is impossible to understand the correct sequence of messages.

After the installation of tin (from the package collection as usual) the only thing left to do is to write the name of the NNTP server in the file /usr/pkg/etc/nntp/server, which you may need to create first. For example:

news.bignet.it

Once this has been done, the program can be started with the command tin. On the screen something similar to the following example will be displayed:

$ tin
Connecting to news.bignet.it...
news.bignet.it InterNetNews NNRP server INN 1.7.2 08-Dec-1997 ready (posting ok).
Reading groups from active file...
Checking for new groups...
Reading attributes file...
Reading newsgroups file...
Creating newsrc file...
Autosubscribing groups...
Reading newsrc file...

Be patient when you connect for the first time, because tin downloads an immense list of newsgroups to which you can subscribe and this takes several minutes. When the donwload is finished, the program's main screen is displayed; usually no groups are displayed; to see the list of groups press y. To subscribe to a group, move on the group's name and press y.

Once that you have subscribed to some newsgroups you can start tin more quickly with the command tin -Q. The search for new groups is disabled (-q), only active groups are searched (-n) and newsgroup description are not loaded (-d): it will not be possible to use the y (yank) command in tin. When tin is started with this option it can't tell if a newsgroup is moderated or not.

Note that if you are connecting from an internal network (like in our example), when you send ("post") a message the address at the beginning of the message will be wrong (because it is the internal address). To solve the problem, use the option “mail_address” in the tin configuration file (~/.tin/tinrc) or set the REPLYTO environment variable.