Validating Mail connectivity, performance, configuration
[Ref: OpenBSD 5.5, Postfix 2.11.0 (from ports)]
Table of Content
- Remote Connectivity * smtp * pop3 * imap
- SSL/TLS Transport * smtp with starttls * smtps
- Powershell
- Behaviour * Performance * Publicly Accessible Services * Open Relay
- Queues
As a conglomeration of various services, there are a number of separate tests that can be carried out to confirm, validate a systems activity, performance.
General Remote/Local connections are through the standard protocols ( smtp, pop3, and imap ) whereas service behaviour have some other attributes that can also be monitored.
The following are a number of utilities, methodologies for validating your server’s behaviour, performance during development and after deployment.
Remote Connectivity
SMTP - Simple Message Transfer Protocol
telnet
Telnet is a useful tool for connectivy testing with servers that respond with Text responses and support protocols that can be entered at the command-line. Services that generally support this include smtp, imap, and pop3.
Note: RFC 2282 via may require more fields than specified in these instructions (such as FROM: and DATE: inside the DATA section)
$ telnet localhost smtp
Trying ::1... Connected to localhost. Escape character is '^]'. 220 myhost.example.org ESMTP Postfix
EHLO example.org
250-myhost.example.org 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN
MAIL FROM: <samt@example.org>
250 2.1.0 Ok
RCPT TO: <samt@example.org>
250 2.1.5 Ok
DATA
354 Enter mail, end with '.' on a line by itself
From: <samt@example.org>
Subject: This is my subject line
I continue writing until I'm out of interesting things to say
which is not that far away
.
250 2.0.0 Ok: queued as 699ACBA2D7
QUIT
221 2.0.0 Bye Connection closed by foreign host.
I’ve just used capital letters for the SMTP commands, but obviously they work fine with lowercase.
If your server is not yet online with a valid DNS record, then you can test using RCPT TO: samt@localhost.
Log Entries
[ tail ]
Confirm where your MTA logs it’s messages, but in general mail entries are logged in the /var/log/maillog file, and we can monitor general activities through this file.
Screen Session
# tail -f /var/log/maillog
starting the Postfix mail system daemon started -- version 2.3.2, configuration /etc/postfix connect from localhost[::1] 5E4A5BA2D4: client=localhost[::1] 5E4A5BA2D4: message-id=<20061212080251.5E4A5BA2D4@hostname.example.org> 5E4A5BA2D4: from=<samt@example.org>, size=457, nrcpt=1 (queue active) 5E4A5BA2D4: to=<samt@example.org>, relay=local, delay=77, delays=77/0.05/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox) 5E4A5BA2D4: removed disconnect from localhost[::1]
From the manpage: ‘tail(1)’
The tail utility displays the contents of file or, by default, its
standard input, to the standard output.
-f Do not stop when end-of-file is reached; instead, wait for
additional data to be appended to the input. If the file is
replaced (i.e., the inode number changes), tail will reopen the
file and continue. If the file is truncated, tail will reset its
position to the beginning. This makes tail more useful for
watching log files that may get rotated. The -f option is
ignored if the standard input is a pipe, but not if it is a FIFO.
Using the ‘-f’ parameter tells ’tail’ to continue looking at the recent additions to the file (such that updates to the file are displayed on the screen for us.) Use Ctrl+C (i.e. hold the Ctrl key while pressing C) to break out of the log review session shown above
[Ref: mail(1) ]
While we’re testing with real system user accounts, we can use the unix ‘mail’ program to check our mail message.
Screen Session
# /usr/bin/mail -u samt
Mail version 8.1.2 01/15/2001. Type ? for help. '/var/mail/samt': 1 message 1 new >N 1 samt@example.org Tue Dec 12 21:03 18/605 This is my subject line
& more 1
Message 1: From samt@example.org Tue Dec 12 21:03:54 2006 X-Original-To: samt@myhost.example.org Delivered-To: samt@myhost.example.org Subject: This is my subject line From: samt@example.org To: undisclosed-recipients:; I continue writing until I'm out of interesting things to say which is not that far away
& q
Saved 1 message in mbox
In the above example, we enter mail for the user samt (’-u samt’) and the ‘mail’ client shows a list of current email for user ‘samt’ and then gives us the ‘&’ ampersand prompt.
We can read the email message by typing the message number, and ‘mail’ supports the use of a screen ‘pager’ such as ‘more’ so that we can scroll through longer messages.
From the manpage: mail(1)
Reading mail
In normal usage, mail is given no arguments and checks your mail out of
the post office, then prints out a one line header of each message found.
The current message is initially set to the first message (numbered 1)
and can be printed using the print command (which can be abbreviated p).
Moving among the messages is much like moving between lines in ed(1); you
may use + and - to shift forwards and backwards, or simply enter a
message number to move directly.
Quit. We quit out of ‘mail’ using the ‘q’ command.
The above reference to the log files and mail client is to provide you with more tools for validating your installation.
Mail Receipt
[Ref: telnet, mail, /var/log/maillog ]
Verify that the mail server will accept tcp connections on smtp from the below by sending mail to a user account on the mail host:
- localhost
- relay network
- external network
With the verification, ensure that email is recieved by watching the maillog to note:
- mail is accepted (queued) by the mail server
- mail is delivered (sent) by the mail server (in this case, delivery is to a local mail box)
An internal client (destination email) should have successfully recieved mail from the above tests.
Mail Delivery
[ telnet, mail, /var/log/maillog ]
Verify the mail server will accept tcp connections on smtp and can send mail ’to the world’ by sending messages from:
- localhost
- LAN
Ensure that email can be sent through and to a destination by watching the maillog to note:
- mail is accepted from the mail server host (eg. telnet from host to remote smtp server)
- mail is queued by the mail server
- mail is delivered by the mail server
An external client (destination email) should have successfully recieved mail from the above tests.
Summary
We now have a fully functional SMTP server that can receive email messages, and store those messages for users.
POP3 - Post Office Protocol
[Ref: The Network People, Inc. Mail Server Testing ]
If you’ve successfully installed dovecot with mysql above, and have gone through the Configuring a Virtual Email Service - MySQL in our postfix installation guide, (or you have installed your own MySQL virtual user accounts) then we can perform some testing, validating whether our configuration actually works.
Screen Session
$ telnet localhost pop3
Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. +OK Dovecot ready.
user charlie@alpha.example.org
+OK
pass charlie
+OK Logged in.
list
+OK 3 messages: 1 503 2 445 3 503 .
retr 3
+OK 503 octets Return-Path: <samt@example.org> X-Original-To: charlie@alpha.example.org Delivered-To: charlie@alpha.example.org Received: from example.org (unknown [IPv6:::1]) by myhost.example.org (Postfix) with ESMTP id 9A6165A950; Fri, 9 Feb 2007 13:50:26 +1300 (TOT) Subject: Welcome MySQL based virtual users Message-Id: <20070209005037.9A6165A950@myhost.example.org> Date: Fri, 9 Feb 2007 13:50:26 +1300 (TOT) From: samt@example.org To: undisclosed-recipients:; Hopefully you've received this email message without fault ? .
QUIT
+OK Logging out. Connection closed by foreign host.
The maillog file should show success similar to the below
File Fragment: /var/log/maillog
pop3-login: Login: user=<charlie@alpha.example.org>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured POP3(charlie@alpha.example.org): Disconnected: Logged out top=0/0, retr=1/519, del=0/3, size=1451
Again, a review of the mysql transaction log can be helpful in diagnosing errors.
File Fragment: /var/mysql/myhost.log
Connect dovecot@localhost on mail Query SELECT password FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1' Query SELECT maildir, 901 AS uid, 901 AS gid FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'
Simple Errors -ERR Authentication failed.
You get an Authentication failed even though you know and swear that you have entered the correct password?
- Check the /var/mysql/myhost.log file to ensure that the correct query is sent by dovecot to the MySQL Server (i.e. SELECT password FROM mailbox WHERE username = ‘VIRTUALACCOUNT@VIRTUALDOMAIN’ AND active = ‘1’)
- Check that your dovecot configuration is using the same encryption method for creating/reading passwords, as postfixadmin. For example, in our exercise we are using CRYPT: default_pass_scheme = CRYPT.
IMAP - Internet Message Access Protocol
We use telnet on the localhost to test imap’s configuration
Screen Session
$ telnet localhost imap
Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK Dovecot ready.
a1 login charlie@alpha.example.org charlie
a1 OK Logged in.
a2 select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 3 EXISTS * 0 RECENT * OK [UNSEEN 1] First unseen. * OK [UIDVALIDITY 1170991431] UIDs valid * OK [UIDNEXT 4] Predicted next UID a2 OK [READ-WRITE] Select completed.
a3 fetch 3 body[text]
* 3 FETCH (BODY[TEXT] {66}
Hopefully you've received this email message without fault ?
)
a3 OK Fetch completed.
a4 close
a4 OK Close completed.
a5 logout
* BYE Logging out a5 OK Logout completed. Connection closed by foreign host.
Note:
a1, a2, .., a5 are randomly selected unique leaders (in this case we’re just making things sequential)
"a3 fetch 3 body[text]", the number ‘3’ refers to the ‘3_ EXISTS_’ in the list returned by ‘a2 select inbox’
Your maillog file is your friend and will give you clues to where you can check for other errors.
File Fragment: /var/log/maillog
auth-worker(default): mysql: Connected to localhost (mail) imap-login: Login: user=<charlie@alpha.example.org>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured IMAP(charlie@alpha.example.org): Disconnected: Logged out
Likewise the mysql transaction log should give further assistance should the installation be having problems.
File Fragment: /var/mysql/myhost.log
Connect dovecot@localhost on mail Query SELECT password FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1' Query SELECT maildir, 901 AS uid, 901 AS gid FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'
SSL / TLS Transport
[Ref: openssl s_client]
The s_client command implements a generic SSL/TLS client which connects to a remote host using SSL/TLS. It is a very useful diagnostic tool for SSL servers.
For all the above protocols (smtp, pop3, imap) we can use openssl s_client to connect to the remote host, and use the standard diagnostics commands, as if it were a cleartext connection.
SMTP is provided into two flavours, a cleartext connection which can then be negotiated into an encrypted connection, and the 2nd with encrypted connection to begin with.
SMTP using STARTTLS
openssl s_client provides a command-line mechanism for initiating a connection, and starting STARTTLS to encrypt connection link.
openssl s_client -starttls smtp -crlf -connect localhost:25
A lot of certificate information is exchanged, and shown on the screen ... 250 DSN
MAIL FROM: <samt@example.com>
250 2.1.0 Ok
RCPT TO: mylocaluser
250 2.1.5 Ok
DATA
354 End data with.
From: <samt@example.com> Subject: STARTTLS Test Message Postfix will decrypt .
250 2.0.0 Ok: queued as XXXXXXXXXXX
quit
221 2.0.0 Bye
openssl’s s_client -starttls smtp tells it to make the encryption/decrypting between our console and an SMTP server. Once connected, then we can go through our normal smtp test message commands. (Shown in CAPITAL letters above.)
SMTPS
Verify the wrapmode smtps service is working correctly using the generic openssl s_client connection.
openssl s_client -connect localhost:465
CONNECTED(0000000X) Plenty of Certificate negotiation/information 220 mx.coco.nut.to ESMTP Postfix
At this point, we have an example of connecting SSL secured to SMTPS. The same approach is made for connecting to other SSL services:
openssl s_client -connect localhost:${PORT}
Powershell
[Ref: Send-MailMessage
Behaviour
Performance
[ ports/benchmarks/smtp-benchmark ]
smtp-benchmark is a client/server program used to push as much data through the pipe to visit the mail server and review it’s capacity to handle load.
smtp-benchmark consists of two programs, smtpsend and smtpsink. smtpsend is used to send generated e-mail messages using SMTP to a mail transfer agent, smtpsink is designed to dispose of received messages as quick as possible.
- Server
- Client
smtp-benchmark: smtpsink
The smtpsink Server runs on the destination side of the mail server, expecting to recieve mail FROM the mail server. It basically accepts the message delivery from the mail server and dumps it to /dev/null
smtpsink -v -p 25
smtp-benchmark: smtpsend
The smtpsend Client measures the time spent sending e-mails and the number of e-mails actually sent and outputs statistics after the program run.
smtp-send can fork one or more parallel senders each using one or more sequential connections to a SMTP server to deliver one or more messages per connection.
smtp-sink comes in handy when the relaying performance of a MTA is to be measured.
smtpsend -s 1 -m 100 -b 512 -t 120 -p 25 -F samt@example.com \ -T user@valid-domain.com -S "Test Message" -vv SMTP_SERVER
| s | Maximum number of parallel senders |
| m | number of messages to send per connection |
| t | Total time to run (exclusive of -n) |
| n | send at least 'n' messages. |
Now, if you’ve got my patches then you can do something similar to the below, which is required in my environment due to some of the restrictions (configurations) we have in place.
smtpsend -s 1 -m 1 -b 126 -t 1 -p 25 -H gmail.com -F myname@gmail.com -T myname@example.com \
-S "Server Connectivity Test to MAILSERVER" -vv MAILSERVEREnhancements?
| -H | The domain used in the initiating HELO line (e.g. gmail.com) |
| -B | Do we want a different 'body' message instead of the generic one? |
Publicly Accessible Services
[ ports/net/nmap ]
Verify with nmap that only the expected services are running on this host.
Open Relay
Since a configuration failing to allow “Open Relay” is just not excusable.
Verify with telnet that open relay is not supported, other than those specifically included in the server configuration.
An attempt to relay should be rejected by the mail server.
tcpdump -nettti interface
Our standard configurations will require mail to traverse at least two firewalls, use tcpdump at appropriate locations to verify that the smtp traffic is tranversing the route between the client and our server.
- NIC - verify traffic reaches the NIC and has a route back to the client.
- Firewall External NIC - verify traffic reaches the NIC and is not blocked
- Firewall Internal NIC - verify traffic goes in/out the Internal link and indicates return route from the client.
Queue
[Ref: Postfix Bottleneck Analysis]
When Postfix accepts an email message for processing (delivery) it places these messages into a queue. Finding out, knowing what’s in the queue, and what’s happening to stuff in there is always invaluable to diagnosing performance issues with our mail server.
Postfix places messages into the queue(s) (default: maildrop, hold, incoming, active and deferred).
New messages are put into the ‘incoming’ queue for processing by the Queue Manager (qmgr)
We want to effect the Mail Queue (email messages that are in our system and yet to be delivered) for two things:
- Viewing Queue Status/Content
- Processing Queue Content
postsuper and postqueue are the general tools but there are a host of other tools with specific value.
Viewing Status/Content
To view the status of the mail queue, three tools are generally used.
- mailq
- postqueue -p
- qshape
mailq/postqueue -p
mailq is implemented with postqueue -p and is the one most commonly used as a quick look at what’s happening (especially since mailq is a similar utility for Sendmail.)
Both tools provide a summary status of each message in the queue, with at least the following:
- queue-id
- size
- arrival-time
- sender/recipient
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
67A381D576E* 75102 Fri Jan 24 12:17:32 yum@spaceinvaders.com.au
a.fitzpatrick@example.com
-- 74 Kbytes in 1 Request.
qshape
qshape provides a summary of volume and time in queue, and I generally use it through the below style of executation
$ gnuwatch -n 2 sudo /usr/local/sbin/qshape
Every 2s: sudo /usr/local/sbin/qshape Fri Jan 24 13:11:37 2014
T 5 10 20 40 80 160 320 640 1280 1280+
TOTAL 1319 47 62 158 222 529 301 0 0 0 0
breakfast.com.au 1036 34 50 106 167 432 247 0 0 0 0
challenges.com 92 3 1 38 14 19 17 0 0 0 0
needsome.com.au 48 1 6 0 12 23 6 0 0 0 0
extracare.com.au 36 3 2 1 7 19 4 0 0 0 0
especially.co.nz 27 1 1 5 5 8 7 0 0 0 0
when.com.au 19 1 0 2 3 5 8 0 0 0 0
outwalking.com.au 17 1 1 2 7 4 2 0 0 0 0
without.com.au 12 0 0 0 2 6 4 0 0 0 0
adequate.com.au 9 3 0 2 0 3 1 0 0 0 0
recompense.com.au 8 0 0 0 1 6 1 0 0 0 0
considering.co.uk 7 0 0 1 1 2 3 0 0 0 0
the.to 3 0 0 1 1 1 0 0 0 0 0
weather.com 3 0 1 0 1 1 0 0 0 0 0
implications.com 2 0 0 0 1 0 1 0 0 0 0
Process Queue Content
The Queue-ID is powerful, and let’s you perform queries/actions on the message such as:
- postcat -q queue-id to view the message and envelope Ref: example
- postsuper -d queue-id to delete
- postsuper -h queue-id to put it on ‘hold’