MTA Benchmark

I made a new MTA performance shootout to see how fast MTAs deliver in a LAN. The machines I used are shown below.

Benchmark Machine

This is the machine that did the actual MTA load testing. /var/log/maillog was disabled in syslog.conf.

Hardware

CPU:	AMD K6-2, 300 MHz
Cache:	1 MB, 100 MHz
RAM:	128 MB SDRAM, 100 MHz, CL 2
SCSI:	Tekram DC-390 (AMD Am53C974 "PCscsi II")
Drive:	Micropolis 4345WS (UWSCSI, 7200/min, 4.3 GB, Write Cache off)
Net:	3C900 Combo

Software

SMTP benchmark

QMTP benchmark

Setup

syslog did not log mail locally, but just across the network to the helper machine that writes its mail log asynchronously.

In the qmail test cases, I ran qmail-smtpd or qmail-qmqpd from tcpserver as described in /var/qmail/doc/FAQ, I merely adjusted user and group ids as necessary. Inetd would have deactivated qmail-smtpd after a while.

In the Postfix QMQP test case, I enabled QMQP as described in QMQP_README.

I ran pidentd with -w -t120 rather than inetd's internal identd to prevent it from being disabled by inetd.

I recompiled my kernel to allow for more simultaneous connections, FreeBSD would run out of buffer space otherwise with several hundred sockets in TIME_WAIT state.

All tests were on the same /var partition with softupdates. If listed as "no softupdates", I unmounted the partition, switched off softupdates and remounted it.

Helper machine

This box provided DNS, is the loghost, and sinked and discarded the incoming test mail. It ran Wietse Venema's smtp-sink that comes with Postfix with a listen backlog of 1,000.

Hardware

Software

Test

I ran smtp-source to inject mail addressed to the helper machine to localhost.

Results SMTP

Notes:

1. Exim only takes 10 recipients through a single connection for immediate delivery and queues the rest. The manual queue run took part after smtp-source was finished and used -qqf.
2. Qmail and Exim can only use noasync and sync file systems without softupdates as per qmail's FAQ: http://cr.yp.to/qmail/faq/reliability.html and as per a post to the linux kernel mailing list: .

Results QMQP

Observations and Commentary

• (not visible from the table) Postfix' nqmgr gives no advantage in this test, 20 cached connections injecting 1,000 mails are still delivered in 62 s.
• (not visible from the table) qmail's preprocessing is a real performance killer. The throughput ramps up considerably after all mail has been preprocessed. (I did a separate test I don't report above.)
  NEW: André Oppermann <oppermann@pipeline.ch> has graphs and a patch to back this observation with a concurrency profile and remedy this.
• qmail is only really safe with mount -o sync /var because it may lose bounce message contents over a crash otherwise. See INTERNALS that comes with the qmail-1.03.tar.gz tarball.
• exim cannot handle multiple simultaneous incoming connections well, it's much faster with 5 clients than with 20. Other MTAs don't exhibit this behaviour.
• The per-mail overhead is significant, multi-RCPT is always superiour performance-wise to single-RCPT.
• Sendmail benefits from connection caching by the injection clients, it seems its ident lookups are slow. Exim shows the strange "after 10 mails, go to queue-only mode for this connection" behaviour. Postfix and qmail hardly respond to client connection caching. tcpserver's ident lookups seem to be more efficient than sendmail's.
• QMQP has no visible, relevant performance benefit, it is yet another useless protocol that requires more additional code, SMTP suits just as well for mail injection at no additional cost. Postfix scales better than qmail as the number of reciepients per mail increases. A qmail-smtpc wrapper that behaves like Jef Poskanzer's mini_sendmail would have suited the job just fine, however, Postfix' QMQP daemon may ease the migration from qmail to Postfix.

History

2002-09-16: Came across André Oppermann's qmail-ldap and his and Claudio Jeker's EXTTODO patches and added links.

2001-11-29: Russell Nelson asked why only qmail was supposed to be unreliable on softupdates, so I checked again and figured Exim was also unreliable ("unsafe") on softupdates file systems. Tagged as "unsafe", so the figures shown for exim are probably better than those for production systems.

2001-09-01: I found an updated Postfix port for FreeBSD, and figured it didn't include qmqp. I fixed that, filed a problem report with my diff, and made a postfix-vs-qmail qmqpd shootout. I also fixed the W3C CSS logo to not use style-sheets but attributes for its size, this prevents Netscape 4.x from malformatting the page.

2001-08-31: I got a report from Claus Aßmann that my figures were low for softupdates and without connection caching, so I did the test again for sendmail with 20 threads injecting, this time with sendmail 8.11.6 on FreeBSD 4.4-RC. The figures are: softupdates: 117/99 s without/with connection caching. No softupdates: 135/136 s without/with connection caching.

What I cannot explain is that with softupdates, connection caching on the client side seems to help performance, while it makes no difference without softupdates.

Matthias Andree Last modified: Mon Sep 16 12:17:45 CEST 2001