|
After I read a claim in de.comm.software.mailserver that qmail's queue system was supposed to give better performance -- without mentioning what it was compared to -- I decided to make my own qmail-vs.-postfix benchmark. The claim has been withdrawn and apologied for, the benchmark however is still useful, though it's very particular.
The benchmark tested 1,000 one-to-one mailings in a LAN, and was supposed to shed some light on the local queue management of the tested MTAs.
After I sent a note about this benchmark to the postfix-users mailing list, I received some feedback, most of which resulted in added explanations on this page, however, I cannot currently address one topic: the CPU load. I simply did not pay attention to the CPU load when I ran the test, because I assumed the test as I/O bound anyhow, and the sheer write access figures back that.
This is the machine that did the actual MTA load testing.
| CPU: | AMD K6-2, 300 MHz |
|---|---|
| Cache: | 1 MB, 100 MHz |
| RAM: | 128 MB SDRAM, 100 MHz, CL 2 |
| SCSI: | Tekram DC-390 (AMD Am53C974 "PCscsi II") |
| Drive: | Micropolis 4345WS (UWSCSI, 7200/min, 4.3 GB, Write Cache off) |
| Net: | 3C900 Combo |
OS: | FreeBSD 4.3-STABLE port collection cvsupped 2001-08-22 |
|---|---|
| Contestant #1: | qmail-1.03_1 w/ tcpserver running qmail-smtpd as per /var/qmail/doc/FAQ |
| Contestant #2: | postfix-20010525_3 |
Either package was built out-of-the box from the FreeBSD ports package. No tuning took place. Both MTAs logged through syslog as per their default install. syslogd itself logged (both at the same time) to a file in /var/log and across a 10Base2 ethernet to my loghost.
In the qmail test cases, I ran qmail-smtpd from tcpserver as described in /var/qmail/doc/FAQ, I merely adjusted user and group ids as necessary, because inetd would have deactivated qmail-smtpd after a while.
I did not use the -H and -R options to tcpserver (which would have suppressed peer DNS and identd lookups), since these are not mentioned in qmail's FAQ. Since FreeBSD runs auth from inetd, it was deactivated after approx. 50 seconds because inetd thought auth was looping.
This box provided DNS, is the loghost, and sinked the incoming test mail to /dev/null. It is substantially faster, I/O-wise, and has less load to bear than the benchmark machine.
| CPU: | AMD Duron, 700 MHz |
|---|---|
| RAM: | 320 MB SDRAM, 100 MHz, CL 2 |
| IDE: | VIA KT133 South Bridge (UDMA 4 "UATA/66") |
| Drive: | Western Digital AC420400D (5400/min, ~ 20 GB, Write Cache off) |
| Net: | 3C900 Combo |
| OS: | hacked-up SuSE Linux 7.0, Linux Kernel 2.4.9 with ext3fs 0.9.6 |
|---|---|
| MTA: | Postfix Snapshot 2001-07-14 running on a /var/spool/postfix that's got its +S (sync) flags removed (safe for ext3fs) |
| DNS: | djbdns 1.05 providing all relevant DNS information to the other machine |
smtp-source -c -l 3000 -t emma+killme@emma1.emma.line.org \
-s 5 -m 1000 localhost
That is: send 1000 mails total, 3000 bytes payload in the body, with 5 simultaneously running processes to one address, injecting to localhost's SMTP.
Note again that syslogd was running and logging to the same /var partition that the MTA was using, which was a standard ffs "noasync" (i. e. synchronous meta data/directory updates) file system without softupdates (softdeps) unless stated otherwise. This is a common setup, mailers do log and logging affects performance.
grouped by softupdates state, then sorted in order of descending throughput
| software/figure | time to inject [s] | time to deliver [s] | sync writes [1] | async writes [1] | throughput [mail/s] | notes |
|---|---|---|---|---|---|---|
| postfix-20010525_3 softupdates enabled | 75 | 87 | 2,158 | 4,789 | 11.5 | * as of 2001-08-22 |
| UNSAFE! exim-3.33 softupdates enabled UNSAFE! | 112 | 113 | 5,214 | 4,350 | 8.8 | * a crash may cause mail loss * supplement 2001-08-27 |
| sendmail (8.11.3) softupdates enabled | 161 | 162 | 5,605 | 3,309 | 6.2 | * auth disabled after 86 s * supplement 2001-08-27 |
| UNSAFE! qmail-1.03_1 softupdates enabled UNSAFE! | 101 | 184 | 8,107 | 8,179 | 5.4 | * auth disabled after 25 s * a crash may cause mail loss * supplement 2001-08-24 * approximate figures (see below) |
| software/figure | time to inject [s] | time to deliver [s] | sync writes [1] | async writes [1] | throughput [mail/s] | notes |
| postfix-20010525_3 | 111 | 158 | 6,086 | 7,798 | 6.3 | * as of 2001-08-22 |
| qmail-1.03_1 | 153 | 313 | 14,138 | 16,696 | 3.2 | * auth disabled after 50 s * as of 2001-08-22 |
| software/figure | time to inject [s] | time to deliver [s] | sync writes [1] | async writes [1] | throughput [mail/s] | notes |
Note #1: When I ran the test, I started smtp-source, clocked, clocked again when smtp-source was done, and again when the last mail had been delivered. Both machines had their time synchronized with ntpd.
"Time to inject" is the time span from starting smtp-source until its exit, "time to deliver" the time span from starting smtp-source until the time stamp of the last log entry for the last mail delivered. Obviously, most of the time smtp-source was running, injection and delivery overlapped.
After that, I checked with mailq on the test machine to be sure the mail queue was empty.
Note #2: I don't currently have CPU usage figures (neither absolute nor relative), I will try to make a more comprehensive test along with other MTA software packages like Exim, Sendmail, ZMailer which will then include CPU figures. Sorry.
in no particular order
2001-11-29: add notes that exim does not support softupdates file systems either. Found by Russell Nelson.
2001-08-27: ran exim-3.33 port and sendmail of FreeBSD 4.3-RELEASE-p14 benchmarks.
An older version of this document claimed FreeBSD was using in.identd, which is wrong. FreeBSD has been using the inetd-internal auth service in default configuration. Fixed 2001-08-24.
2001-08-24: In spite of what Dan's document on qmail reliability and file systems says, I ran qmail on a softupdates file system to show it might benefit from softupdates support. Do not run qmail on softupdated /var filesystems on production machines, your queue will not be crashproof then! Note also that the figures will probably differ should a qmail appear that supports softupdates because qmail will probably have to be changed.
| Matthias Andree Last modified: Tue May 21 03:55:33 CEST 2002 | |