3.15 Databases

Some mailfromd functions use DBM databases to save their persistent state data. Each database has a unique identifier, and is assigned several pieces of information for its maintenance: the database file name and the expiration period, i.e. the time after which a record is considered expired.

To obtain the list of available databases along with their preconfigured settings, run mailfromd --show-defaults (see Examining Defaults). You will see an output similar to this:

version:             9.0
script file:         /etc/mailfromd.mfl
preprocessor:        /usr/bin/m4 -s
user:                mail
statedir:            /var/run/mailfromd
socket:              unix:/var/run/mailfromd/mailfrom
pidfile:             /var/run/mailfromd/mailfromd.pid
default syslog:          blocking
supported databases:     gdbm, bdb
default database type:   bdb
optional features:   DKIM GeoIP2 STARTTLS
greylist database:      /var/run/mailfromd/greylist.db
greylist expiration:    86400
tbf database:        /var/run/mailfromd/tbf.db
tbf expiration:      86400
rate database:      /var/run/mailfromd/rates.db
rate expiration:    86400
cache database:      /var/run/mailfromd/mailfromd.db
cache positive expiration: 86400
cache negative expiration: 43200

The text below ‘optional features’ line describes the available built-in databases. Notice that the ‘cache’ database, in contrast to the rest of databases, has two expiration periods associated with it. This is explained in the next subsection.

3.15.1 Database Formats

The version 9.0 runs the following database types (or formats):

‘cache’

Cache database keeps the information about external emails, obtained using sender verification functions (see Checking Sender Address). The key entry to this database is an email address or email:sender-ip string, for addresses checked using strict verification. The data its stores for each key are:

1. Address validity. This field can be either success or not_found, meaning the address is confirmed to exists or it is not.
2. The time when the entry was entered into the database. It is used to check for expired entries.

The ‘cache’ database has two expiration periods: a positive expiration period, that is applied to entries with the first field set to success, and a negative expiration period, applied to entries marked as not_found.

‘rate’

The mail sending rate data, maintained by rate function (see Rate limiting functions). A record consists of the following fields:

timestamp

The time when the entry was entered into the database.

interval

Interval during which the rate was measured (seconds).

count

Number of mails sent during this interval.

‘tbf’

This database is maintained by tbf_rate function (see TBF). Each record represents a single bucket and consists of the following keys:

timestamp

Timestamp of most recent token, as a 64-bit unsigned integer (microseconds resolution).

expirytime

Estimated time when this bucket expires (seconds since epoch).

tokens

Number of tokens in the bucket (size_t).

‘greylist’

This database is maintained by greylist function (see Greylisting). Each record holds only the timestamp. Its semantics depends on the greylisting implementation in use (see greylisting types). In traditional implementation, it is the time when the entry was entered into the database. In Con Tassios implementation, it is the time when the greylisting period expires.

3.15.2 Basic Database Operations

The mfdbtool utility is provided for performing various operations on the mailfromd database.

To list the contents of a database, use --list option. When used without any arguments it will list the ‘cache’ database:

$ mfdbtool --list
abrakat@mail.com           success Thu Aug 24 15:28:58 2006
baccl@EDnet.NS.CA          not_found Fri Aug 25 10:04:18 2006
bhzxhnyl@chello.pl       not_found Fri Aug 25 10:11:57 2006
brqp@aaanet.ru:24.1.173.165  not_found Fri Aug 25 14:16:06 2006

You can also list data for any particular key or keys. To do so, give the keys as arguments to mfdbtool:

$ mfdbtool --list abrakat@mail.com brqp@aaanet.ru:24.1.173.165
abrakat@mail.com           success Thu Aug 24 15:28:58 2006
brqp@aaanet.ru:24.1.173.165  not_found Fri Aug 25 14:16:06 2006

To list another database, give its format identifier with the --format (-H) option. For example, to list the ‘rate’ database:

$ mfdbtool --list --format=rate
sam@mail.net-62.12.4.3 Wed Sep  6 19:41:42 2006  139   3 0.0216 6.82e-06
axw@rame.com-59.39.165.172 Wed Sep  6 20:26:24 2006  0  1  N/A  N/A

The --format option can be used with any database management option, described below.

Another useful operation you can do while listing ‘rate’ database is the prediction of estimated time of sending, i.e. the time when the user will be able to send mail if currently his mail sending rate has exceeded the limit. This is done using --predict option. The option takes an argument, specifying the mail sending rate limit, e.g. (the second line is split for readability):

$ mfdbtool --predict="180 per 1 minute"
ed@fae.net-21.10.1.2 Wed Sep 13 03:53:40 2006  0 1 N/A N/A; free to send
service@19.netlay.com-69.44.129.19 Wed Sep 13 15:46:07 2006 7 2
   0.286   0.0224; in 46 sec. on Wed Sep 13 15:49:00 2006

Notice, that there is no need to use --list --format=rate along with this option, although doing so is not an error.

To delete an entry from the database, use --delete option, for example: mfdbtool --delete abrakat@mail.com. You can give any number of keys to delete in the command line.

3.15.3 Database Maintenance

There are two principal operations of database management: expiration and compaction. Expiration consists in removing expired entries from the database. In fact, it is rarely needed, since the expired entries are removed in the process of normal mailfromd work. Nevertheless, a special option is provided in case an explicit expiration is needed (for example, before dumping the database to another format, to avoid transferring useless information).

The command line option --expire instructs mfdbtool to delete expired entries from the specified database. As usual, the database is specified using --format option. If it is not given explicitly, ‘cache’ is assumed.

While removing expired entries the space they occupied is marked as free, so it can be used by subsequent inserts. The database does not shrink after expiration is finished. To actually return the unused space to the file system you should compact your database.

This is done by running mfdbtool --compact (and, optionally, specifying the database to operate upon with --format option). Notice, that compacting a database needs roughly as much disk space on the partition where the database resides as is currently used by the database. Database compaction runs in three phases. First, the database is scanned and all non-expired records are stored in the memory. Secondly, a temporary database is created in the state directory and all the cached entries are flushed into it. This database is named after the PID of the running mfdbtool process. Finally, the temporary database is renamed to the source database.

Both --compact and --expire can be applied to all databases by combining them with --all. It is useful, for example, in crontab files. For example, I have the following monthly job in my crontab:

0 1 1 * * /usr/bin/mfdbtool --compact --all