Next: , Previous: , Up: Top   [Contents][Index]

Appendix C Upgrading

The following sections describe procedures for upgrading between the consecutive Mailfromd releases. The absence of a section for a pair of versions x-y numbers means that no specific actions are required for upgrading from x to y.


Next: , Up: Upgrading   [Contents][Index]

C.1 Upgrading from 8.17 to 9.00

Old ‘.mf’ suffix gone

The ‘.mf’ suffix, that has long been used for MFL files and has been deprecated since version 8.15 (see 8140-8150) is no longer supported. MFL files have suffix ‘.mfl’.

Begin and end handlers must be defined using prog

The old syntax was:

begin
do
  ...
end

It is no longer supported. The correct way to declare begin and end handlers is

prog begin
do
  ...
end

Module and include search paths

Since version 8.15 if a module was not found in module search path, the search was retried using include search path. This is no longer the case: the two paths serve different purposes and don’t interact in any way. MFL modules are searched in module search path only.

IPv6

Version 9.0 features full IPv6 support.

The following modules used in previous releases are now deprecated. They are retained for backward compatibility. Any existing code that uses any of these will compile and work as in previous releases, except that a warning will be printed to draw your attention to the fact. You are advised to remove uses of these modules, as they will be removed in future versions:

match_cidr

The function match_cidr is reimplemented as a built-in function. Remove any instances of require 'match_cidr' or any equivalent import statements from your code.

See match_cidr.

is_ip

This module defines the is_ip function:

Library Function: boolean is_ip (string str)

Returns 1 if str is a string representation of an IPv4 address and 0 otherwise.

This function has been obsoleted by is_ip4str function. You are advised to use it, or a more general is_ipstr, function instead. See Internet address manipulation functions, for a detailed discussion of these functions,

revip

This module defines the following function:

Library Function: string revip (string ip)

Reverses octets in ip, which must be a valid string representation of an IPv4 address.

This function has been obsoleted by reverse_ipstr, discussed in Internet address manipulation functions.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.2 Upgrading from 8.16 to 8.17

No special actions are required after upgrading. However, one point is worth noticing. Until now, it was an error to specify two prog definitions with the same handler name. This rule however, applied only to generic milter handlers, special handlers (begin, end, startup, shutdown) being exempt from it. Starting with version 8.17, multiple prog declarations with the same handler name are allowed for all handlers. See Multiple handler definitions, for details.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.3 Upgrading from 8.14 to 8.15

This release introduces certain changes that require the installer’s attention.

MFL file suffix changed

MFL source files now have the suffix ‘.mfl’ instead of ‘.mf’ which was used in previous versions. This is to avoid confusion with Metafont files, which also have suffix ‘.mf’.

All installed MFL module files have been renamed.

As of version 8.15, this change is of advisory nature, and the legacy ‘.mf’ suffix is still supported. Thus, for example if module ‘X’ was required and the file ‘X.mfl’ was not found, mailfromd will look for file ‘X.mf’. Nevertheless, module authors are urged to rename their module files accordingly.

In particular, the default script file is now named mailfromd.mfl. If it does not exist in the system configuration directory, then the file mailfromd.mf is looked up. Installers are advised to rename mailfromd.mf to mailfromd.mfl.

MFL module search path

Until now, mailfromd searched for module files in include search path, which was mostly due to historical reasons. Version 8.15 introduces a separate module search path for this purpose. See module search path, for details.

All modules shipped with mailfromd are installed to the new location: prefix/share/mailfromd/8.15.

For backward compatibility, if unable to find module file in the module search path, mailfromd retries the search using include search path, so that existing installation will still work. A warning message is issued for each module file found in include search path, to help you locate files that need moving to the new place.

This additional search phase will remain active during a transitional period of couple of new releases to help users accommodate to the change.

Special handlers

Special handlers (see begin/end) are defined using the same syntax as standard milter stage handlers, i.e.:

prog begin
do
  …
done

Old syntax (without the prog keyword) is still supported, but causes a deprecation warning. Module writers are advised to update their sources accordingly.

Two new special handlers are provided. See startup/shutdown, for details.

Deprecated features removed

Deprecated configuration statements lock-retry-count and lock-retry-timeout were removed in this version. Please use the locking statement instead. See Mailutils Configuration File in GNU Mailutils Manual, for a detailed decription. To make the long story short:

Deprecated keywordUse this instead
lock-retry-countretry-count
lock-retry-timeoutretry-sleep

So that, instead of

lock-retry-count 10;
lock-retry-timeout 1;

you would write:

locking {
  retry-count 10;
  retry-sleep 1;
}

Next: , Previous: , Up: Upgrading   [Contents][Index]

C.4 Upgrading from 8.13 to 8.14

No special actions are required. There are several important changes in version 8.14, most notable ones being improved dkim_sign function, DNS resolver tolerance for CNAME chains, and buffered I/O.

dkim_sign

This change is important for those who use dkim_sign with Sendmail. See dkim_sign and sendmail, for a detailed discussion.

CNAME chains

Mailfromd dropped support for CNAME chains in version 8.3 (see 820-830). This change was forced by the behavior of the adns resolver library, which had apparently been grounded on somewhat rigorous reading of RFCs 103432 and 218133. Reportedly, many existing DNS servers still employ CNAME chains leading to TXT records (in particular, to DKIM TXT records), so this change made certain DKIM signatures unverifiable for mailfromd.

To fix this, starting from version 8.14 mailfromd implements a work-around that allows it to follow CNAME chains of limited length. The default length is 2 (which means CNAME pointing to a CNAME, pointing to a valid RR). It can be changed using the dns.max-cname-chain configuration statement (see conf-resolver).

I/O buffering

Buffered I/O can tremendously improve performance of getdelim and getline.

The global variables io_buffering and io_buffer_size (see io_buffering) define buffering mode and associated buffer size for file descriptors returned by the subsequent calls to open or spawn. Buffering mode of an already open file descriptor can be changed using the setbuf function.

The io_buffering variable defines the buffering mode. By default it is 0 (BUFFER_NONE), which disables buffering for backward compatibility with the previous versions. Another possible values are: 1 (BUFFER_FULL) and 2 (BUFFER_LINE).

When set to BUFFER_FULL, all I/O operations become fully buffered. The buffer size is defined by the io_buffer_size global variable.

BUFFER_LINE is similar to BUFFER_FILE when used for input. When used for the output, the data are accumulated in buffer and actually sent to the underlying transport stream when the newline character is seen. The io_buffer_size global variable sets the initial value for the buffer size in this mode. The actual size can grow as needed during the I/O.

The default value for io_buffer_size is the size of the system page.

The symbolic constants BUFFER_NONE, BUFFER_FULL and BUFFER_LINE are defined in the status.mf module. E.g.:

  require status

  begin
  do
    io_buffering = BUFFER_FULL
  done

will set up all the subsequent buffer-aware I/O functions for buffered reads and writes using the maximum buffer capacity.

The setbuf function (see setbuf) changes the buffering mode and/or buffer size for an already opened stream, e.g.:

  setbuf(fd, BUFFER_FULL, 4096)

For detailed discussion of various buffering modes and their effect on the I/O, see I/O functions.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.5 Upgrading from 8.7 to 8.8

DKIM support (see DKIM) introduced in this version requires the Nettle cryptographic library34. It you need DKIM, make sure Nettle is installed prior to compiling mailfromd. Otherwise, no special actions are required.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.6 Upgrading from 8.5 to 8.6

New configure option --with-dbm allows you to select any DBM flavor supported by GNU mailutils as the default DBM implementation for mailfromd.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.7 Upgrading from 8.2 to 8.3 (or 8.4)

Versions 8.3 and 8.4 differ only in required minimal version of mailutils (3.3 and 3.4, correspondingly). Apart from that, the following instructions apply to both versions.

In version 8.3 I abandoned the legacy DNS resolver and switched to GNU adns. GNU ands is a standalone resolver library, which provides a number of advanced features. It is included in most distributions. The source code of the recent release is available from http://www.chiark.greenend.org.uk/~ian/adns/adns.tar.gz.

This change brought a number of user-visible changes. In particular, arbitrary limits on the sizes of the RRs are removed. Consequently, the following configurations statements are withdrawn:

runtime.max-dns-reply-a
runtime.max-dns-reply-ptr
runtime.max-dns-reply-mx
max-match-mx
max-callout-mx

Secondly, the new resolver is less tolerant to deviations from the standard. This means that remote DNS misconfigurations that would have slipped unnoticed in previous versions, will be noticed by mailfromd 8.3. For example, a CNAME record pointing to another CNAME is treated as an error.

A new command line option was added to mailfromd and calloutd: --resolv-conf-file. This option instructs the programs to read resolver settings from the supplied file, instead of the default /etc/resolv.conf.

Another user-visible change is in handling of SPF checks. Previously, results if SPF checks were cached in a database. This proved to cause more problems than solutions and was removed in this version. As a result, the following MFL global variables have been withdrawn:

spf_ttl
spf_cached
spf_database
spf_negative_ttl

Next: , Previous: , Up: Upgrading   [Contents][Index]

C.8 Upgrading from 7.0 to 8.0

Version 8.0 is a major rewrite, that introduces a lot of new concepts and features. Nevertheless, it is still able to run the MFL scripts from version 7.0 without modifications.

Note the following important points:


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.9 Upgrading from 6.0 to 7.0

The release 7.0 removes the features which were declared as obsolete in 6.0 and introduces important new features, both syntactical, at the MFL level, and operational.

Unless your filter used any deprecated features, it should work correctly after upgrade to this version. It will, however, issue warning messages regarding the deprecated features (e.g. the use of ‘%’ in front of identifiers, as described below). To fix these, follow the upgrade procedure described in upgrade procedure.

The removed features are:

The MFL syntax has changed: it is no longer necessary to use ‘%’ in front of a variable to get its value. To reference a variable, simply use its name, e.g.:

set x var + z

The old syntax is still supported, so the following statement will also work:

set x %var + %z

It will, however, generate a warning message.

Of course, the use of ‘%’ to reference variables within a string literal remains mandatory.

Another important changes to MFL are user-defined exceptions (see User-defined Exceptions) and the try–catch construct (see try–catch).

Several existing MFL functions have been improved. In particular, it is worth noticing that the open function, when opening a pipe to or from a command, provides a way to control where the command’s standard error would go (see standard error).

The accept function (or action) issues a warning if its use would cancel any modifications to the message applied by, e.g., header_add and similar functions. See Message modification queue, for a detailed discussion of this feature.

The most important change in mailfromd operation is that the version 7.0 is able to run several servers (see conf-server). Dedicated callout servers make it possible to run sender verifications in background, using a set of long timeouts, as prescribed by RFC 2822 (see SMTP Timeouts). This diminishes the number of false positives, allows for coping with servers showing large delays and also reduces the number of callouts performed for such servers.

This release no longer includes the smap utility. It was moved into a self-standing project, which by now provides much more functionality and is way more flexible than this utility was. If you are interested in using smap, visit http://www.gnu.org.ua/software/smap, for a detailed information, including pointers to file downloads.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.10 Upgrading from 5.x to 6.0

The 6.0 release is aimed to fix several logical inconsistencies that affected the previous versions. The most important one is that until version 5.2, the filter script file contained both the actual filter script, and the run-time configuration for mailfromd (in form of ‘#pragma option’ and ‘#pragma database’ statements). The new version separates run-time configuration from the filter script by introducing a special configuration file mailfromd.conf (see Mailfromd Configuration).

Consequently, the ‘#pragma option’ and ‘#pragma database’ statements become deprecated. Furthermore, the following deprecated pragmas are removed: ‘#pragma option ehlo’, ‘#pragma option mailfrom’. These pragmas became deprecated in version 4.0 (see 31x-400).

The second problem was that the default filter script file had ‘.rc’ suffix, which usually marks a configuration file, not the source. In version 6.0 the script file is renamed to mailfromd.mf. In the absence of this file, the legacy file mailfromd.rc is recognized and parsed. This ensures backward compatibility.

This release also fixes various inconsistencies and dubious features in the MFL language.

The support for unquoted literals is discontinued. This feature was marked as deprecated in version 3.0.

The following features are deprecated: ‘#pragma option’ (pragma-option and ‘#pragma database’ (pragma-database) directives, the legacy style of function declarations (old-style function declarations), calls to functions of one argument without parentheses (operational notation), the ‘#require’ statement (See import, for the new syntax) and implicit concatenation (implicit concatenation). See Deprecated Features, for more information about these.

This release also introduces important new features, which are summarized in the table below:

FeatureReference
ConfigurationSee Mailfromd Configuration.
Module systemSee Modules.
Explicit type castsSee explicit type casts.
Concatenation operatorSee Concatenation.
Scope of visibilitySee scope of visibility.
Precious variablesSee rset.

Mailfromd version ‘6.0’ will work with unchanged scripts from ‘5.x’. When started, it will verbosely warn you about any deprecated constructs that are used in your filter sources and will create a script for upgrading them.

To upgrade your filter scripts, follow the steps below:

  1. Run ‘mailfromd --lint’. You will see a list of warnings similar to this:
    mailfromd: Warning: using legacy script file
     /usr/local/etc/mailfromd.rc
    mailfromd: Warning: rename it to /usr/local/etc/mailfromd.mf
     or use script-file statement in /usr/local/etc/mailfromd.conf
     to disable this warning
    mailfromd: /usr/local/etc/mailfromd.rc:19: warning: this pragma is
     deprecated: use relayed-domain-file configuration statement instead
    mailfromd: /usr/local/etc/mailfromd.rc:23: warning: this pragma is
     deprecated: use io-timeout configuration statement instead
    mailfromd: Info: run script /tmp/mailfromd-newconf.sh
     to fix the above warnings
    … 
    
  2. At the end of the run mailfromd will create a shell script /tmp/mailfromd-newconf.sh for fixing these warnings. Run it:
    $ sh /tmp/mailfromd-newconf.sh
    
  3. When the script finishes, run mailfromd --lint again. If it shows no more deprecation warnings, the conversion went correctly. Now you can remove the upgrade script:
    $ rm /tmp/mailfromd-newconf.sh
    

Notice, that the conversion script attempts to fix only deprecation warnings. It will not try to correct any other type of warnings or errors. For example, you may get warning messages similar to:

mailfromd: /etc/mailfromd.mf:7: warning: including a module file is unreliable and may cause subtle errors
mailfromd: /etc/mailfromd.mf:7: warning: use `require dns' instead

This means that you use ‘#include’ where you should have used ‘require’. You will have to fix such warnings manually, as suggested in the warning message.

If, for some reason, you cannot upgrade your scripts right now, you may suppress deprecation warnings by setting the environment variable MAILFROMD_DEPRECATION to ‘no’ before starting mailfromd. Nonetheless, I recommend to upgrade as soon as possible, because the deprecated features will be removed in version ‘6.1’.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.11 Upgrading from 5.0 to 5.1

Upgrading from 5.0 to 5.1 does not require any changes in your filter scripts. Notice, however, the following important points:


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.12 Upgrading from 4.4 to 5.0

This version of Mailfromd requires GNU mailutils version 2.0 or later.

Upgrading from version 4.4 to 5.0 requires no additional changes. The major differences between these two versions are summarized below:

  1. Support for ‘MeTA1’.
  2. New ‘Mailutils’ configuration file.
  3. New MFL functions.
    1. Message functions. See Message functions.
    2. Mailbox functions. See Mailbox functions.
    3. Mail body functions. See Mail body functions.
    4. Header modification functions. See Header modification functions.
    5. Envelope modification functions. See Envelope modification functions.
    6. Quarantine functions. See Quarantine functions.
    7. getopt and varptr. See getopt.
    8. Macro access functions. See Macro access.
    9. Character type functions. See Character Type.
    10. New string functions (see String manipulation): verp_extract_user, sa_format_report_header, sa_format_score.
    11. Sequential access to DBM files. See dbm-seq.
  4. Changes in MFL
    1. See variadic functions.
    2. See function alias.
  5. New operation mode: See Run Mode.
  6. Improved stack growth technique.

    The stack can be grown either by fixed size blocks, or exponentially. Upper limit can be specified. See stacksize.

  7. Milter ports can be specified using URL notation.
  8. Removed deprecated features.

    Support for some deprecated features has been withdrawn. These are:

    1. Command line options --ehlo, --postmaster-email, and --mailfrom. These became deprecated in version 4.0. See 31x-400.

Next: , Previous: , Up: Upgrading   [Contents][Index]

C.13 Upgrading from 4.3.x to 4.4

The deprecated --domain command line option has been withdrawn. The short option -D now defines a preprocessor symbol (see Preprocessor Options).

This version correctly handles name clashes between constants and variables, which remained unnoticed in previous releases. See variable--constant shadowing, for a detailed description of it.

To minimize chances of name clashes, all symbolic exception codes has been renamed by prefixing them with the ‘e_’, thus, e.g. divzero became e_divzero, etc. The ioerr exception code is renamed to e_io. See status.mf, for a full list of the new exception codes.

For consistency, the following most often used codes are available without the ‘e_’ prefix: success, not_found, failure, temp_failure. This makes most existing user scripts suitable for use with version 4.4 without any modification. If your script refers to any exception codes other than these four, you can still use it by defining a preprocessor symbol OLD_EXCEPTION_CODES, for example:

$ mailfromd -DOLD_EXCEPTION_CODES

Next: , Previous: , Up: Upgrading   [Contents][Index]

C.14 Upgrading from 4.2 to 4.3.x

Upgrading from 4.2 to 4.3 or 4.3.1 does not require any changes to your configuration and scripts. The only notable change in these versions is the following:

The asynchronous syslog implementation was reported to malfunction on some systems (notably on Solaris), so this release does not enable it by default. The previous meaning of the --enable-syslog-async configuration option is also restored. Use this option in order to enable asynchronous syslog feature. To set default syslog implementation, use DEFAULT_SYSLOG_ASYNC configuration variable (see syslog-async).

The following deprecated features are removed:

  1. #pragma option ehlo statement.

    It became deprecated in version 4.0. See pragma-option-ehlo.

  2. #pragma option mailfrom statement.

    It became deprecated in version 4.0. See pragma-option-ehlo.

  3. The --config-file command line option.

    It became deprecated in version 3.1. See 30x-31x.

  4. Built-in exception codes in catch statements.

    They are deprecated since version 4.0. See 31x-400.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.15 Upgrading from 4.1 to 4.2

Upgrading to this version does not require any special efforts. You can use your configuration files and filter scripts without any changes. The only difference worth noticing is that starting from this version mailfromd is always compiled with asynchronous syslog implementation. The --enable-syslog-async configuration file option is still available, but its meaning has changed: it sets the default syslog implementation to use (see syslog-async). Thus, it can be used the same way it was in previous versions. You can also select the syslog implementation at run time, see –syslog-async option, for more detailed information.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.16 Upgrading from 4.0 to 4.1

Upgrading to this version does not require any special efforts. You can use your configuration files and filter scripts without any changes. Notice only the following major differences between 4.1 and 4.0:


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.17 Upgrading from 3.1.x to 4.0

Before building this version, please re-read the chapter See Building, especially the section Using non-blocking syslog.

Starting from the version 4.0, MFL no longer uses the predefined symbolic names for exception codes (previous versions used the ‘&’ prefix to dereference them). Instead, it relies on constants defined in the include file status.mfh (see status.mf).

However, the script files from 3.1 series will still work, but the following warning messages will be displayed:

Warning: obsolete constant form used: &failure
Warning: remove leading '&' and include <status.mfh>

Warning: Using built-in exception codes is deprecated
Warning: Please include <status.mfh>

Another important difference is that pragmatic options ‘ehlo’ and ‘mailfromd’ are now deprecated, as well as their command line equivalents --ehlo and --domain. These options became superfluous after the introduction of mailfrom_address and ehlo_domain built-in variables. For compatibility with the previous versions, they are still supported by mailfromd 4.0, but a warning message is issued if they are used:

warning: `#pragma option ehlo' is deprecated,
 consider using `set ehlo_domain "domain.name"' instead

To update your startup scripts for the new version follow these steps:

  1. Change #pragma option mailfrom value to set mailfrom_address value. Refer to mailfrom_address, for a detailed discussion of this variable.
  2. Change #pragma option ehlo value to set ehlo_domain value. Refer to ehlo_domain, for a detailed discussion of this variable.
  3. Include status.mfh. Add the following line to the top of your startup file:
    #include_once <status.mfh>
    
  4. Remove all instances of ‘&’ in front of the constants. You can use the following sed expression: ‘s/&\([a-z]\)/\1/g’.
  5. If your code uses any of the following functions: hostname, resolve, hasmx or ismx, add the following line to the top of your script:
    #require dns
    

    See Modules, for a detailed description of the module system.

  6. Replace all occurrences of next with pass.
  7. If your code uses function match_cidr, add the following line to the top of your script:
    #require match_cidr
    

    See Modules, for a description of MFL module system.


Next: , Previous: , Up: Upgrading   [Contents][Index]

C.18 Upgrading from 3.0.x to 3.1

  1. The mailfromd binary no longer supports --config-file (-c) option. To use an alternative script file, give it as an argument, i.e. instead of:
    $ mailfromd --config-file file.rc
    

    write:

    $ mailfromd file.rc
    

    For backward compatibility, the old style invocation still works but produces a warning message. However, if mailfromd encounters the -c option it will print a diagnostic message and exit immediately. This is because the semantics of this option will change in the future releases.

  2. If a variable is declared implicitly within a function, it is created as automatic. This differs from the previous versions, where all variables were global. It is a common practice to use global variables to pass additional information between handlers (See HELO Domain, for an example of this approach). If your filter uses it, make sure the variable is declared as global. For example, this code:
    prog helo
    do
      # Save the host name for further use    
      set helohost $s
    done
    

    Figure C.1: Implicit declaration, old style

    has to be rewritten as follows:

    set helohost ""
    
    prog helo
    do
      # Save the host name for further use    
      set helohost $s
    done
    

    Figure C.2: Implicit declaration, new style

  3. Starting from version 3.1 the function dbmap takes an optional third argument indicating whether or not to count the terminating null character in key (see dbmap). If your startup script contained any calls to dbmap, change them as follows:
    in 3.0.xin 3.1
    dbmap(db, key)dbmap(db, key, 1)

Next: , Previous: , Up: Upgrading   [Contents][Index]

C.19 Upgrading from 2.x to 3.0.x

Update your startup scripts and/or crontab entries. The mailfromd binary is now installed in ${prefix}/sbin.

We also encourage you to update the startup script (run cp etc/rc.mailfromd /wherever-your-startup-lives), since the new version contains lots of enhancements.


Previous: , Up: Upgrading   [Contents][Index]

C.20 Upgrading from 1.x to 2.x

If you are upgrading from version 1.x to 2.0, you will have to do the following:

  1. Edit your script file and enclose the entire code section into:
    prog envfrom
    do
      …
    done
    

    See Handlers, for more information about the prog statement.

  2. If your code contained any rate statements, convert them to function calls (see rate), using the following scheme:
    Old statement: if rate key limit / expr
    New statement: if rate(key, interval("expr")) > limit
    

    For example,

       rate $f 180 / 1 hour 25 minutes
    

    should become

       rate($f, interval("1 hour 25 minutes")) > 180
    
  3. Rebuild your databases using the following command:
    mailfromd --compact --all   
    

    This is necessary since the format of mailfromd databases has changed in version 2.0: the key field now includes the trailing ‘NUL’ character, which is also reflected in its length. This allows for empty (zero-length) keys. See Database Maintenance, for more information about the database compaction.


Footnotes

(32)

https://www.rfc-editor.org/rfc/rfc1034#section-3.6.2

(33)

https://www.rfc-editor.org/rfc/rfc2181#section-10.1

(34)

http://www.gnu.org/software/nettle


Previous: , Up: Upgrading   [Contents][Index]