Sidney Markowitz has announced a new version of Apache SpamAssassin, a widely used open source project that provides filtering to categorize email. This version includes various improvements and bug fixes from earlier iterations.

Apache SpamAssassin -- Version 4.0.0

On behalf of the Apache SpamAssassin Project, I am pleased to announce version 4.0.0 is available.


Release Notes -- Apache SpamAssassin -- Version 4.0.0

Introduction
------------

Apache SpamAssassin 4.0.0 contains numerous tweaks and bug fixes over the past releases. In particular, it includes major changes that significantly improve the handling of text in international language.

As with any major release, there are countless functional patches and improvements to upgrade to 4.0.0. Apache SpamAssassin 4.0.0 includes several years of fixes that significantly improve classification and performance. It has been thoroughly tested in production systems. We strongly recommend upgrading as soon as possible.


Important Notes
---------------

*** On March 1, 2020, we stopped publishing rulesets with SHA-1 signatures. If you do not update to 3.4.2 or later, you will be stuck at the last ruleset with SHA-1 signatures. Such an upgrade should be to 3.4.6 to obtain the contained security fixes ***

*** Ongoing development on the 3.4 branch has ceased. All future releases and bug fixes will be on the 4.0 series, unless a new security issue is found that necessitates a 3.4.7 release. ***


Thanks
------

Many thanks to the committers (see CREDITS file), contributors, rule testers, mass checkers, and code testers who have made this release possible. We would also like to thank cPanel for their continued support of new features.

Notable features:
=================


New plugins
-----------

There are three new plugins added with this release:

#1 Mail::SpamAssassin::Plugin::ExtractText

This plugin uses external tools to extract text from message parts, and then sets the text as the rendered part. All SpamAssassin rules that apply to the rendered part will run on the extracted text as well.

#2 Mail::SpamAssassin::Plugin::DMARC

This plugin checks if emails match DMARC policy after parsing DKIM and SPF results.

#3 Mail::SpamAssassin::Plugin::DecodeShortURLs

This plugin looks for URLs shortened by a list of URL shortening services. Upon finding a matching URL, plugin will send a HTTP request to the shortening service and retrieve the Location-header which points to the actual shortened URL. It then adds this URL to the list of URIs extracted by SpamAssassin which can then be accessed by uri rules and plugins such as URIDNSBL.


Removed plugin
--------------

HashCash module, formerly deprecated, has now been removed completely


Notable changes
---------------

This release includes fixes for the following:

- Support for international text such as UTF-8 rules has been completed and significantly improved to include native UTF-8 processing

- Bayes plugin has been improved to skip common words aka noise words written in languages other than English

- OLEVBMacro plugin has been improved in order to detect more Microsoft Office macros and dangerous content. It has also been improved to extract URIs from Office documents for automatic inclusion in rules such as RBL lookups.

- You can now use Captured Tags to use tags “captured” in one rule inside other rules

- sa-update(1) tool has been improved with three new options:

#1 forcemirror: forces sa-update to use a specific mirror server,

#2 score-multiplier: adjust all scores from update channel by a given multiplier to quickly level set scores to match your preferred threshold

#3 score-limit adjusts all scores from update channel over a specified limit to a new limit

* SSL client certificate support has been improved and made easier to implement with spamc/spamd

* DKIM plugin can now detect ARC signatures

* More work on improving the configuration and internal coding to use more inclusive and less divisive language

* spamc(1) speed has been improved when both SSL and compression are used

* The normalize_charset option is now enabled by default. NOTE: Rules should not expect specific non-UTF-8 or UTF-8 encoding in the body. Matching is done against the raw body, which may vary depending on normalize_charset setting and whether UTF-8 decoding was successful.

* Mail::SPF is now the only supported module used by the SPF plugin.

* Mail::SPF::Query use is deprecated, along with settings do_not_use_mail_spf, do_not_use_mail_spf_query.

* SPF lookups are not done asynchronously and you may consider using an SPF filter at the MTA level (pypolicyd-spf / spf-engine / etc) which generates a Received-SPF header that can be parsed by SpamAssassin.

* The default sa-update ruleset doesn't make ASN lookups or header additions anymore. Configure desired methods (asn_use_geodb / asn_use_dns) and add_header clauses manually, as described in documentation for the Mail::SpamAssassin::Plugin::ASN.


New configuration options
-------------------------

All rules, functions, command line options and modules that contain "whitelist" or "blacklist" have been renamed to "welcomelist" and "blocklist" terms

Old options will continue to work for backwards compatibility until at least the Apache SpamAssassin version 4.1.0 release

New tflag "nolog" added to hide info coming from rules in SpamAssassin reports

New dns_options "nov4" and "nov6" added. IMPORTANT:; You must set nov6 if your DNS resolver is filtering IPv6 AAAA replies.

Razor2 razor_fork option added. It will fork separate Razor2 process and read in the results later asynchronously, increasing throughput. When this is used, rule priorities are automatically adjusted to -100.

Pyzor pyzor_fork option added. It will fork separate Pyzor process and read in the results later asynchronously, increasing throughput. When this is used, rule priorities are automatically adjusted to -100

urirhsbl and urirhssub rules now support "notrim" tflag, which forces querying the full hostname, instead of trimmed domain

report_charset now defaults to UTF-8 which may change the rendering of SpamAssassin reports

Notable Internal changes
------------------------

Meta rules no longer use priority values, they are evaluated dynamically when the rules they depend on are finished

DNS and other asynchronous lookups like DCC or Razor2 plugins are now launched when priority -100 is reached. This allows short circuiting at lower priority without sending unneeded DNS queries

New internal Mail::SpamAssassin::GeoDB module supporting RelayCountry and URILocalBL plugins provides a unified interface to Geographic IP modules. These include:
MaxMind::DB::Reader (GeoIP2)
Geo::IP
IP::Country::DB_File
IP::Country::Fast.

Bayes and TxRep Message-ID tracking now uses a different hashing
method


Other updates
-------------

None noted.


Optimizations
-------------

Apache SpamAssassin 4.0.0 represents years of work by the project with numerous improvements, new rule types, and internal native handling of messages in international languages. These three key optimizations will improve the efficiency of SpamAssassin:

DNS queries are now done asynchronously for overall speed improvements

DCC checks can now use dccifd asynchronously for improved throughput

Pyzor and Razor fork use separate processes done asynchronously for increased throughput


Downloading and availability
----------------------------

Downloads are available from:

https://spamassassin.apache.org/downloads.cgi   

sha256sum of archive files:

e5aa17050a30bc72baa86afdc6048cadea4d1ec2ecc61d787717a059b8319e88 Mail-SpamAssassin-4.0.0.tar.bz2
65979da7d103e3c37563f23a1a24f470090afb33664348968a00bf3d09a84f36 Mail-SpamAssassin-4.0.0.tar.gz
063d59ab2c7a67c1707b5b6a6063f97bdc9e3e8ae1246f1d43aa3dd32bf35d06 Mail-SpamAssassin-4.0.0.zip
ae4ffbb917ebc7fefa7240fc5bb5151dda663f8e4059161ad7c9b42eed1bac6d Mail-SpamAssassin-rules-4.0.0.r1905950.tgz

sha512sum of archive files:

a0fe5f6953c9df355bfa011e8a617101687eb156831a057504656921fe76c2a4eb37b5383861aac579e66a20c4454068e81a39826a35eb0266148771567bad5f Mail-SpamAssassin-4.0.0.tar.bz2
db8e5d0249d9fabfa89bc4c7309a7eafd103ae07617ed9bd32e6b35473c5efc05b1a913b4a3d4bb0ff19093400e3510ae602bf9e96290c63e7946a1d0df6de47 Mail-SpamAssassin-4.0.0.tar.gz
d907d59fd6af1560b0817d5397affeb096feaffd01614481b22a172976798f0ab438a7fb4d6878dfbb8338961f888dd69c2f7d9e743a48164e2842fa6f804571 Mail-SpamAssassin-4.0.0.zip
8ff0e68e18dc52a88fec83239bb9dc3a1d34f2dcb4c03cd6c566b97fa91242e3c8d006612aeb4df0acf43929eaaa59d542eb5cf904498343adf5eadefcb89255 Mail-SpamAssassin-rules-4.0.0.r1905950.tgz

Note that the Rules files, aka *-rules-*.tgz, are only necessary if you cannot, or do not wish to, run "sa-update" after installation. Using sa-update will download the latest rules

See the INSTALL and UPGRADE files in the distribution for important installation notes

GPG Verification Procedure
--------------------------
The release files also have a .asc accompanying them. The file serves as an external GPG signature for the given release file. The signing key is available via the keys.gnupg.net or keys.openpgp.org key
servers, as well as https://www.apache.org/dist/spamassassin/KEYS 


The following key is used to sign SA releases 3.3.0 and later:

pub 4096R/F7D39814 2009-12-02
Key fingerprint = D809 9BC7 9E17 D7E4 9BC2 1E31 FDE5 2F40 F7D3 9814
uid SpamAssassin Project Management Committee
uid SpamAssassin Signing Key (Code Signing Key, replacement for 1024D/265FA05B)
sub 4096R/7B3265A5 2009-12-02

The following key is used to sign rule updates:

pub 4096R/5244EC45 2005-12-20
Key fingerprint = 5E54 1DC9 59CB 8BAC 7C78 DFDC 4056 A61A 5244 EC45
uid updates.spamassassin.org Signing Key
sub 4096R/24F434CE 2005-12-20

To verify a release file, download the file with the accompanying .asc
file and run the following commands:

gpg --verbose --keyserver keys.openpgp.org --recv-key FDE52F40F7D39814
gpg --verify Mail-SpamAssassin-4.0.0.tar.bz2.asc
gpg --fingerprint FDE52F40F7D39814

Then confirm that the key description shown by --verify matches what is shown by --fingerprint.

See https://www.apache.org/info/verification.html  for more information on verifying Apache releases


About Apache SpamAssassin
-------------------------

Apache SpamAssassin is a mature, widely-deployed open source project that provides filtering to classify email to block spam, malware, and phishes.

Apache SpamAssassin uses a variety of mechanisms including mail header and text analysis, Bayesian filtering, DNS blocklists, collaborative filtering databases, and meta concepts to lower incorrect
classification.

Apache SpamAssassin uses a highly modular architecture that allows other technologies to be quickly incorporated as plugins to easily add or replace existing methods.

Apache SpamAssassin typically runs on a server using either command line utilities or an API to classify email so a mail system can use the results before the message reaches mailboxes.

Most of the Apache SpamAssassin is written in Perl natively supporting Unix, Linux, and macOS platforms and Microsoft Windows using Strawberry Perl.

For more information, visit https://spamassassin.apache.org/ 


About The Apache Software Foundation
------------------------------------

Established in 1999, The Apache Software Foundation provides organizational, legal, and financial support for more than 100 freely-available, collaboratively-developed Open Source projects. The pragmatic Apache License enables individual and commercial users to easily deploy Apache software; the Foundation's intellectual property framework limits the legal exposure of its 2,500+ contributors.

For more information, visit https://www.apache.org/ 

--
Sidney Markowitz
Chair, Apache SpamAssassin PMC