Rspamd 3.4 has been released

2022-11-05 00:00:00 +0000

We have released Rspamd 3.4 today. This is a bugfix release with no incompatible changes. Several new features have also been added. Here are the most important changes in this version explained.

Main changes

Sharing hyperscan database among Rspamd processes

Hyperscan databases are now shared between all Rspamd processes reducing memory footprint, especially when multiple worker processes are running.

Critical fix in the compatibility with the integrations and headers alterations

There was a critical compatibility issue, caused by the change in the milter_headers reply block that prevents some Rspamd integrations to be functional. In this release that issue has been fixed, and the compatibility with the previous output format has been restored.

Fix additional fields in the Redis schema

Some fields were no longer accepted in Redis settingsissue. Now it works correctly.

All significant changes

Here is the list of the important changes:

  • [Feature] Milter_headers: Add x-rspamd-action routine
  • [Feature] Share hyperscan database among processes
  • [Fix] Another corner case in url parsing
  • [Fix] Another fix for the enable password
  • [Fix] Another try to fix close method in lua_tcp
  • [Fix] Fix emoji joiner FP
  • [Fix] Fix favicon.ico Content-Type header
  • [Fix] Fix hang when close is used
  • [Fix] Lua_tcp: Sigh, another try to fix close invocation
  • [Fix] Mx_check: Cache the fact of a missing MX record
  • [Fix] Try to fix parsing of the unencoded > characters in html attributes
  • [Fix] Try to fix the case where password == enable_password
  • [Project] (Re)implement hyperscan caching
  • [Project] Rework cleanup
  • [Project] Synchronize hyperscan caches via the main process
  • [Rework] Convert multipattern to use hyperscan tools
  • [Rework] Make http normalize path function a generic function
  • [Rework] Split locked and unlocked files, as mmap does not need flock normally
  • [Rework] Start movement of the hyperscan related routines into a single unit
  • [Rework] Store the current worker, so other libraries could use this information
  • [Rework] Use blocking socket for IPC between main and workers
  • [Rework] Use more predictable size for commands buffers
  • [Rules] Do not insert ONCE_RECEIVED_STRICT on RDNS missing
  • [Rules] Reduce score of HTTP_TO_HTTPS - subject to remove completely

Rspamd 3.3 has been released

2022-10-01 00:00:00 +0000

We have released Rspamd 3.3 today. There are incompatible changes in this release, so please get familiar with the upgrade guide.

Here are the most important changes in this version explained.

Main changes

Reworked and redesigned symbols cache

Symbols cache is responsible for rules exectuion and planning. In this release, there was a major rework of it’s logic and functionality. For example, it can now keep track of timeouts, plan fast events before slow and implement real passthrough for the rules that define such a behaviour. It is useful, when you want some rule to be executed as quick as possible to block or pass evident spam/ham without wasting network/cpu resources. The major drawback of such a rework is that passthrough rules are now really passthrough and can prevent other rules from being executed (that is expected from the design, but it could be not the case before).

Critical fix in the neural network module

There was a regression introduced in the version 3.2 that prevented old keys in Redis to be cleaned that caused infinite Redis database growth. This is fixed in the release 3.3 and the mitigation of this bug are described in the upgrade guide.

DKIM parser now ignores unknown tags

By standard, DKIM checker must ignore unknown tags for forward compatibility. Rspamd will now behave properly and ignore unknown tags as specified in RFC.

Upstreams support in lua_http and lua_tcp modules

It is possible now to use the functionality of the upstreams directly in Lua modules that use lua_http and lua_tcp libraries. It allows better support of the names resolution, IPv6 support for resolving the hostnames and internal handling of the upstreams logic by C code automatically.

CNAME records support in the DNS resolver

Rspamd DNS resolver now supports querying and parsing of the CNAME records. This technique might be useful for fighting some specific spam patterns.

Various memory leaks detected and plugged

In this release, we have found and fixed a good bunch of memory leaks and memory corruptions in the code.

All significant changes

Here is the list of the important changes:

  • [Conf] Add missing groups for whitelist module symbols
  • [CritFix] Neural: Fix keys regression after #3968
  • [Feature] Accept upstream in lua_tcp
  • [Feature] Add ability to statically maintain disabled/enabled patterns
  • [Feature] Add function to store upstreams for HTTP urls
  • [Feature] Allow augmentations set in Lua API
  • [Feature] Allow lua_http module to accept upstreams
  • [Feature] Allow to limit write access to fuzzy storage by key
  • [Feature] Allow to sort symbols output
  • [Feature] Check content for binary stuff before dumping it to Lua
  • [Feature] Implement symbols augmentations
  • [Fix] Add missing flags
  • [Fix] Add more sanity checks for rua in dmarc_report
  • [Fix] Adjust length of the fuzzy checks for short text parts
  • [Fix] Another try to fix add headers compatibility logic
  • [Fix] Another try to fix race condition in the runtime destruction
  • [Fix] Avoid cyclic references in symcache and fix memory leaks
  • [Fix] Avoid overriding IP with Sender IP
  • [Fix] BAD_REP_POLICIES did not trigger when message was classified as spam by Bayes
  • [Fix] Bind AF_UNIX DGRAM client connection to annonymous address
  • [Fix] Disable IPv6 lookups for Blocklist.de RBL
  • [Fix] Distinguish dynamic and static items
  • [Fix] Dkim: Ignore unknown DKIM kv pairs as stated in RFC
  • [Fix] Dmarc report: Use local timezone instead of GMT
  • [Fix] Do not exclude authenticated users from URIBL lookups
  • [Fix] Empty envelopes should not be emitted as arrays (json+messagepack) when populated envelopes are objects. This greatly complicates decoding in strictly typed languages.
  • [Fix] External_relay: Restore the originating hostname check
  • [Fix] Fix DKIM keys with spaces still allowing errors on invalid base64
  • [Fix] Fix copying of sockaddr_un addresses
  • [Fix] Fix crash with cname replies
  • [Fix] Fix dependencies propagation
  • [Fix] Fix iteration over milter headers
  • [Fix] Fix ordering when sorting symcache
  • [Fix] Fix reading of the cached maps
  • [Fix] Fix several issues with the HTTP keepalive parsing
  • [Fix] Fix stack smashing
  • [Fix] Fix synchronous auth/select in lua_redis
  • [Fix] Fix various symcache issues
  • [Fix] Ignore all (I hope) unknown DKIM signature KV pairs
  • [Fix] Ignore directories in RarV5 archives
  • [Fix] Libucl: avoid memory leak on objects merging
  • [Fix] Lua_tcp: Another try to fix closing logic
  • [Fix] Mempool: Fix alloc_array function to actually multiply nmembers by size
  • [Fix] Only check allowed fuzzy worker update ips for non-unix sockets
  • [Fix] Plug memory leak in regexp destruction with pcre2
  • [Fix] Properly check the original email flag
  • [Fix] Properly deal with get_symbol/get_metric_symbol ambiguity
  • [Fix] Properly parse expressions atoms
  • [Fix] Properly set Host in rspamd_proxy
  • [Fix] Rbl: Fix received positioned checks
  • [Fix] Remove check for a score with no symbol being registered
  • [Fix] Same fix for lua_tcp
  • [Fix] Skip cname records when processing SPF records
  • [Fix] Skip sending dmarc reports in no-opt mode fixes https://github.com/rspamd/rspamd/issues/4241
  • [Fix] Stop slow timer on task destruction
  • [Fix] Symcache: Do not use C style comparators in C++ sorts
  • [Fix] Try to avoid a corner case for @ pattern
  • [Fix] Try to fix dkim reputation adjustements
  • [Fix] Try to fix passthrough results processing logic
  • [Fix] Try to fix the mess with read only flag
  • [Fix] Upstreams: Don’t ignore revive_time config option
  • [Fix] Use proper format string, sigh…
  • [Fix] Use space category in ragel automata to resolve space characters
  • [Fix] Zstd: Fix compression with the new Zstd API
  • [Fix] milter_headers: Header fields may be inserted at wrong position.
  • [Project] Rework symbols cache
  • [Rework] Rewrite rspamc in C++

Rspamd 3.2 has been released

2022-03-26 00:00:00 +0000

We have released Rspamd 3.2 today. This version is mostly bugfix release with several new features implemented.

Here are the most important changes in this version explained.

Main changes

DNS over TCP support

For a long time, Rspamd was unable to switch to TCP when processing DNS replies that are too large to be transferred over UDP. The portion of such a messages was never high, but there are some notable examples of the records that cannot fit into a UDP packet even with EDNS0 extension enabled. That are mostly poorly maintained TXT records that contains lot’s of legacy Google verification junk. However, it affected the SPF authentication, so I have decided to implement TCP fallback after all.

BIMI support

Rspamd can now download and verify logotypes from a validated certificates using a dedicated helper written in Rust. With this plugin, Rspamd can enrich your emails, appending a header with the logotype image (in SVG format) if all BIMI validation steps were successful.

Average scan time

It is now possible to fetch an average messages scan time via Rspamd HTTP API, Prometheus endpoint and even via ps command (supported on some OS only).

Monitor helper tool

You can now attach a CLI helper to Rspamd to get some real time performance graphs directly from your terminal:

Rspamd-mon-screenshot

Cloudmark support

You can now use Cloudmark via the external services module.

Other fixes and improvements

Core and API

  • Fixed SSL support in many places
  • Switched to XXHash3 as a fast hash source
  • Fix upstreams name resolution when there is also a port
  • Allow hyperscan for ppc64, as vectorscan now suports it
  • Lua_magic: Add a sane CSV heuristic
  • Allow to restore SSL handlers after keepalive pooling

Plugins

  • Neural: dd ROC feature to neural network plugin
  • Fixed retention settings in Clickhouse plugin
  • Fixed important issues in the reputation plugin

Rules

  • Added some sanity limits for symbol groups
  • Fix symbol for DKIM temporary failure
  • Remove ancient and inefficient rules

All changes

Here is the list of the important changes:

  • [Conf] Score MIME_OBFUSCATED_ARCHIVE to 8 points
  • [Conf] Set one_shot for URIBL rules by default
  • [CritFix] Fix upstreams name resolution when there is also a port
  • [Feature] Add ROC feature to neural network plugin
  • [Feature] Add public suffic compilation utility
  • [Feature] Add support of Cloudmark
  • [Feature] Allow hyperscan for ppc64, as vectorscan now suports it.
  • [Feature] Allow to skip DNS resolution for keep-alive connections
  • [Feature] Aws_s3: Allow to store large parts separately
  • [Feature] BIMI: Add preliminary version of the BIMI plugin
  • [Feature] JSON endpoint for querying maps
  • [Feature] Lua_magic: Add a sane CSV heuristic
  • [Feature] Lua_mime: Add schema for message transfer
  • [Feature] Output average scan time in /stat endpoint
  • [Feature] Show average scan time in rspamc stat output
  • [Fix] Add guards to avoid race condition on TCP connection
  • [Fix] Allow spaces in DKIM key records
  • [Fix] Apply the similar fix to the url_reputation
  • [Fix] Avoid overwriting whitelisted_signers_map
  • [Fix] Backport PR from libucl
  • [Fix] Clear SSL errors
  • [Fix] ClickHouse cleanup of old partitions
  • [Fix] Do not double call error handler on ssl errors in the timeout path
  • [Fix] Do not forget to clear pointers on IOC reset
  • [Fix] External_relay: Remove useless check of the map value
  • [Fix] Find suspicious url encodings that could break url extraction
  • [Fix] Fix HTTP(s) client timeout
  • [Fix] Fix exclude flags setting
  • [Fix] Fix expanding of the variables
  • [Fix] Fix host header usage in lua_http
  • [Fix] Fix http maps shared memory cache cleanup
  • [Fix] Fix logic in HTML processing FSM
  • [Fix] Fix parsing of the compound mailto urls
  • [Fix] Fix processing captures from pcre2
  • [Fix] Fix removing from khash
  • [Fix] Fix stuctured headers pushing
  • [Fix] Further fix for i386 compilation
  • [Fix] Improve duplicate settings error reporting
  • [Fix] Lua: task:remove_result didn’t work in some cases
  • [Fix] Output service parts as well
  • [Fix] Phishing: Deal with phishing + redirected URL
  • [Fix] Phishing: Fix finding domains in the phishing map
  • [Fix] Plug memory leak by using mempool for a copied address
  • [Fix] Properly find the request and the number of requested entries
  • [Fix] Rbl: Fix inversed logic of the url_full_hostname
  • [Fix] Read file maps if they were not pre-read during preload
  • [Fix] Restrict x86_64 assembly to x86_64
  • [Fix] Return a real number of recipients when dealing with aliases
  • [Fix] Rework unshedule DNS request function
  • [Fix] Support definition of ungrouped symbol in conf file, use group info from lua or other conf file
  • [Fix] Unschedule DNS request when clearing IO channel
  • [Fix] When checking for phishing, we need to convert punicode -> UTF8, not vice versa
  • [Fix] lua_cfg_transform - actions without score (discard)
  • [Fix] lua_cfg_transform - silly break break actions
  • [Fix] ratelimit - symbol per bucket
  • [Project] BIMI: Fix helper integration issues
  • [Project] Further DNS over TCP architecturing
  • [Project] Rdns: Add more functions for TCP based requests
  • [Project] Rdns: Add preliminary reading logic for TCP channels
  • [Project] Rdns: Add reaper for inactive TCP connections
  • [Project] Rdns: Add timeout logic for TCP requests
  • [Project] Rdns: Do not treat TCP channels failure as fatal
  • [Project] Rdns: Fix TCP connection mess
  • [Project] Rdns: Fix TCP stuff cleanup
  • [Project] Rdns: Fix various ownership issues
  • [Project] Rdns: Implement TCP writing logic
  • [Project] Rdns: Initial support of TCP IO channels
  • [Project] Rdns: More fixes in TCP handling
  • [Project] Rdns: Restore the previous EDNS0 size
  • [Project] Rdns: Send truncated replies via TCP
  • [Project] Rdns: Unregister TCP requests
  • [Rework] Allow to restore SSL handlers after keepalive pooling
  • [Rework] Allow to set a different behaviour for actions from settings
  • [Rework] Include SSL flag into keepalive hash
  • [Rework] Make rspamadm dmarc_report default behaviour more sane
  • [Rework] Mempool: Use explicit alignment
  • [Rework] Rdns: Use faster and more compact hash table for DNS requests
  • [Rework] Rework SSL flag operations
  • [Rework] Take disabled flag into account
  • [Rework] Timeouts are now global per event and not reseted by IO activity
  • [Rework] Use xxh3 as a default hash and fix memory/alignment issues
  • [Rules] Fix old rules to stop global functions usage
  • [Rules] Fix symbol for DKIM temporary failure
  • [Rules] Remove ancient and inefficient rules
  • [Rules] Slightly reduce MULTIPLE_FROM score

Rspamd 3.0 has been released

2021-08-19 00:00:00 +0000

We have released Rspamd 3.0 today. The decision to increase the major version number was taken because of the significant changes in the internal architecture Rspamd in many parts, specifically (but not limited) related to the HTML parsing. Rspamd now includes a CSS parser that can deal with the modern emails and properly extract data from them. New code is now written not in Plain C but in C++17 (might be extended to C++20 in future). Hence, to build Rspamd from the source code one would need a C++17 compatible compiler. This release includes contributions from many people, but I would like to say special thanks to the following persons:

  • Anton Yuzhaninov for testing Rspamd and being patient when dealing with the bugs, as well as for the valuable feedback on almost all questions
  • Andrew Lewis for constant work on Lua plugins and suggestions about Rspamd architecture
  • Alexander Moisseev for WebUI support and rules testing

Here are the most important changes in this version explained.

Main changes

HTML parser rework

Rspamd now parses HTML using a DOM model, so it can build and construct a parsed tree of HTML tags instead of a simple ad-hoc parsing. Adding this feature was quite untrivial, as it required full rewrite of the HTML related code. However, we can see that the new parser can deal with emails in HTML format that were completely ruined by a previous parser. Furthermore, better representation of HTML could lead to more sophisticated rules that take HTML structure into account.

CSS parser

Rspamd now has an embedded CSS parser that is currently limited to simple selectors but it could be extended in future. In conjunction with the modern HTML parser, CSS support is very valuable to extract the content from emails and distinguish visible and invisible content precisely.

Amazon S3 support

Rspamd now includes AWS API support (e.g. their signatures schema) which allows to interact with Amazon cloud services directly from Lua API. A simple plugin that stores all messages in AWS S3 cloud has also been written.

DMARC reporting rework

Rspamd DMARC reporting has proven to be troublesome in the previous version. Therefore, I have decided to move reports sending logic to a dedicated tool called rspamadm dmarc_report. The usage of this tool and DMARC reports in principle is already documented.

DMARC munging support

From version 3.0, Rspamd supports DMARC munging for the mailing list. In this mode, Rspamd will change the From: header to some pre-defined address (e.g. a mailing list address) for those messages who have valid DMARC policy with reject/quarantine that would otherwise fail during mailing list forwarding. An example of this technique is defined here.

External relay plugin

Many plugins, such as SPF, are using IP address provided by an MTA to Rspamd directly. However, in many cases there is a trusted relay that do some initial processing and hides the actual sender’s IP address. With help of this new plugin, Rspamd can now ‘fix’ this issue by treating sender’s IP as it is reported by that trusted relay.

Bayes export tool

Rspamd now allows to save and restore bayes tokens using rspamadm bayes_dump subcommand. This feature allows to move tokens between instances of Rspamd, to merge them and to analyse them manually.

Pyzor support

Rspamd now supports Pyzor via external services plugin, thanks to @defkev for this work!

Monitoring rework

Monitoring tools are now awaken less frequently reducing load on the external modules.

Other fixes and improvements

Core and API

  • Fixed retries on broken maps servers
  • Fixed handling of spaces in MIME From domains
  • Fixed handling of invalid IDN domains containing 8-bit characters
  • Fixed if..else..elseif handling in Lupa templates

Plugins

  • Neural: fixed profile filtering (offline) & unwanted retraining (online)
  • Fixed escaping in Clickhouse plugin
  • Fixed missing config schema & reconfigurability in RBL plugin (disabling RBLs; selectors)

Rules

  • Fixed CTYPE_MIXED_BOGUS for text attachments
  • Fixed PCRE-mode handling of BITCOIN_ADDR rule
  • Fixed REPLYTO_ADDR_EQ_FROM for normalised addresses

### Controller

  • OpenMetrics-compatible controller endpoint (@mrueg)
  • Health and readiness endpoints for Kubernetes (@mrueg)

All changes

Here is the list of the important changes:

  • [Conf] Align ARC scores with DKIM scores
  • [CritFix] Neural: Fix sorting application
  • [Feature] Add a simple dumper for bayes tokens
  • [Feature] Add lua_maps.fill_config_maps function
  • [Feature] Add preliminary exporter to AWS S3
  • [Feature] Add preliminary restore bayes support
  • [Feature] Add race condition protection against hs_helper restarts
  • [Feature] Add rspamd_utf8_strcmp utility
  • [Feature] Add zstd streaming API
  • [Feature] Allow to log severity level explicitly
  • [Feature] Allow to save and show attachment name when inserting AV scan results
  • [Feature] Allow to sort urls for Lua
  • [Feature] Allow to specify different timeouts/retransmits for fuzzy rules
  • [Feature] Aws_s3: Allow to compress data stored
  • [Feature] CMakeLists.txt: Change check and run-test to use rspamd-test-cxx * fixes #3807
  • [Feature] Dmarc_report: allow sending reports in batches
  • [Feature] Fuzzy_check: Allow to disable subject when making short text hash
  • [Feature] Lua_cryptobox: Add keyed ssl hash functions via HMAC
  • [Feature] Lua_task: Add get_urls_filtered method
  • [Feature] Make monitored checks less frequent
  • [Feature] Milter_headers: Add x-rspamd-pre-result header
  • [Feature] Neural: Allow to balance FP/FN for the network
  • [Feature] Ppopagate monitored errors from rbl module
  • [Feature] Pyzor calculate score dynamically Count - WL-Count of default_score in percent
  • [Feature] Rbl: Distinguish flattened and non-flattened selectors in RBL requests
  • [Feature] Re-add pyzor support
  • [Feature] Settings: add ip_map check and rework structure slightly
  • [Feature] Spamassassin: Allow to set the default priority for SA scores
  • [Feature] Strip smtp comments from message id
  • [Feature] add SYSTEM_ZSTD cmake option To use the system zstd instead on the bundled version
  • [Feature] external_relay plugin
  • [Feature] rspamadm clickhouse neural_train subcommand
  • Fix #3400 milter_headers: fix inverted logic for extended_headers_rcpt
  • Fix ASN: fix _FAIL symbol for when main symbol is disabled
  • Fix Add a special logic for text part with no text extraction
  • Fix Add diacritics flag for several eu languages
  • Fix Another FSM fix to accomodate possibility of multiple consequent ?
  • Fix Avoid curse of dynamic array referencing
  • Fix Avoid reinitialising neural settings
  • Fix Check remain before processing TXT records
  • Fix Enable error multiplier on http errors
  • Fix Finally rework parsing entities logic
  • Fix Fix ‘==’ parsing in the content type attributes parser
  • Fix Fix IPv6 expansion for SPF macros
  • Fix Fix Mozilla Message-ID detection
  • Fix Fix an edge case in BITCOIN_ADDR rule
  • Fix Fix brain-damaged behaviour when http request has a custom Host header
  • Fix Fix check of limits in email address parsing
  • Fix Fix copy&paste error and rework
  • Fix Fix expressions logic for and/or and float values
  • Fix Fix fuzzy retransmits
  • Fix Fix http maps with no or invalid expires data
  • Fix Fix last quote character parsing in the content-type state machine
  • Fix Fix normalisation flags propagation
  • Fix Fix overflow when appending many broken tags
  • Fix Fix parsing of rfc2047 tokens with ‘?’ inside
  • Fix Fix phishing flag set
  • Fix Fix rfc2047 embedded into rfc2231 pieces in special headers
  • Fix Fix round-robin rotation
  • Fix Fix searching for symbols
  • Fix Fix storing of the regexps inside variant
  • Fix Fix tokenization near exceptions
  • Fix Fix visibility calculations
  • Fix Html: Attach inline tags to the structure
  • Fix Html: Do not treat empty tags as block tags
  • Fix Ical: Do not extract urls from all flags using merely specific ones
  • Fix Initialise symcache even if it cannot be loaded properly
  • Fix Lua_fuzzy: Remove text parts check when checking image dimensions
  • Fix Lua_maps: Fix adjustments for the map type in the complex map definitions
  • Fix Lua_task: Fix deleted symbols in has_symbol/get_symbol
  • Fix Move metric and symcache link from validation to the init stage
  • Fix Oletools: Another try to fix table sorting
  • Fix One more default behaviour fix
  • Fix Phishing: Rework urls processing
  • Fix RBL: was missing some config schema
  • Fix Replies: Fix ‘Reply-To’ handling in task:get_reply_sender
  • Fix Rework metrics handling
  • Fix Save symcache on exit
  • Fix Selectors: Filter nil elements in lists
  • Fix Selectors: Properly fix implicit tostring for nils
  • Fix Try to fix some broken code in DMARC reporting plugin
  • Fix Urls: Fix processing of html urls when it comes to the flags
  • Fix Use proper buffer length
  • Fix Various visibility fixes
  • [Project] Add a simple css rule definition
  • [Project] Add css style skeleton
  • [Project] Add css syntax (adopted from ebnf)
  • [Project] Add css_selectors
  • [Project] Add doctest unit testing library
  • [Project] Add expected library
  • [Project] Add fmt library for simple string ops
  • [Project] Add fu2 library to better functions abstractions
  • [Project] Add hashing method
  • [Project] Add parsers skeleton
  • [Project] Add preliminary support of vcard parser
  • [Project] Add process exceptions for invisible text
  • [Project] Add some methods for css parser
  • [Project] Allow static libstdc++
  • [Project] Another whitespace hack
  • [Project] CSS: Various fixes in the declarations and values parsing
  • [Project] Cpp: Add robin-hood hash map library
  • [Project] Css: Add AST debug
  • [Project] Css: Add colors conversion functions
  • [Project] Css: Add dimensions handling
  • [Project] Css: Add display value support
  • [Project] Css: Add frozen library from https://github.com/serge-sans-paille/frozen/
  • [Project] Css: Add opacity support
  • [Project] Css: Add parser helpers to simplify debugging
  • [Project] Css: Add preliminary stylesheet support
  • [Project] Css: Add rules processing functions and tests
  • [Project] Css: Add simple selectors unit tests
  • [Project] Css: Add some c++ unit tests
  • [Project] Css: Add some debug methods
  • [Project] Css: Add some debug statements for the css parser
  • [Project] Css: Add some logical skeleton for declarations parser
  • [Project] Css: Add url/function tokens
  • [Project] Css: Allow at rules parsing
  • [Project] Css: Declarations parsing logic skeleton
  • [Project] Css: Enable conditional css parsing support from the HTML parser
  • [Project] Css: Finish generic lexer cases
  • [Project] Css: Fix HSL conversion
  • [Project] Css: Fix minus parsing
  • [Project] Css: Fix parser consumers nesting
  • [Project] Css: Fix parsing of the qualified rules
  • [Project] Css: Fix rules merging
  • [Project] Css: Further fixes to lexer
  • [Project] Css: Further steps to parse css colors + rework
  • [Project] Css: Further work on parser’s methods
  • [Project] Css: Implement backlog of css tokens
  • [Project] Css: Implement numbers and ident parsers
  • [Project] Css: Implement simple css selectors lookup
  • [Project] Css: Implement styles merging
  • [Project] Css: Make debug strings json like to simplify tests
  • [Project] Css: Minor adjustments
  • [Project] Css: More meat to the lexer
  • [Project] Css: Move some of the tests to the doctest
  • [Project] Css: Projected a parser
  • [Project] Css: Properties attachment logic
  • [Project] Css: Remove ragel from build targets (maybe keep for reference)
  • [Project] Css: Rework css block structure
  • [Project] Css: Rework flags of css properties
  • [Project] Css: Rework tokens structure
  • [Project] Css: Several fixes + tests
  • [Project] Css: Simplify checks
  • [Project] Css: Simplify debug code
  • [Project] Css: Start css selectors parsing logic
  • [Project] Css: Start semantic parsing for rules
  • [Project] Css: Start stylesheet implementation
  • [Project] Css: Tidy up lambdas
  • [Project] Css: rework tokeniser
  • [Project] Dmarc: Add dmarc report tool (WIP)
  • [Project] Dmarc: Add munging configuration
  • [Project] Dmarc: Add preliminary munging logic
  • [Project] Dmarc: Fix header removal
  • [Project] Dmarc: Fix munging logic
  • [Project] Dmarc: Use full recipient address instead of a domain map
  • [Project] Dmarc: Use zlists for dmarc reports
  • [Project] Dmarc_report: Add message generation logic
  • [Project] Dmarc_report: Add preliminary sending support
  • [Project] Fix lua bindings
  • [Project] Fix xml/sgml tags processing
  • [Project] Handle new modification
  • [Project] Html/CSS: Add transform from a CSS rule to html block
  • [Project] Html/CSS: Link html and css styles
  • [Project] Html/CSS: Switch styles parsing to css parser
  • [Project] Html/Css: Fix some issues found
  • [Project] Html/Css: Implement visibility rules for a block
  • [Project] Html: Add more tests cases and fix some more corner issues
  • [Project] Html: Add rows display type support
  • [Project] Html: Allow decode entities function to normalise spaces + unit tests
  • [Project] Html: Another rework of the tags structure
  • [Project] Html: Another try to fix unbalanced cases
  • [Project] Html: Fix crossing spans
  • [Project] Html: Fix parent propagation
  • [Project] Html: Further rework of the html parsing stuff
  • [Project] Html: Implement logic for tags pairing
  • [Project] Html: Implement rawtext state machine
  • [Project] Html: Insert closing tags as well :(
  • [Project] Html: More fixes
  • [Project] Html: More fixes
  • [Project] Html: More spaces logic fixes
  • [Project] Html: One more attempt to write text content
  • [Project] Html: Replace \0 in html content
  • [Project] Html: Rework img/a tags handling
  • [Project] Html: Rework propagation method
  • [Project] Html: Rework tags placement
  • [Project] Html: Rework transparency logic
  • [Project] Html: Support ‘hidden’ attribute
  • [Project] Html: Try another approach to append tags content
  • [Project] Html: Try to deal with bad unknown tags properly
  • [Project] Lua_aws: Add canonicalisation utility
  • [Project] Lua_aws: Add function to produce AWS Authorisation header
  • [Project] Lua_aws: Implement request signing
  • [Project] Lua_mime: Add lua_mime.modify_headers routine
  • [Project] Lua_task: Add modify_header method
  • [Project] Lua_task: Allow to extract modified headers
  • [Project] Make unescape code public for unit testing
  • [Project] More fixes for closed tags
  • [Project] More fixes to calculations
  • [Project] Rework API for the modified headers
  • [Project] Rework html visibility rule
  • [Project] Skeleton of the css library
  • [Project] Start headers modification API structure
  • [Project] Start working on AWS Lua API
  • [Project] Use lua_mime to modify headers
  • [Project] Use modified headers on dkim signing
  • [Project] Use string_view to constexpr variant unpacking
  • [Rework] Add composites manager concept
  • [Rework] Add tags definitions
  • [Rework] Allow C code to be compiled with C++ compiler
  • [Rework] Clickhouse: Store url flags
  • [Rework] Composites: Rewrite the composites logic
  • [Rework] Composites: Start rework of the composites framework
  • [Rework] Dmarc: Move check policy function to the common utils
  • [Rework] Dmarc: Rework reports keys structure
  • [Rework] Further work to make html content private
  • [Rework] Html/CSS: Remove css C bindings as they are useless now
  • [Rework] Html/CSS: Rework Lua bindings
  • [Rework] Html/Css: Start rework of the html blocks
  • [Rework] Html: Add images processing logic
  • [Rework] Html: Add traverse function
  • [Rework] Html: Another steps to get rid of gnode
  • [Rework] Html: Convert to variant
  • [Rework] Html: Deal with the utf_content part
  • [Rework] Html: Final rework part for the html processing code
  • [Rework] Html: Fix Lua bindings
  • [Rework] Html: Forgot to add the internal include
  • [Rework] Html: Further html urls rework
  • [Rework] Html: Further rework of the tags content extraction
  • [Rework] Html: Make parameters as a vector again
  • [Rework] Html: Move blocks part
  • [Rework] Html: Move images processing stuff
  • [Rework] Html: Rework lua bindings
  • [Rework] Html: Start html text extraction rework
  • [Rework] Html: Start refactoring of the html tags handling
  • [Rework] Html: Start removing of GNode stuff
  • [Rework] Html: Start rework of the html content structure
  • [Rework] Lua_magic: Try to detect text parts with 8bit characters for non-utf8 encodings
  • [Rework] Move HTML url functions and rework them
  • [Rework] Move and adopt entities handling logic
  • [Rework] Move common and rarely used dmarc code to the library
  • [Rework] Move compression routines outside of rspamd_util library
  • [Rework] Move entities/tags handling
  • [Rework] Phishing: Split from redirectors usage
  • [Rework] Redesign html blocks propagation logic
  • [Rework] Remove tag name string
  • [Rework] Rename phished url to a linked url
  • [Rework] Reorganize dmarc plugin and remove unsupported reporting code
  • [Rework] Reputation: Use more flexible types in get/set functions
  • [Rework] Require proper C++ environment for Rspamd build
  • [Rework] Rework extended urls output
  • [Rework] Rework tags parsing machine
  • [Rework] Slightly improve old regexp API
  • [Rework] Start conversion of the redis pool code to c++
  • [Rework] Try to resolve failed upstreams more agressively
  • [Rework] Use C++ utf8 library with unit tests to trim whitespaces
  • [Rework] Use C++ version for unicode normalisation
  • [Rework] Use C++ version of the lua threads pool
  • [Rules] Add raw addresses to MULTIPLE_FROM options
  • [Rules] Another fix to HTTP_TO_HTTPS rule
  • [Rules] Do not trigger HTML_SHORT_LINK_IMG on external images
  • [Rules] Extend FORGED_X_MAILER
  • [Rules] Extend OLD_X_MAILER
  • [Rules] Fix CTYPE_MIXED_BOGUS for text attachments
  • [Rules] Fix FPs for CTYPE_MIXED_BOGUS
  • [Rules] Fix HTTP_TO_HTTPS rule
  • [Rules] Fix HTTP_TO_HTTPS rule
  • [Rules] Fix zerofont rule (partially)
  • [Rules] Micro-optimize X_PHP_EVAL
  • [Rules] Reduce default weight for R_MISSING_CHARSET

Rspamd 2.7 has been released

2021-01-08 00:00:00 +0000

We have released Rspamd 2.7 today.

This is mostly a bug-fix release with no incompatible changes.

Here is a list of the major projects and serious bugfixes where applicable.

Fixed issues with DKIM and ARC verification

It was possible for some DKIM checks to fail where multiple signatures are present due to a canonicalisation bug. This issue has now been fixed. Arc plugin has also been fixed to support certain CV values.

Added support for S/MIME containers

From this version, Rspamd supports .p7 containers and extracting signed parts during the checks. For details see the following issue.

Several important rules rework

Anton Yuzhaninov has reworked many old rules in Rspamd improving their quality and has removed several outdated rules as well.

Support of caching for regexp multimaps

Regexp maps can now be cached on disk which should improve loading speed of large maps on reload/restart of Rspamd if they are unchanged.

Neural plugin offline learning

In this mode, Rspamd can train neural network from Clickhouse so it is possible to define better training conditions and manage learning for large systems with more fine grained control. Please refer to the corresponding documentation section for more details. Thanks to Andrew Lewis for implementing this functionality.

Other changes

Here is the list of the important changes:

  • [Conf] Add R_DKIM_PERMFAIL to the metric
  • [CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented
  • [CritFix] Fix controller paths normalisation
  • [Feature] Add INVALID_DATE rule
  • [Feature] Add controller endpoint for training neural
  • [Feature] Add sanity checks for actions thresholds
  • [Feature] Add support of ‘==’ and ‘!=’ in Rspamd expressions
  • [Feature] Composites: Improve composite atoms parser
  • [Feature] Docker: use Debian slim variant
  • [Feature] Elastic: Add some missing fields
  • [Feature] Extract text from img alt attributes
  • [Feature] Improve charset detection logic
  • [Feature] Lua_clickhouse: Add optional row callback for large selections
  • [Feature] Lua_dns_resolver: Add idna_convert_utf8 method
  • [Feature] Lua_mime: Add ability to do multipattern replacement
  • [Feature] Lua_trie: Allow to report start of the match
  • [Feature] Multimap: support adding map values as extra options
  • [Feature] Neural: Move PCA learning to a subprocess
  • [Feature] RBL: support matching content/image URLs only
  • [Feature] RBL: support use of multiple selectors
  • [Feature] Reputation: Allow to specify ip masks
  • [Feature] Support SMIME signed messages container
  • [Feature] Support multiple conditions for symbols
  • [Feature] Support ping in milter mode
  • [Feature] Support rspamd_text in selector regexps
  • [Feature] Use own daemonization routine
  • [Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade
  • [Feature] rspamadm clickhouse command
  • [Feature] allow hyperscan for aarch64
  • [Fix] Allow to set priorities between post init scripts
  • [Fix] Allow to use maps for strings that are not zero terminated
  • [Fix] Apply max_lua_urls limit for emails as well
  • [Fix] Arc: Fix CV check on signing
  • [Fix] Arc: Fix signing of the broken ARC chains
  • [Fix] Clickhouse: escape carriage return
  • [Fix] Composites: Allow partial match
  • [Fix] Deduct type of a table methods
  • [Fix] Do not load errored hyperscan database
  • [Fix] Do not process links in ignored html tags
  • [Fix] Fix ClamAV result for cached encrypted file (#3395)
  • [Fix] Fix canonicalisation when l= tag is presented
  • [Fix] Fix flag shift
  • [Fix] Fix handling of skip/skip_process http flags
  • [Fix] Fix html attachments checks
  • [Fix] Fix issue with pushing binary formats to Lua strings
  • [Fix] Fix logging for rspamadm
  • [Fix] Fix off-by-one with init check
  • [Fix] Fix parsing of escape characters in quoted pairs
  • [Fix] Fix pushing ucl strings with \0 inside
  • [Fix] Fix quoted-printable soft newlines bugged case
  • [Fix] Fix settings in case actions are set to null (#3415)
  • [Fix] Fix several issues with auth results producing
  • [Fix] Fix smtp comments exclusion
  • [Fix] Fix smtp date syntax definition
  • [Fix] Fix substring search in case if srchlen == inlen
  • [Fix] Fix text selectors
  • [Fix] Honour systemd setting when logging to console (#3514)
  • [Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities)
  • [Fix] Lua_auth_results: Quote potentially bad values in AR header
  • [Fix] Multimap: Fix flags usage
  • [Fix] Multimap: Fix scoring for combined maps
  • [Fix] Plug GList * leak in redis pool
  • [Fix] RBL: allow for multiple matches of the same label if types are different
  • [Fix] Rely on libev checks for file maps
  • [Fix] Restore simple dkim canonicalisation mode
  • [Fix] Return MimeCharset as we work with emails…
  • [Fix] Spamassassin: Fix pcre_only flags
  • [Fix] Spamassassin: Preserve ‘pcre_only’ flag when dealing with regexp replacements
  • [Fix] Try to fix GError leak
  • [Fix] Try to fix a mess with settings loading by adding priorities
  • [Fix] Try to move setings initialisation to a later stage
  • [Fix] Use dup fd in milter handler to avoid races with the proxy
  • [Fix] Use message pointer to avoid obsolete data to be cached
  • [Project] Rbl: Migrate to checks
  • [Project] Rbl: Move config code outside of the plugin
  • [Project] Ressurect empty prefilters as connection filters
  • [Project] Support connection filters registration from Lua
  • [Rework] Add final cleanup logic
  • [Rework] Add preliminary support of hyperscan caching for re maps
  • [Rework] Add stale cache removal
  • [Rework] Clickhouse: Improve performance
  • [Rework] Distinguish between strict config test mode
  • [Rework] Furhter logging improvements
  • [Rework] Milter_headers: improve extended_headers_rcpt support
  • [Rework] Move parsers to a separate lua library
  • [Rework] Neural: Skip composite symbols
  • [Rework] Rbl: Rework defaults logic
  • [Rework] Some tunes to cache saving
  • [Rework] Track maps origins
  • [Rework] Use full crypto hash for regexp maps
  • [Rules] Remove broken rule