I'm using several email services, but most of my private email is on an IMAP based service. A lot of the received emails should be automatically moved to a sub-folder on the server without my involvement. In most email clients you can easily create rules to do this. However, I would like to do this even when I have no email clients up and running. For my work email service (Outlook) this can be done. Outlook provides server side rules that can move emails to the appropriate sub-folders. With a standard IMAP service this is not the case. However, I can use a specialised client running somewhere performing the tasks of such rules on my IMAP accounts. The solution is to set up this client on a server performing my IMAP rules on a regular basis.

The search for a solution

One tool that can do this is the command line tool IMAPFilter by Lefteris Chatzimparmpas. See these two blog posts for more information and examples on how to do it:

• If you can't beat 'em, clean 'em: Using imapfilter for remote rules to an IMAP mailbox
• Filtering IMAP mail with imapfilter

The IMAPFilter tool is written in C, but it uses the Lua programming language as a configuration and extension language. When I started on this path, I soon discovered that the configuration file with all the rules became too large and complex (or unfamiliar) for me to maintain. It was too easy for me to introduce an error in the configuration file.

The conclusion was: I needed a better tool (for me), and I might have to create it myself. Since my preferred programming language is Python, I searched for a Python solution. The imaplib module from the Python Standard Library could be a starting point. With imaplib, I could create my own program performing the matching of emails and operations on the IMAP server performing my email rules. But I quickly found out that this would soon become a rather large project. In particular, the implementation of email matching would be a significant body of work. I was not motivated for that.

However, another option popped up close to the top of the search results. The imap_tools Python module by Vladimir Kaukin seems to fit my needs better. It is build on top of imaplib, but it provides some nice extra functionality limiting the number of lines of code I had to write. What I found especially important for my needs included:

• An easy way to create an expression to match messages where a given rule should apply.
• An easy way to move a message from the INBOX to a mailbox in a sub-folder.

The solution: build it myself

My solution is a Python program with some command line arguments (to provide IMAP server and account details) and a config file with the rules. In my first attempt, the rules could include the following commands:

• seen: mark emails as seen
• move: move emails to a folder
• copy: copy emails to a folder

Later, I extended the program with the following command using the smtplib module from the Python Standard Library:

• redirect: redirect emails to an email address

To search for matching emails we use the search citeria from imap_tools:

• AND: combines by logical and condition
• OR: combines by logical or condition
• NOT: inverts the logical expression
• Header: for search by headers
• UidRange: for search by UID range
• Date: for search by dates

The search criteria are case-insensitive and they can be complex. An example:

OR(AND(from_="facebookmail.com"),AND(from_="facebook.com"))

This example is an or-expression matching messages, either from an address containing "facebookmail.com", or from an address containing "facebook.com". The from-argument has the trailing underscore, since from without it is a reserved word in Python. Since Python does not allow two arguments with the same name (from_, in this case), the two from-email addresses are wrapped inside and-expressions. We can simplify this example with the usage of a list of strings:

OR(from_=["facebook.com","facebookmail.com"])

With the use of a list of strings, we can skip wrapping the arguments in and-expressions. This is also possible with and- and not-expressions. This example matches emails where the subject contain both the text "gathering" and the text "beach":

AND(subject=["gathering","beach"])

OR(from_="facebook.com",to="my@facebook.email")

This example returns messages where the from address contains "facebook.com" or where the to address contains "my@facebook.email".

The config file with the rules

The config file is just a Python file with a single dictionary-expression evaluated in a strict namespace (with only the above commands and search criteria). Each item in the dictionary is one rule. For each rule we have three items:

• "ENABLED": boolean, where the rule is enabled if True
• "FETCH": tuple with the (possibly nested) search criteria
• "ACTIONS": list of the commands performed on the matching emails

This is an example of a config file with one rule:

{        
    "Facebook": {
        "ENABLED": True,
        "FETCH": (OR(from_=["facebook.com","facebookmail.com"]),),
        "ACTIONS": [
            (seen,),
            (move, "path/to/archive/Facebook")
        ]        
    }
}

The rule above has the title "Facebook", is enabled, and performs the two commands seen and move in the listed order, where the last moves the matching messages to the folder "path/to/archive/Facebook". In the example, the email-matching ("FETCH") is an or-expression matching messages, either from an address containing "facebookmail.com", or from an address containing "facebook.com".

This is another example of a config file. It has 4 rules:

{        

    "Important Show": {
        "ENABLED": True,
        "FETCH": (AND(to="my@patreon.email", subject="Important Show"),),
        "ACTIONS": [
            (seen,),
            (move, "INBOX/Archive/Important Show")
        ]        
    },

    "Patreon": {
        "ENABLED": True,
        "FETCH": (AND(AND(to="my@patreon.email"), NOT(subject="Important Show")),),
        "ACTIONS": [
            (seen,),
            (move, "INBOX/Archive/Patreon")
        ]        
    },

    "Smash and Bananas": {
        "ENABLED": True,
        "FETCH": (AND(to="smash@bananas.email"),),
        "ACTIONS": [
            (seen,),
            (copy, "INBOX/Archive/Smash")
            (move, "INBOX/Archive/Bananas")
        ]        
    },

    "Wrong receiver": {
        "ENABLED": True,
        "FETCH": AND(to="wrong@receiver.email"),
        "ACTIONS": [
            (redirect, "right@receiver.email"),
            (move, "INBOX/Redirected")
        ]        
    },

}

In the last rule, you might notice that the search criteria is not given as a tuple. This is just a simplification of the syntax: if only one search criteria is given (possibly a nested one), we can specify it directly, and not necessary as a tuple with a single item. This is also true for the comands with no arguments. For example, the first rule above can be written like this in a more simplified syntax:

"Important Show": {
    "ENABLED": True,
    "FETCH": AND(to="my@patreon.email", subject="Important Show"),
    "ACTIONS": [
        seen,
        (move, "INBOX/Archive/Important Show")
    ]        
}

It is a large number of possible argument types (keys) that can be used in the search criteria. See the Search key table for a complete list. A few examples with some of these argument types:

• All emails sent to "my@private.email" containing the text "garage rock" either in the headers or in the body with a sent date earlier than January 1, 2000:

  AND(to="my@private.email", text="garage rock", sent_date_lt=Date(2000,1,1))
  

• All emails not seen with an "X-Mailer" header containing the text "ColdFusion":

  AND(seen=False, header=Header("X-Mailer","ColdFusion"))
  

• All emails with an "X-Spam-Score" header containing (at least) 15 pluses (spam score greater than 15) or email message with size greater than 8192 octets:

  OR(header=Header("X-Spam-Score","+++++++++++++++"), size_gt=8192)
  

The program and its arguments

The config file does not include any information about the IMAP-server and email accounts. The program fetches this from the command line arguments. This is the help-message describing the command-line arguments:

usage: IMAPrules [-h] --config CONFIG --server SERVER [--port PORT] [--smtp SMTP]
                 [--smtp-port SMTP_PORT] --account ACCOUNT [--smtp-account SMTP_ACCOUNT]
                 [--pw PASSWORD] [--smtp-pw PASSWORD] [--log LOG]

Perform IMAP rules.

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       the config file with the rules
  --server SERVER       the IMAP server
  --port PORT           the IMAP server port (default used if not given)
  --smtp SMTP           the SMTP server (default is the IMAP server)
  --smtp-port SMTP_PORT
                        the SMTP server port (default used if not given)
  --account ACCOUNT     the IMAP account
  --smtp-account SMTP_ACCOUNT
                        the SMTP account (default is the IMAP account)
  --pw PASSWORD         the IMAP account password (not from keychain)
  --smtp-pw PASSWORD    the SMTP account password (not from keychain)
  --log LOG             print log messages to stdout (0, 1 or 2)

A more detailed description of the arguments:

--config:		The path to the config file with the rules (required)
--server:		The hostname or IP-address of the IMAP server (required)
--port:		The IMAP server port (default port used if not provided)
--smtp:		The SMTP server (only used if redirect rules are used, default IMAP server)
--smtp-port:		The SMTP server port (default port used if not provided)
--account:		The IMAP account (the user's email account, required)
--smtp-account:		The SMTP account (default IMAP account)
--pw:		The IMAP account password (default fetch it from the keychain/keyring)
--smtp-pw:		The SMTP account password (default the password of the IMAP account)
--log:		The log message level (default 0 [silent], possible values 0, 1 or 2)

We recommend that the password are stored in the keychain or keyring of the system. We use the Python keyring module to implement this, and I guess the supported systems are qual to the ones supported by that module (however, it is only tested on macOS). This is how I typically run the program:

IMAPrules --config imaprules.py --server mail.server --account my@account --log 1

On a Mac, I can add the password to the system keychain with the following command (you will be prompted for the password):

security add-generic-password -a my@account -s mail.server -w

The program can also be used as a module, but currently, you have to use the source code as the documentation for this. I install the program without the .py extension somewhere in the path found by my shell. Inspect or download the program from here:

• IMAPrules.py (raw)
• IMAPrules.py (htmlized code)