Network Working Group T. Showalter Internet Draft: Sieve Carnegie Mellon Document: draft-showalter-sieve-07.txt February 24, 1999 Expire in six months Sieve: A Mail Filtering Language Status of this memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html The protocol discussed in this document is experimental and subject to change. Persons planning on either implementing or using this protocol are STRONGLY URGED to get in touch with the author before embarking on such a project. Copyright Notice Copyright (C) The Internet Society 1999. All Rights Reserved. Showalter Expire in Six Months [Page 1] Internet DRAFT Sieve February 24, 1999 Abstract This document describes a language for filtering e-mail messages at time of final delivery. It is designed to be implementable on either a mail client or mail server. It is meant to be extensible, simple, and independent of access protocol, mail architecture, and operating system. It is suitable for running on a mail server where users may not be allowed to execute arbitrary programs, such as on black box IMAP servers, as it has no variables, loops, or ability to shell out to external programs. Showalter Expire in Six Months [Page 2] Internet DRAFT Sieve February 24, 1999 Table of Contents Status of this memo ............................................... 1 Copyright Notice .................................................. 1 Abstract .......................................................... 2 0. Meta-information on this draft ............................ 5 0.1. Discussion ............................................... 5 0.2. Known Issues ............................................. 5 0.2.1. Probable Extensions ...................................... 5 0.2.2. Known Bugs ............................................... 5 0.3. Noted Changes ............................................ 6 0.3.1. since -06 ................................................ 6 0.3.2. since -05 ................................................ 7 0.3.3. since -04 ................................................ 7 1. Introduction .............................................. 9 1.1. Conventions Used in This Document ........................ 9 1.2. Example mail messages .................................... 10 2. Design .................................................... 11 2.1. Form of the Language ..................................... 11 2.2. Whitespace ............................................... 11 2.3. Comments ................................................. 11 2.4. Literal Data ............................................. 12 2.4.1. Numbers .................................................. 12 2.4.2. Strings .................................................. 12 2.4.2.1. String Lists ............................................. 13 2.4.2.2. Headers .................................................. 13 2.4.2.3. Addresses ................................................ 13 2.5. Tests .................................................... 14 2.5.1. Test Lists ............................................... 14 2.6. Arguments ................................................ 14 2.6.1. Positional Arguments ..................................... 14 2.6.2. Tagged Arguments ......................................... 14 2.6.3. Optional Arguments ....................................... 15 2.6.4. Types of Arguments ....................................... 15 2.7. String Comparison ........................................ 15 2.7.1. Match Type ............................................... 16 2.7.2. Comparisons Across Character Sets ........................ 17 2.7.3. Comparators .............................................. 17 2.7.4. Comparisons Against Addresses ............................ 18 2.8. Blocks ................................................... 18 2.9. Commands ................................................. 19 2.10. Evaluation ............................................... 19 2.10.1. Mutually Exclusive Delivery Actions ...................... 19 Showalter Expire in Six Months [Page 3] Internet DRAFT Sieve February 24, 1999 2.10.2. Implicit Keep ............................................ 19 2.10.3. Message Uniqueness in a Mailbox .......................... 20 2.10.4. Limits on Numbers of Actions ............................. 20 2.10.5. Extensions and Optional Features ......................... 20 3. Control Commands .......................................... 21 4. Action Commands ........................................... 22 4.1. Action reject ............................................ 22 4.2. Action fileinto .......................................... 23 4.3. Action redirect .......................................... 24 4.4. Action keep .............................................. 24 4.5. Action stop .............................................. 24 4.6. Action discard ........................................... 24 4.7. Action require ........................................... 25 5. Test Commands ............................................. 25 5.1. Test address ............................................. 25 5.2. Test allof ............................................... 26 5.3. Test anyof ............................................... 26 5.4. Test envelope ............................................ 26 5.5. Test exists .............................................. 27 5.6. Test false ............................................... 27 5.7. Test header .............................................. 28 5.8. Test not ................................................. 28 5.9. Test size ................................................ 28 5.10. Test true ................................................ 29 6. Extensibility ............................................. 29 6.1. Capability String ........................................ 29 6.2. Registry ................................................. 30 6.3. Capability Transport ..................................... 30 7. Transmission .............................................. 30 8. Parsing ................................................... 31 8.1. Lexical Tokens ........................................... 31 8.2. Grammar .................................................. 32 9. Extended Example .......................................... 33 10. Security Considerations ................................... 34 11. Acknowledgments ........................................... 35 12. Author's Address .......................................... 35 Appendix A. References ........................................... 35 Appendix B. Full Copyright Statement ............................. 36 Showalter Expire in Six Months [Page 4] Internet DRAFT Sieve February 24, 1999 0. Meta-information on this draft This information is intended to facilitate discussion. It will be removed when this document leaves the Internet-Draft stage. 0.1. Discussion This draft is being discussed on the MTA Filters mailing list at . Subscription requests can be sent to (send an email message with the word "subscribe" in the body). More information on the mailing list along with a WWW archive of back messages is available at . 0.2. Known Issues 0.2.1. Probable Extensions The following suggestions have been made, and will probably be addressed by extensions. An extension for regular expressions will be written. While regular expressions are of questionable utility for most users, the programmers writing implementations desperately want regular expressions. "Detailed" addressing or "sub-addressing" (i.e., the "foo" in an address "tjs+foo@andrew.cmu.edu") is not handled, and will be moved to an extension for those systems that offer it. A vacation command has been requested for an extension.Vacation functionality isn't required by the draft because having vacation assumes you can store the addresses of people who have already received vacation notifications, which isn't always the case. It is in a separate document for no particular reason. A suggestion was made to set IMAP flags on delivery (e.g., \Flagged, \Deleted, \Answered, \Seen). An "include" command is not included, but has been suggested for an extension. 0.2.2. Known Bugs I have punted on multiple fileinto. The title of 2.10.1 is probably open for suggestions. Given the existence of DSNs and the similarity of names, I'm not happy with it. Showalter Expire in Six Months [Page 5] Internet DRAFT Sieve February 24, 1999 Some discussion on how many commands are allowed per command type, site-defined limits, etc., needs to be clarified; those parts that are discussed in the Security Considerations section need to be cleaned up. 0.3. Noted Changes 0.3.1. since -06 Larry Greenfield supplied a rewrite of the grammar that separates things out into a tokenizer and a parser. This grammar also allows UTF-8 characters in strings (previous versions limited characters to the 0x01-0x7F range). Steve Hole made a number of editorial suggestions that were taken. This includes discussing a tokenizer in 2.1 and renaming sections 3, 4, and 5 ("Control Structures" became "Control Commands", "Actions" became "Action Commands", and "Tests" became "Test Commands"). Other uses of these terms in this document should have been changed to match, but I probably missed some. Lots of new rules were added to section 2.10, and should be reviewed carefully. I think that they reflect consensus, but am not sure. Tokens are defined as being case insensitive. Envelope takes a COMPARATOR argument. ADDRESS-PART defaults to :all. Test "true" has been put back. Truth was accidentally deleted. Gregory Sereda provided several examples, including a long one which has been inserted as section 9. The copyright date has been fixed and the copyright and I-D boilerplate updated with the latest and greatest from the IETF web site. Unnecessary brackets were remvoed around various syntax elements in section 2.7. Acknowledgments were moved further towards the end. Several other more minor fixes were made. Showalter Expire in Six Months [Page 6] Internet DRAFT Sieve February 24, 1999 0.3.2. since -05 Draft -05 was never published in the Internet-Drafts repository, but was circulated on the ietf-mta-filters@imc.org mailing list. All nits submitted by Greg Sereda are hopefully addressed. Most of these were example bugs, but he also pointed out that types for arguments were under-specified and in several cases orders of arguments disagreed with the syntax. "Match keyword" was changed to "match type" as an editorial change. "Forward" was renamed to "redirect" because of the conflict between multiple meanings of "forward" in order to make it clear exactly what we meant. Limitation of one redirect per message should be removed. The types of arguments have been added to their syntax line. Added "require" back in a slightly different form. "Require" is now an action (arbitrarily) and has been added to sec. 2.10 as well. Implementations are responsible for not allowing mail loops. All discussion of short-circuit evaluation has been removed. On a related note, tests must not have side effects. Envelope is required to drop source routes. An address-matching primitive has been added. 0.3.3. since -04 Here are a list of changes from draft 04. (It may not be complete.) * Concensus: i;ascii-casemap is required. * Consensus: i;ascii-casemap is the default. * Header name compares are always case-insensitive; the draft now says so. * Several examples were fixed, but it is likely that errors remain. * Bug: Section 7, remove reference to "support". * There are two namespaces for extension names, one "vnd.", one Showalter Expire in Six Months [Page 7] Internet DRAFT Sieve February 24, 1999 everything else, like MIME. * All XXXs have been removed, except for in IANA section. * Fileinto is optional, and discussion of local mail folders and POP3 has been removed. * A non-present comparator is considered to be basically a syntax error. * Resent headers are not to be added by the "redirect" command. * Tagged arguments must follow the keyword, and may not be interspersed with positional arguments. * Envelope-matching commands are to be added with the syntax that Barry suggested. * Put back :matches match type. * What happens when an error occurs has been dropped. * Reject is now optional. * Implementations are encouraged to decode header charsets, and if they don't, are required to not do compares on 8-bit data. Showalter Expire in Six Months [Page 8] Internet DRAFT Sieve February 24, 1999 1. Introduction This memo documents a language that can be used to create filters for electronic mail. It is not tied to any particular operating system or mail architecture. It requires the use of [IMAIL]-compliant messages, but should otherwise generalize to many systems. The language is powerful enough to be useful, but limited in power in order to allow for a safe server-side filtering system. The intention is to make it impossible for users to do anything more complex (and dangerous) than write simple mail filters, along with facilitating GUI-based editors. The language is not Turing-complete, and provides no way to write a loop or a function. Variables are not provided. Implementations of the language are expected to take place at time of final delivery, when the message is moved to the user-accessible mailbox. In systems where the MTA does final delivery, such as traditional Unix mail, it is reasonable to sort when the MTA deposits mail into the user's mailbox. There are a number of reasons to use a filtering system. Mail traffic for most users has been increasing due both to increased usage of e-mail, the emergence of unsolicited email as a form of advertising, and increased usage of mailing lists. Experience at Carnegie Mellon has shown that if a filtering system is made available to users, many will make use of it in order to file messages from specific users or mailing lists. However, many others did not make use of the Andrew system's FLAMES filtering language [FLAMES] due to difficulty in setting it up. Because of the expectation that users will make use of filtering if it is offered and easy to use, this language has been made simple enough to allow many users to make use of it, but rich enough that it can be used productively. However, it is expected that GUI-based editors will be the preferred way of editing filters for a large number of users. 1.1. Conventions Used in This Document In the sections of this document that discuss the requirements of various keywords and operators, the following conventions have been adopted. The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", "CAN", and "MAY" in this document are to be interpreted as defined in [KEYWORDS]. Showalter Expire in Six Months [Page 9] Internet DRAFT Sieve February 24, 1999 Each section on a command (test, action, or control structure) has a line labeled "Syntax:". This line describes the syntax of the command, including its name and its arguments. Required arguments are listed inside angle brackets ("<" and ">"). Optional arguments are listed inside square brackets ("[" and "]"). Each argument is followed by its type, so "" represents an argument called "key" that is a string. Literal strings are represented with double-quoted strings. Alternatives are separated with slashes, and parenthesis are used for grouping, similar to [ABNF]. In the "Syntax" line, there are three special pieces of syntax that are frequently repeated, MATCH-TYPE, COMPARATOR, and ADDRESS-PART. These are discussed in sections 2.7.1, 2.7.3, and 2.7.4, respectively. The formal grammar for these commands in section 10 and is the authoritative reference on how to construct commands, but the formal grammar does not specify the order, semantics, number or types of arguments to commands, nor the legal command names. The intent is to allow for extension without changing the grammar. 1.2. Example mail messages The following mail messages will be used throughout this document in examples. Message A ----------------------------------------------------------- Date: Tue, 1 Apr 1997 09:06:31 -0800 (PST) From: coyote@desert.org To: roadrunner@birdseed.org Subject: I have a present for you Look, I'm sorry about the whole anvil thing, and I really didn't mean to try and drop it on you from the top of the cliff. I want to try to make it up to you. I've got some great birdseed over here at my place--top of the line stuff--and if you come by, I'll have it all wrapped up for you. I'm really sorry for all the problems I've caused for you over the years, but I know we can work this out. -- Wile E. Coyote "Super Genius" coyote@znic.net ----------------------------------------------------------- Showalter Expire in Six Months [Page 10] Internet DRAFT Sieve February 24, 1999 Message B ----------------------------------------------------------- From: youcouldberich!@reply-by-postal-mail.invalid Sender: b1ff@de.res.frobnitzm.edu To: rube@landru.melon.net Date: Mon, 31 Mar 1997 18:26:10 -0800 Subject: $$$ YOU, TOO, CAN BE A MILLIONAIRE! $$$ YOU MAY HAVE ALREADY WON TEN MILLION DOLLARS, BUT I DOUBT IT! SO JUST POST THIS TO SIX HUNDRED NEWSGROUPS! IT WILL GUARANTEE THAT YOU GET AT LEAST FIVE RESPONSES WITH MONEY! MONEY! MONEY! COLD HARD CASH! YOU WILL RECEIVE OVER $20,000 IN LESS THAN TWO MONTHS! AND IT'S LEGAL!!!!!!!!! !!!!!!!!!!!!!!!!!!111111111!!!!!!!11111111111!!1 JUST SEND $5 IN SMALL, UNMARKED BILLS TO THE ADDRESSES BELOW! ----------------------------------------------------------- 2. Design 2.1. Form of the Language The language consists of a set of commands. Each command consists of a set of tokens delimited by whitespace. The first token is the command string followed by zero or more arguments. Arguments may be literal data, tags, blocks of commands, or test commands. The language is represented in UTF-8, as specified in [UTF-8]. Tokens in the ASCII range are considered case-insensitive. 2.2. Whitespace Whitespace is used to separate tokens. Whitespace is made up of tabs, newlines (CRLF, never just CR or LF), and the space character. The amount of whitespace used is not significant. 2.3. Comments Comments begin with a "#" character that is not contained within a string and continue until the next CRLF. Comments are semantically equivalent to whitespace and are permitted to be used anyplace that whitespace is (with one exception in multi-line strings, as described in the grammar). Example: if size :over 100K { # this is a comment discard; } Showalter Expire in Six Months [Page 11] Internet DRAFT Sieve February 24, 1999 2.4. Literal Data Literal data means data that is not executed, merely evaluated "as is", to be used as arguments to commands. Literal data is limited to numbers and strings. 2.4.1. Numbers Numbers are given as ordinary decimal numbers. However, those numbers that have a tendency to be fairly large, such as message sizes, MAY have a "K", "M", or "G" appended to indicate a multiple of a base-two number. To be comparable with the power-of-two-based versions of SI units that computers frequently use, K specifies kilo, or 1,024 (2^10) times the value of the number; M specifies mega, or 1,048,576 (2^20) times the value of the number; and G specifies giga, or 1,073,741,824 (2^30) times the value of the number. Implementations MUST provide 31 bits of magnitude in numbers, but may provide more. Negative, fractional, and decimal numbers are not permitted by this specification. 2.4.2. Strings Scripts involve large numbers of strings, as they are used for pattern matching, addresses, and textual bodies, etc. Typically, short quoted strings suffice for most uses, but a more convenient form is provided for longer strings such as bodies of messages. A quoted string starts and ends with a single double quote (the <"> character, ASCII 34). A backslash ("\", ASCII 92) inside of a quoted string is followed by either another backslash or a double quote. This two-character sequence represents a single backslash or double- quote within the string, respectively. Other escape sequences may be permitted depending on context. An undefined escape sequence (such as "\a" in a context where "a" has no special meaning) is interpreted as if there were no backslash (in this case, "\a" is just "a"). Non-printing characters such as tabs, CR and LF, and control characters are permitted in strings. NUL (ASCII 0) is not allowed in strings. For entering larger amounts of text, such as an email message, a multi-line form is allowed. It starts with the keyword "text:", followed by a CRLF, and ends with the sequence of a CRLF, a single Showalter Expire in Six Months [Page 12] Internet DRAFT Sieve February 24, 1999 period, and another CRLF. In order to allow the message to begin lines with a single-dot, lines are dot-stuffed. That is, when composing a message body, an extra `.' is added before each line which begins with a `.'. When the server interprets the script, these extra dots are removed. Note that a comment or whitespace may occur in between the "text:" and the CRLF, but not within the string itself. 2.4.2.1. String Lists When matching patterns, it is frequently convienent to match against groups of strings instead of single strings. For this reason, a list of strings is allowed in many tests, implying that if the test is true using any one of the strings, then the test is true. Implementations are encouraged to use short-circuit evaluation in these cases. For instance, the test `header :contains ["To", "Cc"] ["me@frobnitzm.edu", "me00@landru.melon.edu"]' is true if either the To header or Cc header of the input message contains either of the e-mail addresses "me@frobnitzm.edu" or "me00@landru.melon.edu". Conversely, in any case where a list of strings would be appropriate, a single string is allowed without being a member of a list: it is equivalent to a list with a single member. This means that the test `exists "To"' is equivalent to the test `exists ["To"]'. 2.4.2.2. Headers Headers are a subset of strings. In the Internet Message Specification [IMAIL], each header line is allowed to have whitespace nearly anywhere in the line, including after the field name and before the subsequent colon. Extra spaces between the header name and the ":" in a header field are ignored. A header name never contains a colon. The "From" header refers to a line beginning "From:" (or "From :", etc.). No header will match the string "From:" due to the trailing colon. 2.4.2.3. Addresses A number of commands call for email addresses, which are also a subset of strings. These addresses must be compliant with [IMAIL]. Implementations MUST ensure that the addresses are syntactically valid, but need not ensure that they are actually deliverable. Showalter Expire in Six Months [Page 13] Internet DRAFT Sieve February 24, 1999 2.5. Tests Tests are given as arguments to commands in order to control how the run. Generally, a test is used to decide if a block of code should be evaluated. Tests MUST NOT have side effects. That is, a test must not make changes to state. No tests in this specification have side effects, and side effects are forbidden in extensions as well. The rationale for this is that tests with side effects impair readability and maintainability and are difficult to represent in a graphic interface to generating scripts, so side effects have been confined to actions where they are clearer. 2.5.1. Test Lists Some tests (allof and anyof, which implement logical and and or, respectively) need to take more than a single test as an argument. The test-list syntax element provides a way of grouping tests. Example: if anyof (not exists ["From", "Date"], header :contains "from" "fool@znic.edu") { discard; } 2.6. Arguments In order to specify what to do, most commands take arguments. There are three types of arguments: positional, tagged, and optional. 2.6.1. Positional Arguments Positional arguments are given to a command which discerns their meaning based on their order. When a command takes positional arguments, all positional arguments must be supplied, and must be in the order prescribed. 2.6.2. Tagged Arguments This document provides for tagged arguments in the style of CommonLISP. These are also similar to flags given to commands in most command-line systems. A tagged argument is an an argument for a command that begins with ":", and consists of a tag naming the argument, such as ":contains". This argument means that zero or more of the next tokens have some particular meaning, depending on the argument. These next tokens may Showalter Expire in Six Months [Page 14] Internet DRAFT Sieve February 24, 1999 be numbers or strings, but are never blocks. So tagged arguments are similar to positional arguments, except that instead of the meaning being derived from the command, it is derived from the tag. Tagged arguments must appear before positional arguments, but they may appear in any order. For convenience, this is not expressed in the syntax definitions with commands, but they still may be reordered arbitrarily provided they appear before positional arguments. Tagged arguments may be mixed with optional arguments. To keep the language simple, tagged arguments should not take tagged arguments as arguments. 2.6.3. Optional Arguments Optional arguments are exactly like tagged arguments except that they may be left out, in which case a default value is implied. Because optional arguments tend to result in shorter scripts, they have been used far more than tagged arguments. One particularly noteworthy case is the ":comparator" argument, which allows the user to specify which [ACAP] comparator will be used to compare two strings, since different languages may impose different orderings on UTF-8 [UTF-8] characters. 2.6.4. Types of Arguments Abstractly, arguments may be literal data, tests, or blocks of commands. In this way, an "if" control structure is merely a command that happens to take a test and a block as arguments and may execute the block of code. However, this abstraction is ambiguous from a parsing standpoint. The grammar in section 9.2 presents a parsable version of this: arguments are string-lists, numbers, and tags, which may be followed by a test or a test-list, which may be followed by a block of commands. No more than one test or test list, nor more than one block of commands, may be used, and commands that end with blocks of commands do not end with semicolons. 2.7. String Comparison When matching one string against another, there are a number of ways of performing the match. These are accomplished with three types of matches: exact match, a substring match, and a wildcard glob-style match. In order to provide for matches between character sets and Showalter Expire in Six Months [Page 15] Internet DRAFT Sieve February 24, 1999 case insensitivity, Sieve borrows ACAP's comparator registry. However, when a string is being used to represent the name of a header, the comparator is never user-specified. Header comparisons are always done in a case-insensitive manner, since this is the way things are specified in the message specification [IMAIL]. That is, header-name comparisons are always done with the "i;ascii-casemap" comparator. 2.7.1. Match Type There are three allowed match types describing the allowed match in this draft: they are ":is", ":contains", and ":matches". Match type are supplied to those commands which allow them to specify whether the match is to be a complete match or not. These are used as tagged arguments to tests that perform string comparison. Exactly one of them is necessary for a command. The ":contains" version describes a substring match. If the value argument contains the key argument as a substring, the match is true. For instance, the string "frobnitzm" contains "frob" and "nit", but not "fbm". The null key ("") is contained in all values. The ":is" version describes an absolute match; if the contents of the first string are absolutely the same as the contents of the second string, they match. Only the string "frobnitzm" is the string "frobnitzm". The null key only ":is" the null value. The ":matches" version specifies a wildcard match using the characters "*" and "?". "*" matches any number of characters, and "?" matches a single character. "?" and "*" may be escaped as "\?" and "\*" in strings to match against those characters. In order to specify what type of match is supposed to happen, commands that support matching take optional tagged arguments ":matches", ":is", and ":contains". Commands default to using ":is" matching. Note that these modifiers may interact with comparators; in particular, some comparators are not suitable for matching with ":contains" or ":matches". It is an error to use a comparator with ":contains" or ":matches" that is not compatible with it. For convenience, the "MATCH-TYPE" syntax element is defined here as follows: Syntax: ":is" / ":contains" / ":matches" Showalter Expire in Six Months [Page 16] Internet DRAFT Sieve February 24, 1999 2.7.2. Comparisons Across Character Sets All Sieve scripts are represented in UTF-8, but messages may involve a number of character sets. In order for comparisons to work across character sets, implementations SHOULD implement the following behavior: Implementations decode header charsets to UTF-8. Two strings are considered equal if their UTF-8 representations are identical. Implementations should decode charsets represented in the forms specified by [MIME] for both message headers and bodies. Implementations must be capable of decoding US-ASCII, ISO-8859-1, the ASCII subset of ISO-8859-* character sets, and UTF-8. If implementations fail to support the above behavior, they MUST conform to the following: No two strings can be considered equal if one contains octets greater than 127. 2.7.3. Comparators In order to allow for language-independent, case-independent matches, the match type may be coupled with a comparator name. Comparators are described for [ACAP]; a registry is defined for ACAP, and this specification uses that registry. ACAP defines multiple comparator types. Only equality types are used in this specification. All implementations MUST support the "i;octet" comparator (simply compares octets) and the "i;ascii-casemap" comparator (which treats uppercase and lowercase English characters as the same). If left unspecified, the default is "i;ascii-casemap". Some comparators may not be usable with substring matches; that is, they may only work with ":is". It is an error to try and use a comparator with ":matches" or ":contains" that is not compatible with it. A comparator is specified with commands that support matching by the ":comparator" option. This option is followed by a string providing the name of the comparator to be used. For convenience, the syntax of a comparator is abbreviated to "COMPARATOR", and (repeated in several tests) is as follows: Syntax: ":comparator" Showalter Expire in Six Months [Page 17] Internet DRAFT Sieve February 24, 1999 So in this example, Example: if header :contains :comparator "i;octet" "Subject" "MAKE MONEY FAST" { discard; } would discard any message with subjects like "You can MAKE MONEY FAST", but not "You can Make Money Fast", since the comparator used is not case-sensitive. If a comparator is not known to an implementation, it is treated in the same way as an error. Both ":matches" and ":contains" match type are compatible with the "i;octet" and "i;ascii-casemap" comparators and may be used with them. 2.7.4. Comparisons Against Addresses Addresses are probably one of the most frequent representations as strings. Because these are structured and being able to compare against the local-part or the domain of an address is useful, some tests that act exclusively on addresses take an additional optional argument that specifies what the test acts on. These optional arguments are ":localpart", ":domain", and ":all", which act on the local-part (left-side), the domain part (right- side), and the whole address. The kind of comparison is done, such as whether or not the comparison done is case-insensitive, is specified as a comparator argument to the test. If an optional address-part is omitted, there is an impled default of ":all". For convenience, the "ADDRESS-PART" syntax element is defined here as follows: Syntax: ":localpart" / ":domain" / ":all" 2.8. Blocks Blocks are sets of commands enclosed within curly braces. Blocks are supplied to commands so that the commands can implement control Showalter Expire in Six Months [Page 18] Internet DRAFT Sieve February 24, 1999 commands. A control structure is a command that happens to take a test and a block as one of its argument; depending on the result of the test supplied as another argument, it runs the code in the block some number. (Note that by the commands supplied in the specification, there are no loops, so the control structures supplied--if, elsif, and else--run a block either once or not at all.) 2.9. Commands Sieve scripts are sequences of commands. Commands can take any of the tokens above as arguments, and arguments may be either tagged or positional arguments. There are three kinds of commands, test commands, action commands, and control commands. The simplest is an action command. An action command is an identifier followed by zero or more arguments, terminated by a semicolon. Action commands do not take tests or blocks as arguments. A control command is similar, but it takes a test as an argument, and ends with a block instead of a semicolon. A test command is used as part of a control command. It is used to specify whether or not the block of code given to the control command is executed. 2.10. Evaluation 2.10.1. Mutually Exclusive Delivery Actions Actions that do not affect delivery status can be used multiple times and in any combination with each other. In the base draft, these actions are "fileinto" and "redirect". Only one action that affects delivery status may be taken. An attempt to run more than one such action leads to a run-time error, which has undefined behavior. In the base draft, these actions are "keep", "discard", and "reject". 2.10.2. Implicit Keep Previous experience with filtering systems suggests that cases tend to be missed in scripts. To prevent massive errors, Sieve has an "implicit keep". Showalter Expire in Six Months [Page 19] Internet DRAFT Sieve February 24, 1999 An implicit keep is performed if a message is not written to a mailbox, redirected to a new address, or explicitly thrown out. That is, if a fileinto, a keep, a redirect, or a discard is performed, an implicit keep is not. For instance, with any of the short messages offered above, the following script produces no actions. Example: if size :over 500K { discard; } As a result, the implicit keep would be taken. 2.10.3. Message Uniqueness in a Mailbox Implementations SHOULD NOT write a message to a mailbox where a copy of it already exists, even if a script explicitly asks for a message to be written to a mailbox twice. The test for equality of two messages is not defined by this memo. 2.10.4. Limits on Numbers of Actions Site policy may limit numbers of actions taken. In the event that a policy limits the number of actions taken on a particular message, the actions that are generated first in a script should be followed. 2.10.5. Extensions and Optional Features Because of the differing capabilities of many mail systems, several features of this specification have been specified as optional. Before any of these extensions can be used, they must be declared with the "require" action. If an extension is not enabled with "require", implementations MUST treat it as if they did not support it at all. If a script does not understand an extension declared with require, the script must not be used at all. Note: The reason for this restriction is that prior experiences with languages such as LISP and Tcl suggest that this is a workable way of noting that a given script uses an extension. Experience with languages such as PostScript that have extension mechanisms that allow a script to include information on how to work around a lack of an extension suggest that such mechanisms do not work well in practice. Showalter Expire in Six Months [Page 20] Internet DRAFT Sieve February 24, 1999 3. Control Commands In order for a script to do more than one set of actions, control structures are needed. In Sieve, a control structure is a command that takes a block as an argument. In this document, only "if" is provided. There are three pieces to if: "if", "elsif", and "else". Syntax: if Syntax: elsif Syntax: else The semantics are similar to any other programming language this appears in. When the interpreter sees an "if", it evaluates the test associated with it. If the test is true, it executes the block associated with it. If the test of the "if" is false, it evaluates the test of the first "elsif" (if any). If the test of "elsif" is true, it runs the elsif's block. An elsif may be followed by an elsif, in which case, the interpreter repeats this process until it runs out of elsifs. When the interpreter runs out of elsifs, there may be an "else" case. If there is, and none of the if or elsif tests were true, the interpreter runs the else case. This provides a way of performing exactly one of the blocks in the chain. In the following example, both Message A and B are dropped. Example: require "fileinto"; if header :contains "from" "coyote" { discard; } elsif header :contains ["subject"] ["$$$"] { discard; } else { fileinto "INBOX"; } Showalter Expire in Six Months [Page 21] Internet DRAFT Sieve February 24, 1999 In the script below, when run over message A, redirects the message to acm@frobnitzm.edu; message B, to postmaster@frobnitzm.edu; any other message is redirected to field@frobnitzm.edu. Example: if header :contains ["From"] ["coyote"] { redirect "acm@frobnitzm.edu"; } elsif header :contains "Subject" "$$$" { redirect "postmaster@frobnitzm.edu"; } else { redirect "field@frobnitzm.edu"; } Note that this definition prohibits the "... else if ..." sequence used by C. This is intentional, because this construct produces a shift-reduce conflict. 4. Action Commands This document supplies six actions that may be taken on a message: keep, fileinto, redirect, reject, discard, and stop. 4.1. Action reject Syntax: reject The optional "reject" action resends the message to the sender, wrapping it in a "reject" form, noting that it was rejected by the recipient. In the following script, message A is rejected and returned to the sender. Example: require "reject"; if header :contains "from" "coyote@znic.net" { reject "I am not taking mail from you, and I don't want your birdseed, either!"; } Showalter Expire in Six Months [Page 22] Internet DRAFT Sieve February 24, 1999 A reject message MUST takes the form of a failed DSN as specified by [DSN]. The human-readable portion of the message, the first component of the DSN, contains the human readable message describing the error, although it SHOULD contain additional text alerting the original sender that mail was refused by a filter. This part of the DSN might appear as follows: ------------------------------------------------------------ Message was refused by recipient's mail filtering program. Reason given was as follows: I am not taking mail from you, and I don't want your birdseed, either! ------------------------------------------------------------ The action-value field as defined in the DSN specification MUST be "failed". A rejected message may not be filed, redirected, or kept. A message that triggers a "reject" action is never allowed to be kept by the user, and the "reject" overrides all other actions. A message may only be rejected once. Because some implementations cannot implement the reject command, it is optional. The capability string to be used with the require command is "reject". 4.2. Action fileinto Syntax: fileinto The "fileinto" action drops the message into a named folder. Implementations SHOULD support fileinto, but in some environments this may be impossible. The capability string for use with the require command is "fileinto". In the following script, message A is filed into folder "INBOX.harassment". Example: require "fileinto"; if header :contains ["from"] "coyote" { fileinto "INBOX.harassment"; } Showalter Expire in Six Months [Page 23] Internet DRAFT Sieve February 24, 1999 4.3. Action redirect Syntax: redirect The "redirect" action is used to send the message to another user at a supplied address, as a mail forwarding feature does. The "redirect" action makes no changes to the message body or headers, and only modifies the envelope recipient. The redirect command performs an MTA-style "forward"--that is, what you get from a .forward file using sendmail under UNIX. The address on the SMTP envelope is replaced with the one on the redirect command and the message is sent back out. (This is not an MUA-style forward, which creates a new message with a different sender and message ID, wrapping the old message in a new one.) The redirect command does not add Resent-* headers. A simple script can be used for redirecting: Example: redirect "bart@frobnitzm.edu"; 4.4. Action keep Syntax: keep The "keep" action is whatever action is taken in lieu of all other actions, if no filtering happens at all; generally, this simply means to file the message into the user's main mailbox. This command provides a way to execute this action without needing to know the name of the user's main mailbox, providing a way to call it without needing to understand the user's setup, or the underlying mail system. Example: if size :under 1M { keep; } else { discard; } 4.5. Action stop Syntax: stop The "stop" action ends all processing. If no actions have been executed, then the keep action is taken. 4.6. Action discard Syntax: discard Showalter Expire in Six Months [Page 24] Internet DRAFT Sieve February 24, 1999 Discard drops the message. In the following script, any mail from "idiot@frobnitzm.edu" is thrown out. Example: if header :contains ["from"] ["idiot@frobnitzm.edu"] { discard; } Discard takes no arguments. While an important part of this language, "discard" has the potential to create serious problems for users: a student leaving themselves logged in to a machine in a computer lab may find their script changed to just "discard". In order to protect users in this situation (along with similar situations), implementations MAY keep messages destroyed by a script for an indefinite period, and MAY disallow scripts that throw out all mail. 4.7. Action require Syntax: require The require action notes that a script makes use of an certain extension. Such a declaration is required to use the extension, as discussed in section 2.10.5. Multiple capabilities can be declared with a single require. Example: require ["fileinto", "reject"]; 5. Test Commands Tests are used in conditionals to decide which part(s) of the conditional to execute. 5.1. Test address Syntax: address [ADDRESS-PART] [COMPARATOR] [MATCH-TYPE] The address test matches Internet addresses out of structured headers that contain addresses. It returns true if any header contains any key in the specified part of the address, as modified by the comparator and the match keyword. Like envelope and header, this test returns true if any combination of the header-list and key-list arguments match. Internet email addresses [IMAIL] have the somewhat awkward Showalter Expire in Six Months [Page 25] Internet DRAFT Sieve February 24, 1999 characteristic that the mailbox-part to the left of the at-sign is considered case sensitive, and the domain-part to the right of the at-sign is case insensitive. The "address" command does not deal with this itself, but provides the ADDRESS-PART argument for allowing users to deal with it. The address primitive never acts on the phrase part of an email address, nor on comments within that address. It also never acts on group names, although it does act on the addresses within the group construct. Implementations MUST restrict the address test to headers that contain addresses, but MUST include at least From, To, Cc, Bcc, Sender, Resent-From, Resent-To, and SHOULD include any other header that utilizes an "address-list" structured header body. Example: if address :is :all "from" "tim@example.com" { discard; 5.2. Test allof Syntax: allof ( , , ..., ) The allof test preforms a logical AND on the tests supplied to it. Example: allof (false, false) => false allof (false, true) => false allof (true, true) => true The allof test takes as its argument a test-list. 5.3. Test anyof Syntax: anyof ( , , ..., ) The anyof test preforms a logical OR on the tests supplied to it. Example: anyof (false, false) => false anyof (false, true) => true anyof (true, true) => true 5.4. Test envelope Syntax: envelope [COMPARATOR] [ADDRESS-PART] [MATCH-TYPE] Showalter Expire in Six Months [Page 26] Internet DRAFT Sieve February 24, 1999 The "envelope" test is true if the specified part of the SMTP (or equivalent) envelope matches the specified key. If one of the envelope-part strings is (case insensitive) "from", then matching occurs against the FROM address used in the SMTP MAIL command. If one of the envelope-part strings is (case insensitive) "to", then matching occurs against the TO address used in the SMTP RCPT command that resulted in this message getting delivered to this user. Note that only the most recent TO is avaliable. The envelope-part is a string list and may contain both "from" and "to", in which case the strings specified in the key-list are matched against both parts of the envelope. Like address and header, this test returns true if any combination of the envelope-part and key-list arguments is true. All tests against envelopes MUST drop source routes. If a protocol other than SMTP is used for message transport, implementations are expected to adapt this command, mapping the "from" and "to" envelope parts to the appropriate parts of the envelope. Example: require "envelope"; if envelope :all :is "from" "tim@example.com" { discard; } 5.5. Test exists Syntax: exists The "exists" test is true if the headers listed in the header-names argument exist within the message. All of the headers must exist or the test is false. The following example throws out mail that doesn't have a From header and a Date header. Example: if not exists ["From","Date"] { discard; } Showalter Expire in Six Months [Page 27] Internet DRAFT Sieve February 24, 1999 5.6. Test false Syntax: false The "false" test always evaluates to false. 5.7. Test header Syntax: header [COMPARATOR] [MATCH-TYPE] The "header" test evaluates to true if the any header name matches any key. The type of match is specified by the optional match argument, which defaults to ":is" if not specified, as specified in section 2.6. Like address and envelope, this test returns true if any combination of the string-list and key-list arguments match. If a header listed in the header-names argument exists, it contains the null key (""). However, if the named header is not present, it does not contain the null key. So if a message contained the header X-Caffeine: C8H10N4O2 these tests on that header evaluate as follows: header :is ["X-Caffeine"] [""] => false header :contains ["X-Caffeine"] [""] => true 5.8. Test not Syntax: not The "not" test takes some other test as an argument, and yields the opposite result. 5.9. Test size Syntax: size <":over" / ":under"> The "size" test deals with the size of a message. It takes either a tagged argument of ":over" or ":under", followed by a number representing the size of the message. If the argument is ":over", and the size of the message is greater than the number provided, the test is true; otherwise, it is false. If the argument is ":under", and the size of the message is less than Showalter Expire in Six Months [Page 28] Internet DRAFT Sieve February 24, 1999 the number provided, the test is true; otherwise, it is false. One of ":over" or ":under" must be specified. The size of a message is defined to be the number of octets from the initial header until the last character in the message body. 5.10. Test true Syntax: true The "true" test always evaluates to true. 6. Extensibility New control structures, actions, and tests can be added to the language. Sites must make these features known to their users; this document does not define a way to discover the list of extensions supported by the server. Any extensions to this language MUST define a capability string that uniquely identifies that extension. If a new version of an extension changes the functionality of a previously defined extension, it MUST use a different name. In a situation where there is a submission protocol and an extension advertisement mechanism aware of the details of this language, scripts submitted can be checked against the mail server to prevent use of an extension that that the server does not support. Extensions should state how they interact with constraints defined in section 2.10 (for instance, whether they cancel the implicit keep, or if they change delivery status). 6.1. Capability String Capability strings are typically short strings describing what capabilities are supported by the server. Capability strings beginning with "vnd." represent vendor-defined extensions. Such extensions are not defined by Internet standards or RFCs, but are still registered with IANA in order to prevent conflicts. Extensions starting with "vnd." should be followed by the name of the vendor, such as "vnd.acme.rocket-sled". The following capability strings are defined by this document: envelope The string "envelope" indicates that the implementation Showalter Expire in Six Months [Page 29] Internet DRAFT Sieve February 24, 1999 supports the "envelope" command. fileinto The string "fileinto" indicates that the implementation supports the "fileinto" command. reject The string "reject" indicates that the implementation supports the "reject" command. comparator- The string "comparator-elbonia" is provided if the implementation supports the "elbonia" comparator. Therefore, all implementations have at least the "comparator-i;octet" capability. 6.2. Registry In order to provide a standard set of extensions, a registry is provided by IANA. Capability names may be registered on a first- come, first-served basis. Extensions designed for interoperable use should be defined as standards track or IESG approved experimental RFCs. To: XXX@XXX.XXX Subject: Registration of new Sieve extension Capability name: Capability keyword: Capability arguments: Standards Track/IESG-approved experimental RFC number: Person and email address to contact for further information: 6.3. Capability Transport As the range of mail systems that this draft is intended to apply to is quite large, a method of advertising which capabilities an implementation supports is difficult due to the wide range of possible implementations. Such a mechanism, however, should have the following properties. (1) The implementation can advertise the complete set of extensions that it supports. 7. Transmission The MIME type for a Sieve script is "application/sieve". Showalter Expire in Six Months [Page 30] Internet DRAFT Sieve February 24, 1999 The registration of this type for RFC 2048 requirements is as follows: Subject: Registration of MIME media type application/sieve MIME media type name: application MIME subtype name: sieve Required parameters: none Optional parameters: none Encoding considerations: Most sieve scripts will be textual, written in UTF-8. When non-7bit characters are used, quoted-printable would be appropriate for transport systems that require 7bit encoding. Security considerations: Discussed in section 10 of RFC XXXX. Interoperability considerations: Discussed in section 2.10.5 of RFC XXXX. Published specification: RFC XXXX. Applications which use this media type: sieve-enabled mail servers Additional information: Magic number(s): File extension(s): .siv Macintosh File Type Code(s): Person & email address to contact for further information: See the discussion list at ietf-mta-filters@imc.org. Intended usage: COMMON Author/Change controller: See Author information in RFC XXXX. 8. Parsing The Sieve grammar is seperated into tokens and a seperate grammar as most programming languages are. 8.1. Lexical Tokens Sieve scripts are encoded in UTF-8. The following assumes a valid UTF-8 encoding; special characters in Sieve scripts are all ASCII. The following are tokens in Sieve: - identifiers - tags - numbers - quoted strings - multi-line strings - other separators Blanks, horizonal tabs, newlines, formfeeds, and comments ("white Showalter Expire in Six Months [Page 31] Internet DRAFT Sieve February 24, 1999 space") are ignored except as they separate tokens. Some white space is required to separate otherwise adjacent tokens and in specific places in the multi-line strings. The other separators are single individual characters, and are mentioned explicitly in the grammar. The lexical structure of sieve is defined in the following BNF (as described in [ABNF]): CHAR-NOT-DOT = (%x01-2d / %x2f-%xff) CHAR-NOT-CRLF = (%x01-09 / %x0b-%x0c / %x0e-%xff) comment = "#" *CHAR-NOT-CRLF CRLF identifier = (ALPHA / "_") *(ALPHA DIGIT "_") tag = ":" identifier number = 1*DIGIT [QUANTIFIER] QUANTIFIER = "K" / "M" / "G" quoted-string = DQUOTE *CHAR DQUOTE ;; in general, CHAR inside a string maps to CHAR ;; so ;; note that newlines and other characters are all allowed strings multi-line = "text:" *(SP / HTAB) (comment / CRLF) *((1*CHAR-NOT-DOT *CHAR CRLF) / ("." 1*CHAR-NOT-DOT *CHAR CRLF) / (".." *CHAR CRLF) / CRLF) "." CRLF ;; note when used, ;; a leading ".." on a line is mapped to ".". white-space = 1*(SP / CRLF / HTAB) / comment 8.2. Grammar The following is the grammar of Sieve after it has been lexical interpreted. No white space or comments appear below. The start symbol is "start". argument = string-list / number / tag arguments = *argument [test / test-list] block = "{" commands "}" Showalter Expire in Six Months [Page 32] Internet DRAFT Sieve February 24, 1999 command = identifier arguments ( ";" / block ) commands = *command start = commands string = quoted-string / multi-line string-list = "[" string *("," string) "]" / string ;; if there is only a single string, the brackets are optional test = identifier arguments test-list = "(" test *("," test) ")" 9. Extended Example The following is an extended example of a Sieve script. Note that it does not make use of the implicit keep. # # Example Sieve Filter # Declare any optional features or extension used by the script # require ["fileinto", "reject"]; # # Reject any large messages # if size :over 1M { reject text: Please do not send me large attachments. Put your file on a server and send me the URL. Thank you. ; stop; } # # Handle messages from known mailing lists # Move messages from IETF filter discussion list to filter folder # if header :is "Sender" "owner-ietf-mta-filters@imc.org" { fileinto "filter"; # move to "filter" folder } Showalter Expire in Six Months [Page 33] Internet DRAFT Sieve February 24, 1999 # # Keep all messages from within my company # elsif address :domain :is "From" "company.com" { keep; # keep in "In" folder } # # Move broadcast messages from lists unknown to "spam" folder # elsif not address :all :contains ["To", "Cc", "Bcc"] "me@company.com" { # If message header does not contain my address, # it's from a list. fileinto "spam"; # move to "spam" folder } else { # Move all other (non-company) mail to "personal" # folder. fileinto "personal"; } 10. Security Considerations Users must get their mail. It is imperative that whatever method implementations use to store the user-defined filtering scripts be secure. It is equally important that implementations sanity-check the user's scripts, and not allow users to create on-demand mailbombs. For instance, an implementation that allows a user to reject or redirect multiple times to a single message might also allow a user to create a mailbomb triggered by mail from a specific user. Therefore, an implementation SHOULD only allow one "reject" per message processed, and MAY limit the number of redirect actions taken. An implementation MUST refuse to redirect a message to itself. Several commands, such as "discard", "redirect", and "fileinto" allow for actions to be taken that are potentially very dangerous. In order to prevent mail loops, implementations must prevent messages from passing through a given user twice. Showalter Expire in Six Months [Page 34] Internet DRAFT Sieve February 24, 1999 11. Acknowledgments I am very thankful to Chris Newman for his support and his ABNF syntax checker, to John Myers and Steve Hole for outlining the requirements for the original drafts, to Larry Greenfield for nagging me about the grammar and finally fixing it, to Greg Sereda for repeatedly fixing and providing examples, and to Rob Earhart for an early implementation and a great deal of help. I am also indebted to all of the readers of the ietf-mta-filters@imc.org mailing list. 12. Author's Address Tim Showalter Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 E-Mail: tjs+@andrew.cmu.edu Showalter Expire in Six Months [Page 35] Internet DRAFT Sieve February 24, 1999 Appendix A. References [ABNF] Crocker, D., and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", Internet Mail Consortium, RFC 2234, November 1997. [DSN] Moore, K., and G. Vaudreuil, "An Extensible Message Format for Delivery Status Notifications", RFC 1894, January 1996. [FLAMES] Borenstein, N, and C. Thyberg, "Power, Ease of Use, and Cooperative Work in a Practical Multimedia Message System", Int. J. of Man-Machine Studies, April, 1991. Reprinted in Computer-Supported Cooperative Work and Groupware, Saul Greenberg, editor, Harcourt Brace Jovanovich, 1991. Reprinted in Readings in Groupware and Computer-Supported Cooperative Work, Ronald Baecker, editor, Morgan Kaufmann, 1993. [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, Harvard University, March 1997. [IMAP] Crispin, M., "Internet Message Access Protocol - version 4rev1", RFC 2060, University of Washington, December 1996. [IMAIL] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, University of Delaware, August 1982. [MIME] Freed, N., and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, Innosoft and First Virtual, November 1996. [SMTP] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821, USC/Information Sciences Institute, August 1982. [UTF-8] Yergeau, F. "UTF-8, a transformation format of Unicode and ISO 10646", RFC 2044, Alis Technologies, October 1996. Showalter Expire in Six Months [Page 36] Internet DRAFT Sieve February 24, 1999 Appendix B. Full Copyright Statement Copyright (C) The Internet Society 1999. All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Showalter Expire in Six Months [Page 37]