Tokens

Based on the collection of tokens that we know can be emitted by R, this page is almost a verbatim copy of the corresponding section of the appendix in the original master's thesis.

Every R program is an expression list, identified by the exprlist token type, which consists of several expressions (identified by expr). Consider the following example:

x <- 1 + 2
if(x > 0) {
    print("Hello World!")
}
y <- 3

The corresponding expression list consists of three expressions, namely the assignment of 1 + 2 to x, the if construct, and the assignment of 3 to y.

With the following tables, we provide what is to our knowledge a full list of token types that R produces. For more information, the "Syntax" topic of the R documentation offers a great starting point.

• Tokens Representing Constants
• Tokens Representing Assignments
• Tokens Representing Operators (No Assignments)
• Tokens Representing Control-Flow Structures
• Tokens Indicating a Function Definition
• Tokens Used to Access Objects
• Tokens Representing Names
• Tokens Representing Comments or Directives
• Tokens Used To Delimit Parts of Expressions
• Tokens Representing Meta-Elements

Besides PIPEBIND, which at the moment must be enabled explicitly by setting an environment variable all the tokens shown in the tables are supported by the normalization of flowR - although this is different from supporting all of their uses.

It should be noted that there are many tokens that appear in the source code of the R interpreter but are not listed within the tables. While some of these tokens, like COLON_ASSIGN, are explicitly marked as deprecated, several of them seem to be for internal use only and are - to the best of our knowledge - never emitted by the parser with getParseData. For example:

• newlines are directly consumed to split expressions,
• error tokens produce an explicit error message, and
• the individual tokens for unary operators (like UPLUS) are transformed to the same token as their binary counterparts (like +).