Improve this page Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using local clone. Page wiki View or edit the community-maintained wiki page associated with this page.

Lexical

The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes what those tokens are. The grammar is designed to be suitable for high speed scanning, it has a minimum of special case rules, there is only one phase of translation, and to make it easy to write a correct scanner for. The tokens are readily recognizable by those familiar with C and C++.

Source Text

D source text can be in one of the following formats: UTF-8 is a superset of traditional 7-bit ASCII. One of the following UTF BOMs (Byte Order Marks) can be present at the beginning of the source text:

UTF Byte Order Marks
FormatBOM
UTF-8EF BB BF
UTF-16BEFE FF
UTF-16LEFF FE
UTF-32BE00 00 FE FF
UTF-32LEFF FE 00 00
ASCIIno BOM

If the source file does not start with a BOM, then the first character must be less than or equal to U0000007F.

There are no digraphs or trigraphs in D.

The source text is decoded from its source representation into Unicode Characters. The Characters are further divided into: WhiteSpace, EndOfLine, Comments, SpecialTokenSequences, Tokens, all followed by EndOfFile.

The source text is split into tokens using the maximal munch technique, i.e., the lexical analyzer tries to make the longest token it can. For example >> is a right shift token, not two greater than tokens. There are two exceptions to this rule:

Character Set

Character:
    any Unicode character

End of File

EndOfFile:
    physical end of the file
    \u0000
    \u001A
The source text is terminated by whichever comes first.

End of Line

EndOfLine:
    \u000D
    \u000A
    \u000D \u000A
    \u2028
    \u2029
    EndOfFile
There is no backslash line splicing, nor are there any limits on the length of a line.

White Space

WhiteSpace:
    Space
    Space WhiteSpace

Space:
    \u0020
    \u0009
    \u000B
    \u000C

Comments

Comment:
    BlockComment
    LineComment
    NestingBlockComment

BlockComment:
    /* Characters */

LineComment:
    // Characters EndOfLine

NestingBlockComment:
    /+ NestingBlockCommentCharacters +/

NestingBlockCommentCharacters:
    NestingBlockCommentCharacter
    NestingBlockCommentCharacter NestingBlockCommentCharacters

NestingBlockCommentCharacter:
    Character
    NestingBlockComment

Characters:
    Character
    Character Characters

D has three kinds of comments:

  1. Block comments can span multiple lines, but do not nest.
  2. Line comments terminate at the end of the line.
  3. Nesting block comments can span multiple lines and can nest.

The contents of strings and comments are not tokenized. Consequently, comment openings occurring within a string do not begin a comment, and string delimiters within a comment do not affect the recognition of comment closings and nested "/+" comment openings. With the exception of "/+" occurring within a "/+" comment, comment openings within a comment are ignored. 

a = /+ // +/ 1;    // parses as if 'a = 1;'
a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";'
a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'
Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.

Tokens

Token:
    Identifier
    StringLiteral
    CharacterLiteral
    IntegerLiteral
    FloatLiteral
    Keyword
    /
    /=
    .
    ..
    ...
    &
    &=
    &&
    |
    |=
    ||
    -
    -=
    --
    +
    +=
    ++
    <
    <=
    <<
    <<=
    <>
    <>=
    >
    >=
    >>=
    >>>=
    >>
    >>>
    !
    !=
    !<>
    !<>=
    !<
    !<=
    !>
    !>=
    (
    )
    [
    ]
    {
    }
    ?
    ,
    ;
    :
    $
    =
    ==
    *
    *=
    %
    %=
    ^
    ^=
    ^^
    ^^=
    ~
    ~=
    @
    =>
    #

Identifiers

Identifier:
    IdentifierStart
    IdentifierStart IdentifierChars

IdentifierChars:
    IdentifierChar
    IdentifierChar IdentifierChars

IdentifierStart:
    _
    Letter
    UniversalAlpha

IdentifierChar:
    IdentifierStart
    0
    NonZeroDigit
Identifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D. (This is the C99 Standard.) Identifiers can be arbitrarily long, and are case sensitive. Identifiers starting with __ (two underscores) are reserved.

String Literals

StringLiteral:
    WysiwygString
    AlternateWysiwygString
    DoubleQuotedString

    HexString
    DelimitedString
    TokenString

WysiwygString:
    r" WysiwygCharacters " StringPostfixopt

AlternateWysiwygString:
    ` WysiwygCharacters ` StringPostfixopt

WysiwygCharacters:
    WysiwygCharacter
    WysiwygCharacter WysiwygCharacters

WysiwygCharacter:
    Character
    EndOfLine

DoubleQuotedString:
    " DoubleQuotedCharacters " StringPostfixopt

DoubleQuotedCharacters:
    DoubleQuotedCharacter
    DoubleQuotedCharacter DoubleQuotedCharacters

DoubleQuotedCharacter:
    Character
    EscapeSequence
    EndOfLine

EscapeSequence:
    \'
    \"
    \?
    \\
    \0
    \a
    \b
    \f
    \n
    \r
    \t
    \v
    \x HexDigit HexDigit
    \ OctalDigit
    \ OctalDigit OctalDigit
    \ OctalDigit OctalDigit OctalDigit
    \u HexDigit HexDigit HexDigit HexDigit
    \U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit
    \ NamedCharacterEntity

HexString:
    x" HexStringChars " StringPostfixopt

HexStringChars:
    HexStringChar
    HexStringChar HexStringChars

HexStringChar:
    HexDigit
    WhiteSpace
    EndOfLine

StringPostfix:
    c
    w
    d

DelimitedString:
    q" Delimiter WysiwygCharacters MatchingDelimiter "

TokenString:
    q{ Tokens }

A string literal is either a double quoted string, a wysiwyg quoted string, an escape sequence, a delimited string, a token string, or a hex string.

In all string literal forms, an EndOfLine is regarded as a single \n character.

Wysiwyg Strings

Wysiwyg "what you see is what you get" quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string. There are no escape sequences inside r" ": 

r"hello"
r"c:\root\foo.exe"
r"ab\n" // string is 4 characters,
        // 'a', 'b', '\', 'n'

An alternate form of wysiwyg strings are enclosed by backquotes, the ` character. The ` character is not available on some keyboards and the font rendering of it is sometimes indistinguishable from the regular ' character. Since, however, the ` is rarely used, it is useful to delineate strings with " in them. 

`hello`
`c:\root\foo.exe`
`ab\n`  // string is 4 characters,
        // 'a', 'b', '\', 'n'

Double Quoted Strings

Double quoted strings are enclosed by "". Escape sequences can be embedded into them with the typical \ notation.

"hello"
"c:\\root\\foo.exe"
"ab\n"   // string is 3 characters,
         // 'a', 'b', and a linefeed
"ab
"        // string is 3 characters,
         // 'a', 'b', and a linefeed

Hex Strings

Hex strings allow string literals to be created using hex data. The hex data need not form valid UTF characters. 

x"0A"              // same as "\x0A"
x"00 FBCD 32FD 0A" // same as
                   // "\x00\xFB\xCD\x32\xFD\x0A"
Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.

Adjacent strings are concatenated with the ~ operator, or by simple juxtaposition: 
"hello " ~ "world" ~ "\n" // forms the string
       // 'h','e','l','l','o',' ',
       // 'w','o','r','l','d',linefeed
The following are all equivalent: 
"ab" "c"
r"ab" r"c"
r"a" "bc"
"a" ~ "b" ~ "c"

The optional StringPostfix character gives a specific type to the string, rather than it being inferred from the context. This is useful when the type cannot be unambiguously inferred, such as when overloading based on string type. The types corresponding to the postfix characters are:

String Literal Postfix Characters
PostfixTypeAka
cimmutable(char)[]string
wimmutable(wchar)[]wstring
dimmutable(dchar)[]dstring
"hello"c  // string
"hello"w  // wstring
"hello"d  // dstring

The string literals are assembled as UTF-8 char arrays, and the postfix is applied to convert to wchar or dchar as necessary as a final step.

String literals are read only. Writes to string literals cannot always be detected, but cause undefined behavior.

Delimited Strings

Delimited strings use various forms of delimiters. The delimiter, whether a character or identifer, must immediately follow the " without any intervening whitespace. The terminating delimiter must immediately precede the closing " without any intervening whitespace. A nesting delimiter nests, and is one of the following characters:

Nesting Delimiters
DelimiterMatching Delimiter
[]
()
<>
{}
q"(foo(xxx))"   // "foo(xxx)"
q"[foo{]"       // "foo{"

If the delimiter is an identifier, the identifier must be immediately followed by a newline, and the matching delimiter is the same identifier starting at the beginning of the line: 

writeln(q"EOS
This
is a multi-line
heredoc string
EOS"
);

The newline following the opening identifier is not part of the string, but the last newline before the closing identifier is part of the string. The closing identifier must be placed on its own line at the leftmost column.

Otherwise, the matching delimiter is the same as the delimiter character:

q"/foo]/"          // "foo]"
// q"/abc/def/"    // error

Token Strings

Token strings open with the characters q{ and close with the token }. In between must be valid D tokens. The { and } tokens nest. The string is formed of all the characters between the opening and closing of the token string, including comments. 

q{foo}              // "foo"
q{/*}*/ }           // "/*}*/ "
q{ foo(q{hello}); } // " foo(q{hello}); "
q{ __TIME__ }       // " __TIME__ "
    // i.e. it is not replaced with the time
// q{ __EOF__ }     // error
    // __EOF__ is not a token, it's end of file

Character Literals

CharacterLiteral:
    ' SingleQuotedCharacter '

SingleQuotedCharacter:
    Character
    EscapeSequence
Character literals are a single character or escape sequence enclosed by single quotes, ' '.

Integer Literals

IntegerLiteral:
    Integer
    Integer IntegerSuffix

Integer:
    DecimalInteger
    BinaryInteger
    HexadecimalInteger

IntegerSuffix:
    L
    u
    U
    Lu
    LU
    uL
    UL

DecimalInteger:
    0
    NonZeroDigit
    NonZeroDigit DecimalDigitsUS

BinaryInteger:
    BinPrefix BinaryDigitsUS

BinPrefix:
    0b
    0B

HexadecimalInteger:
    HexPrefix HexDigitsNoSingleUS

NonZeroDigit:
    1
    2
    3
    4
    5
    6
    7
    8
    9

DecimalDigits:
    DecimalDigit
    DecimalDigit DecimalDigits

DecimalDigitsUS:
    DecimalDigitUS
    DecimalDigitUS DecimalDigitsUS

DecimalDigitsNoSingleUS:
    DecimalDigit
    DecimalDigit DecimalDigitsUS
    DecimalDigitsUS DecimalDigit

DecimalDigitsNoStartingUS:
    DecimalDigit
    DecimalDigit DecimalDigitsUS

DecimalDigit:
    0
    NonZeroDigit

DecimalDigitUS:
    DecimalDigit
    _

BinaryDigitsUS:
    BinaryDigitUS
    BinaryDigitUS BinaryDigitsUS

BinaryDigit:
    0
    1

BinaryDigitUS:
    BinaryDigit
    _

OctalDigits:
    OctalDigit
    OctalDigit OctalDigits

OctalDigitsUS:
    OctalDigitUS
    OctalDigitUS OctalDigitsUS

OctalDigit:
    0
    1
    2
    3
    4
    5
    6
    7

OctalDigitUS:
    OctalDigit
    _

HexDigits:
    HexDigit
    HexDigit HexDigits

HexDigitsUS:
    HexDigitUS
    HexDigitUS HexDigitsUS

HexDigitsNoSingleUS:
    HexDigit
    HexDigit HexDigitsUS
    HexDigitsUS HexDigit

HexDigitsNoStartingUS:
    HexDigit
    HexDigit HexDigitsUS

HexDigit:
    DecimalDigit
    HexLetter

HexLetter:
    a
    b
    c
    d
    e
    f
    A
    B
    C
    D
    E
    F
    _

Integers can be specified in decimal, binary, octal, or hexadecimal.

Decimal integers are a sequence of decimal digits.

Binary integers are a sequence of binary digits preceded by a ‘0b’.

C-style octal integer notation was deemed too easy to mix up with decimal notation. The above is only fully supported in string literals. D still supports octal integer literals interpreted at compile time through the std.conv.octal template, as in octal!167.

Hexadecimal integers are a sequence of hexadecimal digits preceded by a ‘0x’.

Integers can have embedded ‘_’ characters, which are ignored. The embedded ‘_’ are useful for formatting long literals, such as using them as a thousands separator: 

123_456       // 123456
1_2_3_4_5_6_  // 123456

Integers can be immediately followed by one ‘L’ or one of ‘u’ or ‘U’ or both. Note that there is no ‘l’ suffix.

The type of the integer is resolved as follows:

Decimal Literal Types
LiteralType
Usual decimal notation
0 .. 2_147_483_647int
2_147_483_648 .. 9_223_372_036_854_775_807long
Explicit suffixes
0L .. 9_223_372_036_854_775_807Llong
0U .. 4_294_967_296Uuint
4_294_967_296U .. 18_446_744_073_709_551_615U ulong
0UL .. 18_446_744_073_709_551_615ULulong
Hexadecimal notation
0x0 .. 0x7FFF_FFFFint
0x8000_0000 .. 0xFFFF_FFFFuint
0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFFlong
0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFF ulong
Hexadecimal notation with explicit suffixes
0x0L .. 0x7FFF_FFFF_FFFF_FFFFLlong
0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFL ulong
0x0U .. 0xFFFF_FFFFUuint
0x1_0000_0000U .. 0xFFFF_FFFF_FFFF_FFFFU ulong
0x0UL .. 0xFFFF_FFFF_FFFF_FFFFULulong

Floating Point Literals

FloatLiteral:
    Float
    Float Suffix
    Integer ImaginarySuffix
    Integer FloatSuffix ImaginarySuffix
    Integer RealSuffix ImaginarySuffix

Float:
    DecimalFloat
    HexFloat

DecimalFloat:
    LeadingDecimal .
    LeadingDecimal . DecimalDigits
    DecimalDigits . DecimalDigitsNoStartingUS DecimalExponent
    . DecimalInteger
    . DecimalInteger DecimalExponent
    LeadingDecimal DecimalExponent

DecimalExponent
    DecimalExponentStart DecimalDigitsNoSingleUS

DecimalExponentStart
    e
    E
    e+
    E+
    e-
    E-

HexFloat:
    HexPrefix HexDigitsNoSingleUS . HexDigitsNoStartingUS HexExponent
    HexPrefix . HexDigitsNoStartingUS HexExponent
    HexPrefix HexDigitsNoSingleUS HexExponent

HexPrefix:
    0x
    0X

HexExponent:
    HexExponentStart DecimalDigitsNoSingleUS

HexExponentStart:
    p
    P
    p+
    P+
    p-
    P-


Suffix:
    FloatSuffix
    RealSuffix
    ImaginarySuffix
    FloatSuffix ImaginarySuffix
    RealSuffix ImaginarySuffix

FloatSuffix:
    f
    F

RealSuffix:
    L

ImaginarySuffix:
    i

LeadingDecimal:
    DecimalInteger
    0 DecimalDigitsNoSingleUS

Floats can be in decimal or hexadecimal format.

Hexadecimal floats are preceded with a 0x and the exponent is a p or P followed by a decimal number serving as the exponent of 2.

Floating literals can have embedded ‘_’ characters, which are ignored. The embedded ‘_’ are useful for formatting long literals to make them more readable, such as using them as a thousands separator: 

123_456.567_8         // 123456.5678
1_2_3_4_5_6_.5_6_7_8  // 123456.5678
1_2_3_4_5_6_.5e-6_    // 123456.5e-6

Floating literals with no suffix are of type double. Floats can be followed by one f, F, or L suffix. The f or F suffix means it is a float, and L means it is a real.

If a floating literal is followed by i, then it is an ireal (imaginary) type.

Examples:

0x1.FFFFFFFFFFFFFp1023 // double.max
0x1p-52                // double.epsilon
1.175494351e-38F       // float.min
6.3i                   // idouble 6.3
6.3fi                  // ifloat 6.3
6.3Li                  // ireal 6.3

It is an error if the literal exceeds the range of the type. It is not an error if the literal is rounded to fit into the significant digits of the type.

Complex literals are not tokens, but are assembled from real and imaginary expressions during semantic analysis: 

4.5 + 6.2i  // complex number (phased out)

Keywords

Keywords are reserved identifiers. See Also: Globally Defined Symbols. 
Keyword:
    abstract
    alias
    align
    asm
    assert
    auto

    body
    bool
    break
    byte

    case
    cast
    catch
    cdouble
    cent
    cfloat
    char
    class
    const
    continue
    creal

    dchar
    debug
    default
    delegate
    delete (deprecated)
    deprecated
    do
    double

    else
    enum
    export
    extern

    false 
    final
    finally
    float
    for
    foreach
    foreach_reverse
    function

    goto

    idouble
    if
    ifloat
    immutable
    import 
    in 
    inout 
    int
    interface
    invariant 
    ireal
    is

    lazy (See also: Lazy E­val­u­a­tion of ­Func­tion Ar­gu­ments)
    long

    macro (Reserved; unused)
    mixin 
    module

    new
    nothrow
    null

    out
    override

    package
    pragma
    private
    protected
    public
    pure

    real
    ref
    return

    scope
    shared
    short
    static
    struct
    super
    switch
    synchronized

    template
    this
    throw
    true 
    try
    typedef (deprecated)
    typeid
    typeof

    ubyte
    ucent
    uint
    ulong
    union
    unittest
    ushort

    version
    void
    volatile (deprecated)

    wchar
    while
    with

    __FILE__
    __MODULE__
    __LINE__
    __FUNCTION__
    __PRETTY_FUNCTION__

    __gshared
    __traits
    __vector
    __parameters

Globally Defined Symbols

These are defined in object_.d, which is automatically imported by the default implementation. 
Symbols:
    string (alias to immutable(char)[]) 
    wstring (alias to immutable(wchar)[])
    dstring (alias to immutable(dchar)[])

    size_t
    ptrdiff_t

Special Tokens

These tokens are replaced with other tokens according to the following table:

Special Tokens
Special TokenReplaced with
__DATE__string literal of the date of compilation "mmm dd yyyy"
__EOF__sets the scanner to the end of the file
__TIME__string literal of the time of compilation "hh:mm:ss"
__TIMESTAMP__string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy"
__VENDOR__Compiler vendor string, such as "Digital Mars D"
__VERSION__Compiler version as an integer, such as 2001

Special Token Sequences

SpecialTokenSequence:
    # line IntegerLiteral EndOfLine
    # line IntegerLiteral Filespec EndOfLine

Filespec:
    " Characters "

Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.

There is currently only one special token sequence, #line.

This sets the source line number to IntegerLiteral, and optionally the source file name to Filespec, beginning with the next line of source text. The source file and line number is used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.

For example: 

int #line 6 "foo\bar"
x;  // this is now line 6 of file foo\bar

Note that the backslash character is not treated specially inside Filespec strings.

Forums | Comments | Search | Downloads | Home