Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.

Lexical

The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text into tokens. The lexical grammar describes the syntax of these tokens. The grammar is designed to be suitable for high-speed scanning and to facilitate the implementation of a correct scanner. It has a minimum of special case rules and there is only one phase of translation.

Source Text

SourceFile:
    ByteOrderMark Moduleopt
    Shebang Moduleopt
    Moduleopt
ByteOrderMark:
    \uFEFF
Shebang: #! Charactersopt EndOfShebang
EndOfShebang: \u000A EndOfFile

Source text can be encoded as any one of the following:

One of the following UTF BOMs (Byte Order Marks) can be present at the beginning of the source text:

UTF Byte Order Marks
FormatBOM
UTF-8EF BB BF
UTF-16BEFE FF
UTF-16LEFF FE
UTF-32BE00 00 FE FF
UTF-32LEFF FE 00 00
ASCIIno BOM

If the source file does not begin with a BOM, then the first character must be less than or equal to U+0000007F.

The source text is decoded from its source representation into Unicode Characters. The Characters are further divided into: WhiteSpace, EndOfLine, Comments, SpecialTokenSequences, and Tokens, with the source terminated by an EndOfFile.

The source text is split into tokens using the maximal munch algorithm, i.e., the lexical analyzer assumes the longest possible token. For example, >> is a right-shift token rather than two greater-than tokens. There are two exceptions to this rule:

Character Set

Character:
    any Unicode character

End of File

EndOfFile:
    physical end of the file
    \u0000
    \u001A

The source text is terminated by whichever comes first.

End of Line

EndOfLine:
    \u000D
    \u000A
    \u000D \u000A
    \u2028
    \u2029
    EndOfFile

White Space

WhiteSpace:
    Space
    Space WhiteSpace
Space: \u0020 \u0009 \u000B \u000C

Comments

Comment:
    BlockComment
    LineComment
    NestingBlockComment
BlockComment: /* Charactersopt */
LineComment: // Charactersopt EndOfLine
NestingBlockComment: /+ NestingBlockCommentCharactersopt +/
NestingBlockCommentCharacters: NestingBlockCommentCharacter NestingBlockCommentCharacter NestingBlockCommentCharacters
NestingBlockCommentCharacter: Character NestingBlockComment
Characters: Character Character Characters

There are three kinds of comments:

  1. Block comments can span multiple lines, but do not nest.
  2. Line comments terminate at the end of the line.
  3. Nesting block comments can span multiple lines and can nest.

The contents of strings and comments are not tokenized. Consequently, comment openings occurring within a string do not begin a comment, and string delimiters within a comment do not affect the recognition of comment closings and nested /+ comment openings. With the exception of /+ occurring within a /+ comment, comment openings within a comment are ignored.

a = /+ // +/ 1;    // parses as if 'a = 1;'
a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";'
a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'

Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.

Tokens

Tokens:
    Token
    Token Tokens
Token: { } TokenNoBraces
TokenNoBraces: Identifier StringLiteral CharacterLiteral IntegerLiteral FloatLiteral Keyword / /= . .. ... & &= && | |= || - -= -- + += ++ < <= << <<= > >= >>= >>>= >> >>> ! != ( ) [ ] ? , ; : $ = == * *= % %= ^ ^= ^^ ^^= ~ ~= @ =>

Identifiers

Identifier:
    IdentifierStart
    IdentifierStart IdentifierChars
IdentifierChars: IdentifierChar IdentifierChar IdentifierChars
IdentifierStart: _ Letter UniversalAlpha
IdentifierChar: IdentifierStart 0 NonZeroDigit

Identifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D of the C99 Standard. Identifiers can be arbitrarily long, and are case sensitive.

Implementation Defined: Identifiers starting with __ (two underscores) are reserved.

String Literals

StringLiteral:
    WysiwygString
    AlternateWysiwygString
    DoubleQuotedString
    DelimitedString
    TokenString

A string literal is either a wysiwyg quoted string, a double quoted string, a delimited string, or a token string.

In all string literal forms, an EndOfLine is regarded as a single \n character.

Wysiwyg Strings

WysiwygString:
    r" WysiwygCharactersopt " StringPostfixopt
AlternateWysiwygString: ` WysiwygCharactersopt ` StringPostfixopt
WysiwygCharacters: WysiwygCharacter WysiwygCharacter WysiwygCharacters
WysiwygCharacter: Character EndOfLine

Wysiwyg ("what you see is what you get") quoted strings can be defined using either of two syntaxes.

In the first form, they are enclosed between r" and ". All characters between the r" and " are part of the string. There are no escape sequences inside wysiwyg strings.

r"I am Oz"
r"c:\games\Sudoku.exe"
r"ab\n" // string is 4 characters,
        // 'a', 'b', '\', 'n'

Alternatively, wysiwyg strings can be enclosed by backquotes, using the ` character.

`the Great and Powerful.`
`c:\games\Empire.exe`
`The "lazy" dog`
`a"b\n`  // string is 5 characters,
         // 'a', '"', 'b', '\', 'n'

Double Quoted Strings

DoubleQuotedString:
    " DoubleQuotedCharactersopt " StringPostfixopt
DoubleQuotedCharacters: DoubleQuotedCharacter DoubleQuotedCharacter DoubleQuotedCharacters
DoubleQuotedCharacter: Character EscapeSequence EndOfLine

Double quoted strings are enclosed by "". EscapeSequences can be embedded in them.

"Who are you?"
"c:\\games\\Doom.exe"
"ab\n"   // string is 3 characters,
         // 'a', 'b', and a linefeed
"ab
"        // string is 3 characters,
         // 'a', 'b', and a linefeed

Delimited Strings

DelimitedString:
    q" Delimiter WysiwygCharactersopt MatchingDelimiter " StringPostfixopt
    q"( ParenDelimitedCharactersopt )" StringPostfixopt
    q"[ BracketDelimitedCharactersopt ]" StringPostfixopt
    q"{ BraceDelimitedCharactersopt }" StringPostfixopt
    q"< AngleDelimitedCharactersopt >" StringPostfixopt
Delimiter: Identifier
MatchingDelimiter: Identifier
ParenDelimitedCharacters: WysiwygCharacter WysiwygCharacter ParenDelimitedCharacters ( ParenDelimitedCharactersopt )
BracketDelimitedCharacters: WysiwygCharacter WysiwygCharacter BracketDelimitedCharacters [ BracketDelimitedCharactersopt ]
BraceDelimitedCharacters: WysiwygCharacter WysiwygCharacter BraceDelimitedCharacters { BraceDelimitedCharactersopt }
AngleDelimitedCharacters: WysiwygCharacter WysiwygCharacter AngleDelimitedCharacters < AngleDelimitedCharactersopt >

Delimited strings use various forms of delimiters. The delimiter, whether a character or identifier, must immediately follow the " without any intervening whitespace. The terminating delimiter must immediately precede the closing " without any intervening whitespace. A nesting delimiter nests, and is one of the following characters:

Nesting Delimiters
DelimiterMatching Delimiter
[]
()
<>
{}
q"(foo(xxx))"   // "foo(xxx)"
q"[foo{]"       // "foo{"

If the delimiter is an identifier, the identifier must be immediately followed by a newline, and the matching delimiter must be the same identifier starting at the beginning of the line:

writeln(q"EOS
This
is a multi-line
heredoc string
EOS"
);

The newline following the opening identifier is not part of the string, but the last newline before the closing identifier is part of the string. The closing identifier must be placed on its own line at the leftmost column.

Otherwise, the matching delimiter is the same as the delimiter character:

q"/foo]/"          // "foo]"
// q"/abc/def/"    // error

Token Strings

TokenString:
    q{ TokenStringTokensopt } StringPostfixopt
TokenStringTokens: TokenStringToken TokenStringToken TokenStringTokens
TokenStringToken: TokenNoBraces { TokenStringTokensopt }

Token strings open with the characters q{ and close with the token }. In between must be valid D tokens. The { and } tokens nest. The string is formed of all the characters between the opening and closing of the token string, including comments.

q{this is the voice of} // "this is the voice of"
q{/*}*/ }               // "/*}*/ "
q{ world(q{control}); } // " world(q{control}); "
q{ __TIME__ }           // " __TIME__ "
                        // i.e. it is not replaced with the time
// q{ __EOF__ }         // error
                        // __EOF__ is not a token, it's end of file

String Postfix

StringPostfix:
    c
    w
    d

The optional StringPostfix character gives a specific type to the string, rather than it being inferred from the context. The types corresponding to the postfix characters are:

String Literal Postfix Characters
PostfixTypeAlias
cimmutable(char)[]string
wimmutable(wchar)[]wstring
dimmutable(dchar)[]dstring
"hello"c  // string
"hello"w  // wstring
"hello"d  // dstring

The string literals are assembled as UTF-8 char arrays, and the postfix is applied to convert to wchar or dchar as necessary as a final step.

Escape Sequences

EscapeSequence:
    \'
    \"
    \?
    \\
    \0
    \a
    \b
    \f
    \n
    \r
    \t
    \v
    \x HexDigit HexDigit
    \ OctalDigit
    \ OctalDigit OctalDigit
    \ OctalDigit OctalDigit OctalDigit
    \u HexDigit HexDigit HexDigit HexDigit
    \U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit
    \ NamedCharacterEntity
Escape Sequences
SequenceMeaning
\'Literal single-quote: '
\"Literal double-quote: "
\?Literal question mark: ?
\\Literal backslash: \
\0Binary zero (NUL, U+0000).
\aBEL (alarm) character (U+0007).
\bBackspace (U+0008).
\fForm feed (FF) (U+000C).
\nEnd-of-line (U+000A).
\rCarriage return (U+000D).
\tHorizontal tab (U+0009).
\vVertical tab (U+000B).
\xnnByte value in hexadecimal, where nn is specified as two hexadecimal digits.
For example: \xFF represents the character with the value 255.
See also: std.conv.hexString.
\n
\nn
\nnn
Byte value in octal.
For example: \101 represents the character with the value 65 ('A'). Analogous to hexadecimal characters, the largest byte value is \377 (= \xFF in hexadecimal or 255 in decimal)
See also: std.conv.octal.
\unnnnUnicode character U+nnnn, where nnnn are four hexadecimal digits.
For example, \u03B3 represents the Unicode character γ (U+03B3 - GREEK SMALL LETTER GAMMA).
\UnnnnnnnnUnicode character U+nnnnnnnn, where nnnnnnnn are 8 hexadecimal digits.
For example, \U0001F603 represents the Unicode character U+1F603 (SMILING FACE WITH OPEN MOUTH).
\nameNamed character entity from the HTML5 specification.
These names begin with & and end with ;, e.g., &euro;. See NamedCharacterEntity.

Character Literals

CharacterLiteral:
    ' SingleQuotedCharacter '
SingleQuotedCharacter: Character EscapeSequence

Character literals are a single character or escape sequence enclosed by single quotes.

'h'   // the letter h
'\n'  // newline
'\\'  // the backslash character

A character literal resolves to one of type char, wchar, or dchar (see Basic Data Types).

Otherwise, it resolves to the type with the smallest size it will fit into.

Integer Literals

IntegerLiteral:
    Integer
    Integer IntegerSuffix
Integer: DecimalInteger BinaryInteger HexadecimalInteger
IntegerSuffix: L u U Lu LU uL UL
DecimalInteger: 0 NonZeroDigit NonZeroDigit DecimalDigitsUS
BinaryInteger: BinPrefix BinaryDigitsNoSingleUS
BinPrefix: 0b 0B
HexadecimalInteger: HexPrefix HexDigitsNoSingleUS
NonZeroDigit: 1 2 3 4 5 6 7 8 9
DecimalDigits: DecimalDigit DecimalDigit DecimalDigits
DecimalDigitsUS: DecimalDigitUS DecimalDigitUS DecimalDigitsUS
DecimalDigitsNoSingleUS: DecimalDigit DecimalDigit DecimalDigitsUS DecimalDigitsUS DecimalDigit
DecimalDigitsNoStartingUS: DecimalDigit DecimalDigit DecimalDigitsUS
DecimalDigit: 0 NonZeroDigit
DecimalDigitUS: DecimalDigit _
BinaryDigitsNoSingleUS: BinaryDigit BinaryDigit BinaryDigitsUS BinaryDigitsUS BinaryDigit BinaryDigitsUS BinaryDigit BinaryDigitsUS
BinaryDigitsUS: BinaryDigitUS BinaryDigitUS BinaryDigitsUS
BinaryDigit: 0 1
BinaryDigitUS: BinaryDigit _
OctalDigit: 0 1 2 3 4 5 6 7
HexDigits: HexDigit HexDigit HexDigits
HexDigitsUS: HexDigitUS HexDigitUS HexDigitsUS
HexDigitsNoSingleUS: HexDigit HexDigit HexDigitsUS HexDigitsUS HexDigit
HexDigitsNoStartingUS: HexDigit HexDigit HexDigitsUS
HexDigit: DecimalDigit HexLetter
HexDigitUS: HexDigit _
HexLetter: a b c d e f A B C D E F

Integers can be specified in decimal, binary, or hexadecimal.

10      // decimal
0b1010  // binary
0xA     // hex

Integers can have embedded ‘_’ characters after a digit to improve readability, which are ignored.

20_000        // leagues under the sea
867_5309      // number on the wall
1_522_000     // thrust of F1 engine (lbf sea level)
0xBAAD_F00D   // magic number for debugging

Integers can be immediately followed by one ‘L’ or one of ‘u’ or ‘U’ or both. Note that there is no ‘l’ suffix.

The type of the integer is resolved as follows:

Decimal Literal Types
LiteralType
Usual decimal notation
0 .. 2_147_483_647int
2_147_483_648 .. 9_223_372_036_854_775_807long
9_223_372_036_854_775_808 .. 18_446_744_073_709_551_615ulong
Explicit suffixes
0L .. 9_223_372_036_854_775_807Llong
0U .. 4_294_967_295Uuint
4_294_967_296U .. 18_446_744_073_709_551_615Uulong
0UL .. 18_446_744_073_709_551_615ULulong
Hexadecimal notation
0x0 .. 0x7FFF_FFFFint
0x8000_0000 .. 0xFFFF_FFFFuint
0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFFlong
0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFFulong
Hexadecimal notation with explicit suffixes
0x0L .. 0x7FFF_FFFF_FFFF_FFFFLlong
0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFLulong
0x0U .. 0xFFFF_FFFFUuint
0x1_0000_0000U .. 0xFFFF_FFFF_FFFF_FFFFUulong
0x0UL .. 0xFFFF_FFFF_FFFF_FFFFULulong

An integer literal may not exceed these values.

Best Practices: Octal integer notation is not supported for integer literals. However, octal integer literals can be interpreted at compile time through the std.conv.octal template, as in octal!167.

Floating Point Literals

FloatLiteral:
    Float Suffixopt
    Integer FloatSuffix ImaginarySuffixopt
    Integer RealSuffixopt ImaginarySuffix
Float: DecimalFloat HexFloat
DecimalFloat: LeadingDecimal . DecimalDigitsNoStartingUSopt LeadingDecimal . DecimalDigitsNoStartingUS DecimalExponent . DecimalDigitsNoStartingUS DecimalExponentopt LeadingDecimal DecimalExponent
DecimalExponent: DecimalExponentStart DecimalDigitsNoSingleUS
DecimalExponentStart: e E e+ E+ e- E-
HexFloat: HexPrefix HexDigitsNoSingleUS . HexDigitsNoStartingUS HexExponent HexPrefix . HexDigitsNoStartingUS HexExponent HexPrefix HexDigitsNoSingleUS HexExponent
HexPrefix: 0x 0X
HexExponent: HexExponentStart DecimalDigitsNoSingleUS
HexExponentStart: p P p+ P+ p- P-
Suffix: FloatSuffix ImaginarySuffixopt RealSuffix ImaginarySuffixopt ImaginarySuffix
FloatSuffix: f F
RealSuffix: L
ImaginarySuffix: i
LeadingDecimal: DecimalInteger 0 DecimalDigitsNoSingleUS

Floats can be in decimal or hexadecimal format, and must have at least one digit and either a decimal point, an exponent, or a FloatSuffix.

Decimal floats can have an exponent which is e or E followed by a decimal number serving as the exponent of 10.

-1.0
1e2               // 100.0
1e-2              // 0.01
-1.175494351e-38F // float.min

Hexadecimal floats are preceded by a 0x or 0X and the exponent is a p or P followed by a decimal number serving as the exponent of 2.

0xAp0                  // 10.0
0x1p2                  // 4.0
0x1.FFFFFFFFFFFFFp1023 // double.max
0x1p-52                // double.epsilon

Floating literals can have embedded _ characters after a digit to improve readability, which are ignored.

2.645_751
6.022140857E+23
6_022.140857E+20
6_022_.140_857E+20_
0.0                    // double
0F                     // float
0.0L                   // real

The literal may not exceed the range of the type. The literal is rounded to fit into the significant digits of the type.

If a floating literal has a . and a type suffix, at least one digit must be in-between:

1f;  // OK, float
1.f; // error
1.;  // OK, double
Note: Floating literals followed by i to denote imaginary floating point values have been deprecated.

Keywords

Keywords are reserved identifiers.

Keyword:
    abstract
    alias
    align
    asm
    assert
    auto
body bool break byte
case cast catch cdouble cent cfloat char class const continue creal
dchar debug default delegate delete deprecated do double
else enum export extern
false final finally float for foreach foreach_reverse function
goto
idouble if ifloat immutable import in inout int interface invariant ireal is
lazy long
macro mixin module
new nothrow null
out override
package pragma private protected public pure
real ref return
scope shared short static struct super switch synchronized
template this throw true try typeid typeof
ubyte ucent uint ulong union unittest ushort
version void
wchar while with
__FILE__ __FILE_FULL_PATH__ __MODULE__ __LINE__ __FUNCTION__ __PRETTY_FUNCTION__
__gshared __traits __vector __parameters

Special Tokens

These tokens are replaced with other tokens according to the following table:

Special Tokens
Special TokenReplaced with
__DATE__string literal of the date of compilation "mmm dd yyyy"
__EOF__tells the scanner to ignore everything after this token
__TIME__string literal of the time of compilation "hh:mm:ss"
__TIMESTAMP__string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy"
__VENDOR__Compiler vendor string
__VERSION__Compiler version as an integer
Implementation Defined: The replacement string literal for __VENDOR__ and the replacement integer value for __VERSION__.

Special Token Sequences

SpecialTokenSequence:
    # line IntegerLiteral Filespecopt EndOfLine
    # line __LINE__ Filespecopt EndOfLine
Filespec:
    " DoubleQuotedCharactersopt "

Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.

Special token sequences are terminated by the first newline that follows the first # token at the beginning of the sequence.

There is currently only one special token sequence, #line.

This sets the line number of the next source line to IntegerLiteral, and optionally the current source file name to Filespec, beginning with the next line of source text.

For example:

int #line 6 "pkg/mod.d"
x;  // this is now line 6 of file pkg/mod.d
Implementation Defined: The source file and line number is typically used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.
Introduction
Grammar