Lexical
The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes the syntax of those tokens. The grammar is designed to be suitable for high speed scanning and to make it easy to write a correct scanner for it. It has a minimum of special case rules and there is only one phase of translation. The tokens are readily recognizable by those familiar with C and C++.Source Text
D source text can be in one of the following formats:- ASCII
- UTF-8
- UTF-16BE
- UTF-16LE
- UTF-32BE
- UTF-32LE
| Format | BOM |
|---|---|
| UTF-8 | EF BB BF |
| UTF-16BE | FE FF |
| UTF-16LE | FF FE |
| UTF-32BE | 00 00 FE FF |
| UTF-32LE | FF FE 00 00 |
| ASCII | no BOM |
If the source file does not start with a BOM, then the first character must be less than or equal to U+0000007F.
There are no digraphs or trigraphs in D.
The source text is decoded from its source representation into Unicode Characters. The Characters are further divided into: WhiteSpace, EndOfLine, Comments, SpecialTokenSequences, Tokens, all followed by EndOfFile.
The source text is split into tokens using the maximal munch technique, i.e., the lexical analyzer tries to make the longest token it can. For example >> is a right shift token, not two greater than tokens. There are two exceptions to this rule:
- A .. embedded inside what looks like two floating point literals, as in 1..2, is interpreted as if the .. was separated by a space from the first integer.
- A 1.a is interpreted as the three tokens 1, ., and a, while 1. a is interpreted as the two tokens 1. and a
Character Set
Character: any Unicode character
End of File
EndOfFile: physical end of the file \u0000 \u001AThe source text is terminated by whichever comes first.
End of Line
EndOfLine: \u000D \u000A \u000D \u000A \u2028 \u2029 EndOfFileThere is no backslash line splicing, nor are there any limits on the length of a line.
White Space
WhiteSpace: Space Space WhiteSpace Space: \u0020 \u0009 \u000B \u000C
Comments
Comment: BlockComment LineComment NestingBlockComment BlockComment: /* Characters */ LineComment: // Characters EndOfLine NestingBlockComment: /+ NestingBlockCommentCharacters +/ NestingBlockCommentCharacters: NestingBlockCommentCharacter NestingBlockCommentCharacter NestingBlockCommentCharacters NestingBlockCommentCharacter: Character NestingBlockComment Characters: Character Character Characters
D has three kinds of comments:
- Block comments can span multiple lines, but do not nest.
- Line comments terminate at the end of the line.
- Nesting block comments can span multiple lines and can nest.
The contents of strings and comments are not tokenized. Consequently, comment openings occurring within a string do not begin a comment, and string delimiters within a comment do not affect the recognition of comment closings and nested "/+" comment openings. With the exception of "/+" occurring within a "/+" comment, comment openings within a comment are ignored.
a = /+ // +/ 1; // parses as if 'a = 1;' a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";' a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.
Tokens
Token: Identifier StringLiteral CharacterLiteral IntegerLiteral FloatLiteral Keyword / /= . .. ... & &= && | |= || - -= -- + += ++ < <= << <<= <> <>= > >= >>= >>>= >> >>> ! != !<> !<>= !< !<= !> !>= ( ) [ ] { } ? , ; : $ = == * *= % %= ^ ^= ^^ ^^= ~ ~= @ => #
Identifiers
Identifier: IdentifierStart IdentifierStart IdentifierChars IdentifierChars: IdentifierChar IdentifierChar IdentifierChars IdentifierStart: _ Letter UniversalAlpha IdentifierChar: IdentifierStart 0 NonZeroDigitIdentifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D. (This is the C99 Standard.) Identifiers can be arbitrarily long, and are case sensitive. Identifiers starting with __ (two underscores) are reserved.
String Literals
StringLiteral: WysiwygString AlternateWysiwygString DoubleQuotedString HexString DelimitedString TokenString WysiwygString: r" WysiwygCharacters " StringPostfixopt AlternateWysiwygString: ` WysiwygCharacters ` StringPostfixopt WysiwygCharacters: WysiwygCharacter WysiwygCharacter WysiwygCharacters WysiwygCharacter: Character EndOfLine DoubleQuotedString: " DoubleQuotedCharacters " StringPostfixopt DoubleQuotedCharacters: DoubleQuotedCharacter DoubleQuotedCharacter DoubleQuotedCharacters DoubleQuotedCharacter: Character EscapeSequence EndOfLine EscapeSequence: \' \" \? \\ \0 \a \b \f \n \r \t \v \x HexDigit HexDigit \ OctalDigit \ OctalDigit OctalDigit \ OctalDigit OctalDigit OctalDigit \u HexDigit HexDigit HexDigit HexDigit \U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit \ NamedCharacterEntity HexString: x" HexStringChars " StringPostfixopt HexStringChars: HexStringChar HexStringChar HexStringChars HexStringChar: HexDigit WhiteSpace EndOfLine StringPostfix: c w d DelimitedString: q" Delimiter WysiwygCharacters MatchingDelimiter " TokenString: q{ Tokens }
A string literal is either a double quoted string, a wysiwyg quoted string, a delimited string, a token string, or a hex string.
In all string literal forms, an EndOfLine is regarded as a single \n character.
Wysiwyg Strings
Wysiwyg ("what you see is what you get") quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string. There are no escape sequences inside r" ":
r"hello" r"c:\root\foo.exe" r"ab\n" // string is 4 characters, // 'a', 'b', '\', 'n'
An alternate form of wysiwyg strings are enclosed by backquotes, the ` character. The ` character is not available on some keyboards and the font rendering of it is sometimes indistinguishable from the regular ' character. Since, however, the ` is rarely used, it is useful to delineate strings with " in them.
`hello` `c:\root\foo.exe` `The "lazy" dog` `a"b\n` // string is 5 characters, // 'a', '"', 'b', '\', 'n'
Double Quoted Strings
Double quoted strings are enclosed by "". Escape sequences can be embedded into them with the typical \ notation.
"hello" "c:\\root\\foo.exe" "ab\n" // string is 3 characters, // 'a', 'b', and a linefeed "ab " // string is 3 characters, // 'a', 'b', and a linefeed
Hex Strings
Hex strings allow string literals to be created using hex data. The hex data need not form valid UTF characters.
x"0A" // same as "\x0A" x"00 FBCD 32FD 0A" // same as // "\x00\xFB\xCD\x32\xFD\x0A"Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2. Adjacent strings are concatenated with the ~ operator, or by simple juxtaposition:
"hello " ~ "world" ~ "\n" // forms the string // 'h','e','l','l','o',' ', // 'w','o','r','l','d',linefeedThe following are all equivalent:
"ab" "c" r"ab" r"c" r"a" "bc" "a" ~ "b" ~ "c"
The optional StringPostfix character gives a specific type to the string, rather than it being inferred from the context. This is useful when the type cannot be unambiguously inferred, such as when overloading based on string type. The types corresponding to the postfix characters are:
| Postfix | Type | Aka |
|---|---|---|
| c | immutable(char)[] | string |
| w | immutable(wchar)[] | wstring |
| d | immutable(dchar)[] | dstring |
"hello"c // string "hello"w // wstring "hello"d // dstring
The string literals are assembled as UTF-8 char arrays, and the postfix is applied to convert to wchar or dchar as necessary as a final step.
String literals are read only. Writes to string literals cannot always be detected, but cause undefined behavior.
Delimited Strings
Delimited strings use various forms of delimiters. The delimiter, whether a character or identifer, must immediately follow the " without any intervening whitespace. The terminating delimiter must immediately precede the closing " without any intervening whitespace. A nesting delimiter nests, and is one of the following characters:
| Delimiter | Matching Delimiter |
|---|---|
| [ | ] |
| ( | ) |
| < | > |
| { | } |
q"(foo(xxx))" // "foo(xxx)" q"[foo{]" // "foo{"
If the delimiter is an identifier, the identifier must be immediately followed by a newline, and the matching delimiter is the same identifier starting at the beginning of the line:
writeln(q"EOS
This
is a multi-line
heredoc string
EOS"
);
The newline following the opening identifier is not part of the string, but the last newline before the closing identifier is part of the string. The closing identifier must be placed on its own line at the leftmost column.
Otherwise, the matching delimiter is the same as the delimiter character:
q"/foo]/" // "foo]" // q"/abc/def/" // error
Token Strings
Token strings open with the characters q{ and close with the token }. In between must be valid D tokens. The { and } tokens nest. The string is formed of all the characters between the opening and closing of the token string, including comments.
q{foo} // "foo" q{/*}*/ } // "/*}*/ " q{ foo(q{hello}); } // " foo(q{hello}); " q{ __TIME__ } // " __TIME__ " // i.e. it is not replaced with the time // q{ __EOF__ } // error // __EOF__ is not a token, it's end of file
Escape Sequences
The following table explains the meaning of the escape sequences listed in EscapeSequence:
| Sequence | Meaning |
|---|---|
| \' | Literal single-quote: ' |
| \" | Literal double-quote: " |
| \? | Literal question mark: ? |
| \\ | Literal backslash: \ |
| \0 | Binary zero (NUL, U+0000). |
| \a | BEL (alarm) character (U+0007). |
| \b | Backspace (U+0008). |
| \f | Form feed (FF) (U+000C). |
| \n | End-of-line (U+000A). |
| \r | Carriage return (U+000D). |
| \t | Horizontal tab (U+0009). |
| \v | Vertical tab (U+000B). |
| \xnn | Byte value in hexadecimal, where nn is
specified as two hexadecimal digits. For example: \xFF represents the character with the value 255. |
| \n \nn \nnn | Byte value in
octal. For example: \775 represents the character with the value 509. |
| \unnnn | Unicode character U+nnnn, where
nnnn are four hexadecimal digits. For example, \u042F represents the Unicode character Я (U+42F). |
| \Unnnnnnnn | Unicode character U+nnnnnnnn,
where nnnnnnnn are 8 hexadecimal digits. For example, \U0001F603 represents the Unicode character U+1F603 (SMILING FACE WITH OPEN MOUTH). |
| \name | Named character entity from the HTML5 specification. See NamedCharacterEntity for more details. |
Character Literals
CharacterLiteral: ' SingleQuotedCharacter ' SingleQuotedCharacter: Character EscapeSequenceCharacter literals are a single character or escape sequence enclosed by single quotes, ' '.
Integer Literals
IntegerLiteral: Integer Integer IntegerSuffix Integer: DecimalInteger BinaryInteger HexadecimalInteger IntegerSuffix: L u U Lu LU uL UL DecimalInteger: 0 NonZeroDigit NonZeroDigit DecimalDigitsUS BinaryInteger: BinPrefix BinaryDigitsUS BinPrefix: 0b 0B HexadecimalInteger: HexPrefix HexDigitsNoSingleUS NonZeroDigit: 1 2 3 4 5 6 7 8 9 DecimalDigits: DecimalDigit DecimalDigit DecimalDigits DecimalDigitsUS: DecimalDigitUS DecimalDigitUS DecimalDigitsUS DecimalDigitsNoSingleUS: DecimalDigit DecimalDigit DecimalDigitsUS DecimalDigitsUS DecimalDigit DecimalDigitsNoStartingUS: DecimalDigit DecimalDigit DecimalDigitsUS DecimalDigit: 0 NonZeroDigit DecimalDigitUS: DecimalDigit _ BinaryDigitsUS: BinaryDigitUS BinaryDigitUS BinaryDigitsUS BinaryDigit: 0 1 BinaryDigitUS: BinaryDigit _ OctalDigit: 0 1 2 3 4 5 6 7 HexDigits: HexDigit HexDigit HexDigits HexDigitsUS: HexDigitUS HexDigitUS HexDigitsUS HexDigitsNoSingleUS: HexDigit HexDigit HexDigitsUS HexDigitsUS HexDigit HexDigitsNoStartingUS: HexDigit HexDigit HexDigitsUS HexDigit: DecimalDigit HexLetter HexDigitUS: HexDigit _ HexLetter: a b c d e f A B C D E F
Integers can be specified in decimal, binary, or hexadecimal.
Decimal integers are a sequence of decimal digits.
Binary integers are a sequence of binary digits preceded by a ‘0b’ or ‘0B’.
C-style octal integer notation was deemed too easy to mix up with decimal notation; it is only fully supported in string literals. D still supports octal integer literals interpreted at compile time through the std.conv.octal template, as in octal!167.
Hexadecimal integers are a sequence of hexadecimal digits preceded by a ‘0x’ or ‘0X’.
Integers can have embedded ‘_’ characters, which are ignored. The embedded ‘_’ are useful for formatting long literals, such as using them as a thousands separator:
123_456 // 123456 1_2_3_4_5_6_ // 123456
Integers can be immediately followed by one ‘L’ or one of ‘u’ or ‘U’ or both. Note that there is no ‘l’ suffix.
The type of the integer is resolved as follows:
| Literal | Type | Usual decimal notation |
|---|---|
| 0 .. 2_147_483_647 | int |
| 2_147_483_648 .. 9_223_372_036_854_775_807 | long | Explicit suffixes |
| 0L .. 9_223_372_036_854_775_807L | long |
| 0U .. 4_294_967_295U | uint |
| 4_294_967_296U .. 18_446_744_073_709_551_615U | ulong |
| 0UL .. 18_446_744_073_709_551_615UL | ulong | Hexadecimal notation |
| 0x0 .. 0x7FFF_FFFF | int |
| 0x8000_0000 .. 0xFFFF_FFFF | uint |
| 0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFF | long |
| 0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFF | ulong | Hexadecimal notation with explicit suffixes |
| 0x0L .. 0x7FFF_FFFF_FFFF_FFFFL | long |
| 0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFL | ulong |
| 0x0U .. 0xFFFF_FFFFU | uint |
| 0x1_0000_0000U .. 0xFFFF_FFFF_FFFF_FFFFU | ulong |
| 0x0UL .. 0xFFFF_FFFF_FFFF_FFFFUL | ulong |
Floating Point Literals
FloatLiteral: Float Float Suffix Integer ImaginarySuffix Integer FloatSuffix ImaginarySuffix Integer RealSuffix ImaginarySuffix Float: DecimalFloat HexFloat DecimalFloat: LeadingDecimal . LeadingDecimal . DecimalDigits DecimalDigits . DecimalDigitsNoStartingUS DecimalExponent . DecimalInteger . DecimalInteger DecimalExponent LeadingDecimal DecimalExponent DecimalExponent DecimalExponentStart DecimalDigitsNoSingleUS DecimalExponentStart e E e+ E+ e- E- HexFloat: HexPrefix HexDigitsNoSingleUS . HexDigitsNoStartingUS HexExponent HexPrefix . HexDigitsNoStartingUS HexExponent HexPrefix HexDigitsNoSingleUS HexExponent HexPrefix: 0x 0X HexExponent: HexExponentStart DecimalDigitsNoSingleUS HexExponentStart: p P p+ P+ p- P- Suffix: FloatSuffix RealSuffix ImaginarySuffix FloatSuffix ImaginarySuffix RealSuffix ImaginarySuffix FloatSuffix: f F RealSuffix: L ImaginarySuffix: i LeadingDecimal: DecimalInteger 0 DecimalDigitsNoSingleUS
Floats can be in decimal or hexadecimal format.
Hexadecimal floats are preceded by a 0x or 0X and the exponent is a p or P followed by a decimal number serving as the exponent of 2.
Floating literals can have embedded ‘_’ characters, which are ignored. The embedded ‘_’ are useful for formatting long literals to make them more readable, such as using them as a thousands separator:
123_456.567_8 // 123456.5678 1_2_3_4_5_6_.5_6_7_8 // 123456.5678 1_2_3_4_5_6_.5e-6_ // 123456.5e-6
Floating literals with no suffix are of type double. Floats can be followed by one f, F, or L suffix. The f or F suffix means it is a float, and L means it is a real.
If a floating literal is followed by i, then it is an ireal (imaginary) type.
Examples:
0x1.FFFFFFFFFFFFFp1023 // double.max 0x1p-52 // double.epsilon 1.175494351e-38F // float.min 6.3i // idouble 6.3 6.3fi // ifloat 6.3 6.3Li // ireal 6.3
It is an error if the literal exceeds the range of the type. It is not an error if the literal is rounded to fit into the significant digits of the type.
Complex literals are not tokens, but are assembled from real and imaginary expressions during semantic analysis:
4.5 + 6.2i // complex number (phased out)
Keywords are reserved identifiers. See Also: Globally Defined Symbols.
Keyword: abstract alias align asm assert auto body bool break byte case cast catch cdouble cent cfloat char class const continue creal dchar debug default delegate delete (deprecated) deprecated do double else enum export extern false final finally float for foreach foreach_reverse function goto idouble if ifloat immutable import in inout int interface invariant ireal is lazy long macro (unused) mixin module new nothrow null out override package pragma private protected public pure real ref return scope shared short static struct super switch synchronized template this throw true try typedef (deprecated) typeid typeof ubyte ucent uint ulong union unittest ushort version void volatile (deprecated) wchar while with __FILE__ __FILE_FULL_PATH__ __MODULE__ __LINE__ __FUNCTION__ __PRETTY_FUNCTION__ __gshared __traits __vector __parameters
Globally Defined Symbols
These are defined in object_.d, which is automatically imported by the default implementation.Symbols: string (alias to immutable(char)[]) wstring (alias to immutable(wchar)[]) dstring (alias to immutable(dchar)[]) size_t ptrdiff_t
Special Tokens
These tokens are replaced with other tokens according to the following table:
| Special Token | Replaced with |
|---|---|
| __DATE__ | string literal of the date of compilation "mmm dd yyyy" |
| __EOF__ | sets the scanner to the end of the file |
| __TIME__ | string literal of the time of compilation "hh:mm:ss" |
| __TIMESTAMP__ | string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy" |
| __VENDOR__ | Compiler vendor string, such as "Digital Mars D" |
| __VERSION__ | Compiler version as an integer, such as 2001 |
Special Token Sequences
SpecialTokenSequence: # line IntegerLiteral EndOfLine # line IntegerLiteral Filespec EndOfLine Filespec: " Characters "
Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.
There is currently only one special token sequence, #line.
This sets the source line number to IntegerLiteral, and optionally the source file name to Filespec, beginning with the next line of source text. The source file and line number is used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.
For example:
int #line 6 "foo\bar" x; // this is now line 6 of file foo\bar
Note that the backslash character is not treated specially inside Filespec strings.