Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.

dmd.lexer

Compiler implementation of the D programming language.
Authors:

Source: lexer.d

immutable ubyte[256] cmtable;
Do our own char maps
class Lexer;
this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken);
Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).
Parameters:
const(char)* filename used for error messages
const(char)* base source code, must be terminated by a null (0) or EOF (0x1A) character
size_t begoffset starting offset into base[]
size_t endoffset the last offset to read into base[]
bool doDocComment handle documentation comments
bool commentToken comments become TOKcomment's
final TOK peekNext();
Look ahead at next token's value.
final TOK peekNext2();
Look 2 tokens ahead at value.
final void scan(Token* t);
Turn next token in buffer into a token.
final Token* peekPastParen(Token* tk);
tk is on the opening (. Look ahead and return token that is past the closing ).
final uint escapeSequence();
Parse escape sequence.
final TOK wysiwygStringConstant(Token* t, int tc);
final TOK hexStringConstant(Token* t);
Lex hex strings: x"0A ae 34FE BD"
final TOK delimitedStringConstant(Token* t);
Lex delimited strings: q"(foo(xxx))" // "foo(xxx)" q"[foo(]" // "foo(" q"/foo]/" // "foo]" q"HERE foo HERE" // "foo\n"

Input: p is on the "

final TOK tokenStringConstant(Token* t);
Lex delimited strings: q{ foo(xxx) } // " foo(xxx) " q{foo(} // "foo(" q{{foo}"}"} // "{foo}"}""

Input: p is on the q

final TOK escapeStringConstant(Token* t, int wide);
final TOK charConstant(Token* t);
final void stringPostfix(Token* t);
Get postfix of string literal.
final TOK number(Token* t);
Read in a number. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.
Returns:
TKnum TKdouble,...
final TOK inreal(Token* t);
Read in characters, converting them to real.
Bugs:
Exponent overflow not detected. Too much requested precision is not detected.
final void poundLine();

parse: #line linnum [filespec] also allow __LINE__ for linnum, and __FILE__ for filespec

final uint decodeUTF();
Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.
final void getDocComment(Token* t, uint lineComment, bool newParagraph);
Parse doc comment embedded between t.ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a lineComment or a blockComment. Append to previous one for this token.
If newParagraph is true, an extra newline will be added between adjoining doc comments.
static const(char)* combineComments(const(char)* c1, const(char)* c2, bool newParagraph);
Combine two document comments into one, separated by an extra newline if newParagraph is true.