dmd.lexer

immutable ubyte[256] cmtable;

Do our own char maps

class Lexer;

this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken);

Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).

Parameters:

const(char)* filename	used for error messages
const(char)* base	source code, must be terminated by a null (0) or EOF (0x1A) character
size_t begoffset	starting offset into base[]
size_t endoffset	the last offset to read into base[]
bool doDocComment	handle documentation comments
bool commentToken	comments become TOKcomment's

final TOK peekNext();

Look ahead at next token's value.

final TOK peekNext2();

Look 2 tokens ahead at value.

final void scan(Token* t);

Turn next token in buffer into a token.

final Token* peekPastParen(Token* tk);

tk is on the opening (. Look ahead and return token that is past the closing ).

final uint escapeSequence();

Parse escape sequence.

final TOK wysiwygStringConstant(Token* t, int tc);

final TOK hexStringConstant(Token* t);

Lex hex strings: x"0A ae 34FE BD"

final TOK delimitedStringConstant(Token* t);

Lex delimited strings: q"(foo(xxx))" // "foo(xxx)" q"[foo(]" // "foo(" q"/foo]/" // "foo]" q"HERE foo HERE" // "foo\n"

Input: p is on the "

final TOK tokenStringConstant(Token* t);

Lex delimited strings: q{ foo(xxx) } // " foo(xxx) " q{foo(} // "foo(" q{{foo}"}"} // "{foo}"}""

Input: p is on the q

final TOK escapeStringConstant(Token* t, int wide);

final TOK charConstant(Token* t);

final void stringPostfix(Token* t);

Get postfix of string literal.

final TOK number(Token* t);

Read in a number. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.

Returns:

TKnum TKdouble,...

final TOK inreal(Token* t);

Read in characters, converting them to real.

Bugs:

Exponent overflow not detected. Too much requested precision is not detected.

final void poundLine();

parse: #line linnum [filespec] also allow __LINE__ for linnum, and __FILE__ for filespec

final uint decodeUTF();

Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.

final void getDocComment(Token* t, uint lineComment, bool newParagraph);

Parse doc comment embedded between t.ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a lineComment or a blockComment. Append to previous one for this token.

If newParagraph is true, an extra newline will be added between adjoining doc comments.

static const(char)* combineComments(const(char)* c1, const(char)* c2, bool newParagraph);

Combine two document comments into one, separated by an extra newline if newParagraph is true.