dmd.lexer

Implements the lexical analyzer, which converts source code into lexical tokens.

Specification: Lexical

Authors

Walter Bright

License

Boost License 1.0

Source: lexer.d

Documentation: https://dlang.org/phobos/dmd_lexer.html

Coverage: https://codecov.io/gh/dlang/dmd/src/master/src/dmd/lexer.d

  • Lexer

    Declaration

    class Lexer;

    • this

      Declaration

      pure nothrow this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken);

      Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).

      Parameters

      const(char)* filename

      used for error messages

      const(char)* base

      source code, must be terminated by a null (0) or EOF (0x1A) character

      size_t begoffset

      starting offset into base[]

      size_t endoffset

      the last offset to read into base[]

      bool doDocComment

      handle documentation comments

      bool commentToken

      comments become TOK.comment's

    • allocateToken

      Declaration

      pure nothrow @safe Token* allocateToken();

      Return Value

      a newly allocated Token.

    • peekNext

      Declaration

      final nothrow TOK peekNext();

      Look ahead at next token's value.

    • peekNext2

      Declaration

      final nothrow TOK peekNext2();

      Look 2 tokens ahead at value.

    • scan

      Declaration

      final nothrow void scan(Token* t);

      Turn next token in buffer into a token.

    • peekPastParen

      Declaration

      final nothrow Token* peekPastParen(Token* tk);

      tk is on the opening (. Look ahead and return token that is past the closing ).

    • combineComments

      Declaration

      static pure nothrow const(char)* combineComments(const(char)[] c1, const(char)[] c2, bool newParagraph);

      Combine two document comments into one, separated by an extra newline if newParagraph is true.