dmd.lexer

Implements the lexical analyzer, which converts source code into lexical tokens.

Specification: Lexical

Authors

Walter Bright

License

Boost License 1.0

Source: lexer.d

Documentation: https://dlang.org/phobos/dmd_lexer.html

Coverage: https://codecov.io/gh/dlang/dmd/src/master/src/dmd/lexer.d

  • CompileEnv

    Declaration

    struct CompileEnv;

    Values to use for various magic identifiers

    • versionNumber

      Declaration

      uint versionNumber;

      __VERSION__

    • date

      Declaration

      const(char)[] date;

      __DATE__

    • time

      Declaration

      const(char)[] time;

      __TIME__

    • vendor

      Declaration

      const(char)[] vendor;

      __VENDOR__

    • timestamp

      Declaration

      const(char)[] timestamp;

      __TIMESTAMP__

    • previewIn

      Declaration

      bool previewIn;

      in means [ref] scope const, accepts rvalues

    • ddocOutput

      Declaration

      bool ddocOutput;

      collect embedded documentation comments

    • masm

      Declaration

      bool masm;

      use MASM inline asm syntax

  • Lexer

    Declaration

    class Lexer;

    • Ccompile

      Declaration

      bool Ccompile;

      true if compiling ImportC

    • boolsize

      Declaration

      ubyte boolsize;

      size of a C Bool, default 1

    • shortsize

      Declaration

      ubyte shortsize;

      size of a C short, default 2

    • intsize

      Declaration

      ubyte intsize;

      size of a C int, default 4

    • longsize

      Declaration

      ubyte longsize;

      size of C long, 4 or 8

    • long_longsize

      Declaration

      ubyte long_longsize;

      size of a C long long, default 8

    • long_doublesize

      Declaration

      ubyte long_doublesize;

      size of C long double, 8 or D real.sizeof

    • wchar_tsize

      Declaration

      ubyte wchar_tsize;

      size of C wchar_t, 2 or 4

    • eSink

      Declaration

      ErrorSink eSink;

      send error messages through this interface

    • compileEnv

      Declaration

      CompileEnv compileEnv;

      environment

    • this

      Declaration

      nothrow scope this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken, ErrorSink errorSink, const CompileEnv* compileEnv);

      Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).

      Parameters

      const(char)* filename

      used for error messages

      const(char)* base

      source code, must be terminated by a null (0) or EOF (0x1A) character

      size_t begoffset

      starting offset into base[]

      size_t endoffset

      the last offset to read into base[]

      bool doDocComment

      handle documentation comments

      bool commentToken

      comments become TOK.comment's

      ErrorSink errorSink

      where error messages go, must not be null

      CompileEnv* compileEnv

      version, vendor, date, time, etc.

    • this

      Declaration

      nothrow this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken, bool whitespaceToken, ErrorSink errorSink, const CompileEnv* compileEnv = null);

      Alternative entry point for DMDLIB, adds whitespaceToken

    • this

      Declaration

      nothrow scope @safe this(ErrorSink errorSink);

      Used for unittests for a mock Lexer

    • resetDefineLines

      Declaration

      final nothrow void resetDefineLines(const(char)[] slice);

      Reset lexer to lex #define's

    • nextDefineLine

      Declaration

      final nothrow void nextDefineLine();

      Set up for next #define line. p should be at start of next line.

    • empty

      Declaration

      final const pure nothrow @nogc @property @safe bool empty();

      Range interface

    • allocateToken

      Declaration

      pure nothrow @safe Token* allocateToken();

      Return Value

      a newly allocated Token.

    • peekNext

      Declaration

      final nothrow TOK peekNext();

      Look ahead at next token's value.

    • peekNext2

      Declaration

      final nothrow TOK peekNext2();

      Look 2 tokens ahead at value.

    • scan

      Declaration

      final nothrow void scan(Token* t);

      Turn next token in buffer into a token.

      Parameters

      Token* t

      the token to set the resulting Token to

    • peekPastParen

      Declaration

      final nothrow Token* peekPastParen(Token* tk);

      tk is on the opening (. Look ahead and return token that is past the closing ).

    • hexStringConstant

      Declaration

      final nothrow TOK hexStringConstant(Token* t);

      Lex hex strings: x"0A ae 34FE BD"

    • parseSpecialTokenSequence

      Declaration

      nothrow bool parseSpecialTokenSequence();

      Parse special token sequence:

      Return Value

      true if the special token sequence was handled

      References: https://dlang.org/spec/lex.html#special-token-sequence

    • poundLine

      Declaration

      final nothrow void poundLine(ref Token tok, bool linemarker);

      Parse line/file preprocessor directive: #line linnum [filespec] Allow __LINE__ for linnum, and __FILE__ for filespec. Accept linemarker format:

      linnum [filespec] {flags}

      There can be zero or more flags, which are one of the digits 1..4, and must be in ascending order. The flags are ignored.

      Parameters

      Token tok

      token we're on, which is linnum of linemarker

      bool linemarker

      true if line marker format and lexer is on linnum

      References: linemarker https://gcc.gnu.org/onlinedocs/gcc-11.1.0/cpp/Preprocessor-Output.html

    • skipToNextLine

      Declaration

      final nothrow void skipToNextLine(OutBuffer* defines = null);

      Scan forward to start of next line.

      Parameters

      OutBuffer* defines

      send characters to defines

    • combineComments

      Declaration

      static pure nothrow const(char)* combineComments(const(char)[] c1, const(char)[] c2, bool newParagraph);

      Combine two document comments into one, separated by an extra newline if newParagraph is true.