Lexer
extends AbstractLexer
in package
Base class for writing simple lexers, i.e. for creating small DSLs.
Lexer moved into its own Component Doctrine\Common\Lexer. This class only stays for being BC.
Tags
Table of Contents
- $lookahead : array<string|int, mixed>
- The next token in the input.
- $token : array<string|int, mixed>
- The last matched/seen token.
- $input : string
- Lexer original input string.
- $peek : int
- Current peek of current lexer position.
- $position : int
- Current lexer position in input string.
- $tokens : array<string|int, mixed>
- Array of scanned tokens.
- getInputUntilPosition() : string
- Retrieve the original lexer's input until a given position.
- getLiteral() : string
- Gets the literal for a given token.
- glimpse() : array<string|int, mixed>|null
- Peeks at the next token, returns it and immediately resets the peek.
- isA() : bool
- Checks if given value is identical to the given token.
- isNextToken() : bool
- Checks whether a given token matches the current lookahead.
- isNextTokenAny() : bool
- Checks whether any of the given tokens matches the current lookahead.
- moveNext() : bool
- Moves to the next token in the input string.
- peek() : array<string|int, mixed>|null
- Moves the lookahead token forward.
- reset() : void
- Resets the lexer.
- resetPeek() : void
- Resets the peek pointer to 0.
- resetPosition() : void
- Resets the lexer position on the input to the given position.
- setInput() : void
- Sets the input data to be tokenized.
- skipUntil() : void
- Tells the lexer to skip input tokens until it sees a token with the given value.
- getCatchablePatterns() : array<string|int, mixed>
- Lexical catchable patterns.
- getModifiers() : string
- Regex modifiers
- getNonCatchablePatterns() : array<string|int, mixed>
- Lexical non-catchable patterns.
- getType() : int
- Retrieve token type. Also processes the token value if necessary.
- scan() : void
- Scans the input string for tokens.
Properties
$lookahead
The next token in the input.
public
array<string|int, mixed>
$lookahead
$token
The last matched/seen token.
public
array<string|int, mixed>
$token
$input
Lexer original input string.
private
string
$input
$peek
Current peek of current lexer position.
private
int
$peek
= 0
$position
Current lexer position in input string.
private
int
$position
= 0
$tokens
Array of scanned tokens.
private
array<string|int, mixed>
$tokens
= array()
Each token is an associative array containing three items:
- 'value' : the string value of the token in the input string
- 'type' : the type of the token (identifier, numeric, string, input parameter, none)
- 'position' : the position of the token in the input string
Methods
getInputUntilPosition()
Retrieve the original lexer's input until a given position.
public
getInputUntilPosition(int $position) : string
Parameters
- $position : int
Return values
string —getLiteral()
Gets the literal for a given token.
public
getLiteral(int $token) : string
Parameters
- $token : int
Return values
string —glimpse()
Peeks at the next token, returns it and immediately resets the peek.
public
glimpse() : array<string|int, mixed>|null
Return values
array<string|int, mixed>|null —The next token or NULL if there are no more tokens ahead.
isA()
Checks if given value is identical to the given token.
public
isA(mixed $value, int $token) : bool
Parameters
- $value : mixed
- $token : int
Return values
bool —isNextToken()
Checks whether a given token matches the current lookahead.
public
isNextToken(int|string $token) : bool
Parameters
- $token : int|string
Return values
bool —isNextTokenAny()
Checks whether any of the given tokens matches the current lookahead.
public
isNextTokenAny(array<string|int, mixed> $tokens) : bool
Parameters
- $tokens : array<string|int, mixed>
Return values
bool —moveNext()
Moves to the next token in the input string.
public
moveNext() : bool
Return values
bool —peek()
Moves the lookahead token forward.
public
peek() : array<string|int, mixed>|null
Return values
array<string|int, mixed>|null —The next token or NULL if there are no more tokens ahead.
reset()
Resets the lexer.
public
reset() : void
Return values
void —resetPeek()
Resets the peek pointer to 0.
public
resetPeek() : void
Return values
void —resetPosition()
Resets the lexer position on the input to the given position.
public
resetPosition(int $position) : void
Parameters
- $position : int
-
Position to place the lexical scanner.
Return values
void —setInput()
Sets the input data to be tokenized.
public
setInput(string $input) : void
The Lexer is immediately reset and the new input tokenized. Any unprocessed tokens from any previous input are lost.
Parameters
- $input : string
-
The input to be tokenized.
Return values
void —skipUntil()
Tells the lexer to skip input tokens until it sees a token with the given value.
public
skipUntil(string $type) : void
Parameters
- $type : string
-
The token type to skip until.
Return values
void —getCatchablePatterns()
Lexical catchable patterns.
protected
abstract getCatchablePatterns() : array<string|int, mixed>
Return values
array<string|int, mixed> —getModifiers()
Regex modifiers
protected
getModifiers() : string
Return values
string —getNonCatchablePatterns()
Lexical non-catchable patterns.
protected
abstract getNonCatchablePatterns() : array<string|int, mixed>
Return values
array<string|int, mixed> —getType()
Retrieve token type. Also processes the token value if necessary.
protected
abstract getType(string &$value) : int
Parameters
- $value : string
Return values
int —scan()
Scans the input string for tokens.
protected
scan(string $input) : void
Parameters
- $input : string
-
A query string.