Lexer
BashLexer
, CLexer
, ClojureLexer
, CppLexer
, DOSBatchLexer
, GroovyLexer
, JavaLexer
, JavaScriptLexer
, JFlexLexer
, LuaLexer
, PropertiesLexer
, PythonLexer
, RubyLexer
, ScalaLexer
, SqlLexer
, TALLexer
, XHTMLLexer
, XmlLexer
, XPathLexer
public abstract class DefaultJFlexLexer extends java.lang.Object implements Lexer
Modifier and Type | Field | Description |
---|---|---|
protected int |
offset |
|
protected int |
tokenLength |
|
protected int |
tokenStart |
Constructor | Description |
---|---|
DefaultJFlexLexer() |
Modifier and Type | Method | Description |
---|---|---|
void |
parse(javax.swing.text.Segment segment,
int ofst,
java.util.List<Token> tokens) |
The DefaultJFlexLexer simply calls the yylex method of a JFLex compatible
Lexer and adds the tokens obtained to an ArrayList.
|
protected Token |
token(TokenType type) |
Create and return a Token of given type.
|
protected Token |
token(TokenType type,
int pairValue) |
Create and return a Token of given type and pairValue.
|
protected Token |
token(TokenType type,
int start,
int length) |
Create and return a Token of given type from start with length
offset is added to start
|
protected Token |
token(TokenType type,
int tStart,
int tLength,
int newStart,
int newLength) |
Helper method to create and return a new Token from of TokenType
tokenStart and tokenLength will be modified to the newStart and
newLength params
|
abstract int |
yychar() |
Return the char number from beginning of input stream.
|
abstract char |
yycharat(int pos) |
Returns the character at position pos from the
matched text.
|
abstract int |
yylength() |
Returns the length of the matched text region.
|
abstract Token |
yylex() |
This is called to return the next Token from the Input Reader
|
abstract void |
yyreset(java.io.Reader reader) |
This will be called to reset the the lexer.
|
abstract java.lang.String |
yytext() |
Returns the text matched by the current regular expression.
|
protected int tokenStart
protected int tokenLength
protected int offset
protected Token token(TokenType type, int tStart, int tLength, int newStart, int newLength)
type
- tStart
- tLength
- newStart
- newLength
- protected Token token(TokenType type, int start, int length)
type
- start
- length
- protected Token token(TokenType type)
yychar()
and length from yylength()
offset is added to starttype
- protected Token token(TokenType type, int pairValue)
yychar()
and length from yylength()
offset is added to starttype
- pairValue
- public void parse(javax.swing.text.Segment segment, int ofst, java.util.List<Token> tokens)
parse
in interface Lexer
segment
- Text to parse.ofst
- offset to add to start of each token (useful for nesting)tokens
- List of Tokens to be added. This is done so that the caller creates the
appropriate List implementation and size. The parse method just adds to the listpublic abstract void yyreset(java.io.Reader reader)
reader
- public abstract Token yylex() throws java.io.IOException
java.io.IOException
public abstract char yycharat(int pos)
pos
- the position of the character to fetch.
A value from 0 to yylength()-1.public abstract int yylength()
public abstract java.lang.String yytext()
public abstract int yychar()