quarkdown-core/com.quarkdown.core.lexer.regex/StandardRegexLexer

StandardRegexLexer

class StandardRegexLexer(val source: CharSequence, patterns: List<TokenRegexPattern>, fillTokenType: (TokenData) -> Token? = null) : RegexLexer

A standard RegexLexer implementation that does not perform any manipulation on the final output and has an optional fixed fill token.

Parameters

source

the content to be tokenized

patterns

the patterns to search for, in descending order of priority

fillTokenType

type of token to assign to uncaptured groups in order to fill the gaps. A token will not be created if null and empty gaps will be present in the output

Constructors

StandardRegexLexer

constructor(source: CharSequence, patterns: List<TokenRegexPattern>, fillTokenType: (TokenData) -> Token? = null)

Properties

currentIndex

open override var currentIndex: Int

Index of the latest scanned character within source.

source

open override val source: CharSequence

Functions

createFillToken

open override fun createFillToken(position: IntRange): Token?

tokenize

open override fun tokenize(): List<Token>

Disassembles some raw string into smaller tokens.

updatePatterns

fun updatePatterns(newPatterns: (List<TokenRegexPattern>) -> List<TokenRegexPattern>): StandardRegexLexer

Creates a copy of this lexer with different patterns.