quarkdown-core/com.quarkdown.core.lexer.regex/RegexLexer

RegexLexer

abstract class RegexLexer(val source: CharSequence, patterns: List<TokenRegexPattern>) : AbstractLexer

A Lexer that identifies tokens by matching Regex patterns.

Tokenization uses an iterative find loop: each iteration calls Regex.find starting from currentIndex, so that walker-driven position advances are respected without restarting the entire regex search.

Parameters

source

the content to be tokenized

patterns

the patterns to search for, in descending order of priority

Inheritors

StandardRegexLexer

Constructors

RegexLexer

constructor(source: CharSequence, patterns: List<TokenRegexPattern>)

Properties

currentIndex

open override var currentIndex: Int

Index of the latest scanned character within source.

source

open override val source: CharSequence

Functions

createFillToken

abstract fun createFillToken(position: IntRange): Token?

tokenize

open override fun tokenize(): Sequence<Token>

Disassembles some raw string into smaller tokens.