quarkdown-core/com.quarkdown.core.lexer.regex/RegexLexer/tokenize

tokenize

open override fun tokenize(): Sequence<Token>

Disassembles some raw string into smaller tokens.

Return

a lazy sequence of tokens, produced on demand