Lexer

interface Lexer

A scanner that transforms raw string data into a list of token. For instance, the Markdown code Hello _Quarkdown_ is tokenized by its implementation into Hello , _Quarkdown_.

Inheritors

Properties

Link copied to clipboard
abstract val source: CharSequence

The content to be tokenized.

Functions

Link copied to clipboard
abstract fun tokenize(): Sequence<Token>

Disassembles some raw string into smaller tokens.