quarkdown-core/com.quarkdown.core.lexer

Package-level declarations

Types

abstract class AbstractLexer(val source: CharSequence) : Lexer

A Lexer that expects iterations through source content.

interface Lexer

A scanner that transforms raw string data into a list of token. For instance, the Markdown code Hello _Quarkdown_ is tokenized by its implementation into Hello , _Quarkdown_.

Token

abstract class Token(val data: TokenData)

A wrapper of a TokenData that may be parsed in order to extract information. A token can be parsed into a com.quarkdown.core.ast.Node.

TokenCache

class TokenCache

Memoizes tokens by source, so that a source tokenized repeatedly, such as the body of a lambda invoked in a loop, is lexed only once.

TokenCoordinates

data class TokenCoordinates(val line: Int, val column: Int)

The position of a token within the source code.

TokenData

data class TokenData(val text: String, val position: IntRange)

Data of a single, usually small, substring of the source code that stores a chunk of information. For instance, the Markdown code Hello _Quarkdown_ contains the tokens Hello , _, Quarkdown, _.

Functions

acceptAll

fun <T> Sequence<Token>.acceptAll(visitor: TokenVisitor<T>): Sequence<T>

Lazily accepts a sequence of tokens to a shared visitor.

toCoordinates

fun IntRange.toCoordinates(source: CharSequence): TokenCoordinates

Converts a range of indexes within a string to its (x, y) coordinates.