quarkdown-core/com.quarkdown.core.lexer

Package-level declarations

Types

abstract class AbstractLexer(val source: CharSequence) : Lexer

A Lexer that expects iterations through source content.

interface Lexer

A scanner that transforms raw string data into a list of token. For instance, the Markdown code Hello _Quarkdown_ is tokenized by its implementation into Hello , _Quarkdown_.

Token

abstract class Token(val data: TokenData)

A wrapper of a TokenData that may be parsed in order to extract information. A token can be parsed into a com.quarkdown.core.ast.Node.

TokenCoordinates

data class TokenCoordinates(val line: Int, val column: Int)

The position of a token within the source code.

TokenData

data class TokenData(val text: String, val position: IntRange, val groups: Sequence<String> = emptySequence(), val namedGroups: Map<String, String> = emptyMap())

Data of a single, usually small, substring of the source code that stores a chunk of information. For instance, the Markdown code Hello _Quarkdown_ contains the tokens Hello , _, Quarkdown, _.

Functions

acceptAll

fun Sequence<Token>.acceptAll(visitor: TokenVisitor<Node>, context: MutableContext, isBlock: Boolean): Sequence<Node>

Like the generic acceptAll, but specialized to Node output: each parsed node is also passed through wrapIfPrimitive, so primitive nodes (e.g. com.quarkdown.core.ast.base.block.Heading) are wrapped into a com.quarkdown.core.ast.quarkdown.FunctionCallNode backed by their stdlib function, both at the top level (see com.quarkdown.core.pipeline.stages.ParsingStage) and inside nested content (see com.quarkdown.core.parser.BlockTokenParser).

toCoordinates

fun IntRange.toCoordinates(source: CharSequence): TokenCoordinates

Converts a range of indexes within a string to its (x, y) coordinates.