quarkdown-core/com.quarkdown.core.pipeline.stages/LexingStage

LexingStage

class LexingStage(source: CharSequence) : PipelineStage<Set<Library>, Sequence<Token>>

Pipeline stage responsible for lexical analysis (tokenization) of the input text.

This stage produces a lazy sequence of tokens as output from the provided source text. It uses the lexer factory from the context's flavor to create a block lexer that processes the source text and breaks it down into tokens.

The tokens produced by this stage are used by the ParsingStage to build an abstract syntax tree.

Parameters

source

the raw input text to be tokenized

Constructors

LexingStage

constructor(source: CharSequence)

Properties

hook

open override val hook: KProperty1<PipelineHooks, Pipeline.(Sequence<Token>) -> Unit>

A hook function that is invoked after the stage completes processing.

Functions

execute

open fun execute(input: Set<Library>, data: SharedPipelineData): Sequence<Token>

Executes this pipeline stage with the given input and shared data.

execute

fun <O> PipelineStage<Unit, O>.execute(data: SharedPipelineData): O

Utility function to execute a pipeline stage that takes Unit as input.

invokeHook

open fun invokeHook(data: SharedPipelineData, input: Set<Library>, output: Sequence<Token>)

Invokes the hook function for this stage, if one is defined.

process

open override fun process(input: Set<Library>, data: SharedPipelineData): Sequence<Token>

Processes the input I and produces an output O.

then

infix fun <A, B, C> PipelineStage<A, B>.then(next: PipelineStage<B, C>): PipelineStage<A, C>

Chains two pipeline stages together to form a new pipeline stage.

thenOptionally

infix fun <A, B> PipelineStage<A, B>.thenOptionally(next: PipelineStage<B, B>?): PipelineStage<A, B>

Conditionally chains this pipeline stage with next if it is not null.